Search…
OpenTelemetry
The OpenTelemetry project provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. A standardized data format for distributed traces and metrics data eliminates the need for vendor-specific integrations.
The Release Agents send OpenTelemetry metrics for each of the conditions in the current policy step. These metrics are in OpenTelemetry Protocol (OTLP) format and use the ValueRecorder instrument. Each data point is labeled with information about the Kubernetes workload and metadata describing the Vamp release.
The following example is a single data point for Vamp Health metrics from step 3 of release from our Vamp Particles demo, taken from the collector log:
1
Metric #1
2
Descriptor:
3
-> Name: vamp.health
4
-> Description:
5
-> Unit: 1
6
-> DataType: DoubleGauge
7
DoubleDataPoints #0
8
Data point labels:
9
-> k8s.clusterName: default
10
-> k8s.deploymentName: particles
11
-> k8s.namespaceName: production
12
-> tags.app: particles
13
-> tags.version: 1.0.7
14
-> vamp.policy.step: 3
15
-> vamp.policy.step.completed: false
16
-> vamp.release.completed: false
17
-> vamp.release.id: e64ec960-7285-493d-aebc-f7612b530c21
18
-> vamp.release.policy.id: 12
19
-> vamp.release.policy.version: 90
20
-> vamp.release.service.name: particles
21
-> vamp.release.service.sourceVersion: 1.0.6
22
-> vamp.release.service.targetVersion: 1.0.7
23
StartTime: 0
24
Timestamp: 1615332904407521299
25
Value: 100.000000
Copied!
And this is the Vamp Health metric for the same release in Lightstep:
Vamp Health and API Success Rate metrics in Lightstep
We have tested the release metrics using Lightstep, New Relic One, and Prometheus endpoints.

Configuration

We leverage the OpenTelemetry Collector to allow the Vamp Release Agents to stream release metrics to any open source or commercial backend that supports OpenTelementry metrics. This greatly simplifies the configuration.
If you are sending metrics to a commercial backend like Lightstep or New Relic, the vendor-specific configuration, such as access tokens, is done in one place: the OpenTelemetry Collector.
There are two primary methods for deploying the OpenTelemetry Collector on Kubernetes:
  1. 1.
    Agent: one Collector instance per Node running as a Kubernetes DaemonSet, plus one Collector instance acting as an egress gateway. In this configuration, the Vamp Release Agents connect to the local agent.
  2. 2.
    Standalone Gateway: one Collector instance per cluster acting as an egress gateway. In this configuration, the Vamp Release Agents connect directly to the gateway Collector.
The Release Agents' OpenTelemetry support is enabled by setting the OTLP_COLLECTOR_GRPC_ADDRESS environment variable to the endpoint of either the local Collector agent or a Collector gateway. This variable is not set by default.
The Release Agent has two containers. TheOTLP_COLLECTOR_GRPC_ADDRESSenvironment must be set on the "vamp-release-agent" container.

Agent

If you want to send metrics through a local agent, the Vamp Release Agents need to be configured with the Pod's own IP address, using the Kubernetes downward API.
1
- name: HOST_IP
2
valueFrom:
3
fieldRef:
4
apiVersion: v1
5
fieldPath: status.hostIP
6
- name: OTLP_COLLECTOR_GRPC_ADDRESS
7
value: $(HOST_IP):55680
Copied!

Standalone Gateway

If you want to send metrics directly to a gateway Collector, the Vamp Release Agents need to be configured to use the gateway's Service.
The name otel-collector is based on the OpenTelemetry Kubernetes Collector deployment example:
1
- name: OTLP_COLLECTOR_GRPC_ADDRESS
2
value: otel-collector.default:55680
Copied!