Search…
OpenTelemetry
The OpenTelemetry project provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. Providing a standardized data format for distributed traces and metrics data allows OpenTelemetry to eliminates the need for vendor-specific integrations.
The Release Agents send OpenTelementry metrics for each of the conditions in the current policy step. These metrics are in OTLP format and use the ValueRecorder instrument. Each data point is labelled with information about the Kubernetes workload and metadata describing the Vamp release.
This is a single data point for Vamp Health metric from step 3 of release of our Vamp Particles demo taken from the collector log:
1
Metric #1
2
Descriptor:
3
-> Name: vamp.health
4
-> Description:
5
-> Unit: 1
6
-> DataType: DoubleGauge
7
DoubleDataPoints #0
8
Data point labels:
9
-> k8s.clusterName: default
10
-> k8s.deploymentName: particles
11
-> k8s.namespaceName: production
12
-> tags.app: particles
13
-> tags.version: 1.0.7
14
-> vamp.policy.step: 3
15
-> vamp.policy.step.completed: false
16
-> vamp.release.completed: false
17
-> vamp.release.id: e64ec960-7285-493d-aebc-f7612b530c21
18
-> vamp.release.policy.id: 12
19
-> vamp.release.policy.version: 90
20
-> vamp.release.service.name: particles
21
-> vamp.release.service.sourceVersion: 1.0.6
22
-> vamp.release.service.targetVersion: 1.0.7
23
StartTime: 0
24
Timestamp: 1615332904407521299
25
Value: 100.000000
Copied!
And this is the Vamp Health metric for the same release in Lightstep:
Vamp Health and API Success Rate metrics in Lightstep
We have tested the release metrics using Lightstep, New Relic One and Prometheus endpoints.

Configuration

We leverage the OpenTelemetry Collector to allow the Vamp Release Agents to stream release metrics to any open-source or commercial backend that supports OpenTelementry metrics. This greatly simplifies the configuration.
If you are sending metrics to a commercial backend like Lightstep or New Relic, then the vendor specific configuration such as access tokens is done in one place, the OpenTelementry Collector.
There are two primary methods of deploying the OpenTelemetry Collector on Kubernetes:
    Agent: one Collector instance per Node running as a Kubernetes DaemonSet plus one Collector instance acting as an egress gateway. In this configuration, the Vamp Release Agents connect to the local agent.
    Standalone Gateway: one Collector instance per cluster acting as an egress gateway. In this configuration, the Vamp Release Agents connect directly to the gateway Collector.
The Release Agents' OpenTelementry support is enabled by setting the OTLP_COLLECTOR_GRPC_ADDRESS environment variable to the endpoint of either the local Collector agent or a Collector gateway. This variable is not set by default.
The Release Agent has 2 containers theOTLP_COLLECTOR_GRPC_ADDRESSenvironment must be set on the "vamp-release-agent" container

Agent

If you want to send metrics via a local agent, then the Vamp Release Agents need to be configured with the Pod's own IP using the Kubernetes downward API.
1
- name: HOST_IP
2
valueFrom:
3
fieldRef:
4
apiVersion: v1
5
fieldPath: status.hostIP
6
- name: OTLP_COLLECTOR_GRPC_ADDRESS
7
value: $(HOST_IP):55680
Copied!

Standalone Gateway

If you want to send metrics direct to a gateway Collector, then the Vamp Release Agents need to be configured to use the gateway's Service.
The name otel-collector is based on the OpenTelementry Kubernetes Collector deployment example.
1
- name: OTLP_COLLECTOR_GRPC_ADDRESS
2
value: otel-collector.default:55680
Copied!
Last modified 7mo ago