OpenTelemetry

‚Äč

The OpenTelemetry project provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. Providing a standardized data format for distributed traces and metrics data allows OpenTelemetry to eliminates the need for vendor-specific integrations.

The Release Agents send OpenTelementry metrics for each of the conditions in the current policy step. These metrics are in OTLP format and use the ValueRecorder instrument. Each data point is labelled with information about the Kubernetes workload and metadata describing the Vamp release.

This is a single data point for Vamp Health metric from step 3 of release of our Vamp Particles demo taken from the collector log:

Metric #1
Descriptor:
-> Name: vamp.health
-> Description:
-> Unit: 1
-> DataType: DoubleGauge
DoubleDataPoints #0
Data point labels:
-> k8s.clusterName: default
-> k8s.deploymentName: particles
-> k8s.namespaceName: production
-> tags.app: particles
-> tags.version: 1.0.7
-> vamp.policy.step: 3
-> vamp.policy.step.completed: false
-> vamp.release.completed: false
-> vamp.release.id: e64ec960-7285-493d-aebc-f7612b530c21
-> vamp.release.policy.id: 12
-> vamp.release.policy.version: 90
-> vamp.release.service.name: particles
-> vamp.release.service.sourceVersion: 1.0.6
-> vamp.release.service.targetVersion: 1.0.7
StartTime: 0
Timestamp: 1615332904407521299
Value: 100.000000

And this is the Vamp Health metric for the same release in Lightstep:

Vamp Health and API Success Rate metrics in Lightstep

We have tested the release metrics using Lightstep, New Relic One and Prometheus endpoints.

Configuration

We leverage the OpenTelemetry Collector to allow the Vamp Release Agents to stream release metrics to any open-source or commercial backend that supports OpenTelementry metrics. This greatly simplifies the configuration.

If you are sending metrics to a commercial backend like Lightstep or New Relic, then the vendor specific configuration such as access tokens is done in one place, the OpenTelementry Collector.

There are two primary methods of deploying the OpenTelemetry Collector on Kubernetes:

  • Agent: one Collector instance per Node running as a Kubernetes DaemonSet plus one Collector instance acting as an egress gateway. In this configuration, the Vamp Release Agents connect to the local agent.

  • Standalone Gateway: one Collector instance per cluster acting as an egress gateway. In this configuration, the Vamp Release Agents connect directly to the gateway Collector.

The Release Agents' OpenTelementry support is enabled by setting the OTLP_COLLECTOR_GRPC_ADDRESS environment variable to the endpoint of either the local Collector agent or a Collector gateway. This variable is not set by default.

The Release Agent has 2 containers theOTLP_COLLECTOR_GRPC_ADDRESSenvironment must be set on the "vamp-release-agent" container

Agent

If you want to send metrics via a local agent, then the Vamp Release Agents need to be configured with the Pod's own IP using the Kubernetes downward API.

- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: OTLP_COLLECTOR_GRPC_ADDRESS
value: $(HOST_IP):55680

Standalone Gateway

If you want to send metrics direct to a gateway Collector, then the Vamp Release Agents need to be configured to use the gateway's Service.

The name otel-collector is based on the OpenTelementry Kubernetes Collector deployment example.

- name: OTLP_COLLECTOR_GRPC_ADDRESS
value: otel-collector.default:55680