The OpenTelemetry project provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. Providing a standardized data format for distributed traces and metrics data allows OpenTelemetry to eliminates the need for vendor-specific integrations.
The Release Agents send OpenTelementry metrics for each of the conditions in the current policy step. These metrics are in OTLP format and use the
ValueRecorder instrument. Each data point is labelled with information about the Kubernetes workload and metadata describing the Vamp release.
This is a single data point for Vamp Health metric from step 3 of release of our Vamp Particles demo taken from the collector log:
Metric #1Descriptor:-> Name: vamp.health-> Description:-> Unit: 1-> DataType: DoubleGaugeDoubleDataPoints #0Data point labels:-> k8s.clusterName: default-> k8s.deploymentName: particles-> k8s.namespaceName: production-> tags.app: particles-> tags.version: 1.0.7-> vamp.policy.step: 3-> vamp.policy.step.completed: false-> vamp.release.completed: false-> vamp.release.id: e64ec960-7285-493d-aebc-f7612b530c21-> vamp.release.policy.id: 12-> vamp.release.policy.version: 90-> vamp.release.service.name: particles-> vamp.release.service.sourceVersion: 1.0.6-> vamp.release.service.targetVersion: 1.0.7StartTime: 0Timestamp: 1615332904407521299Value: 100.000000
And this is the Vamp Health metric for the same release in Lightstep:
We have tested the release metrics using Lightstep, New Relic One and Prometheus endpoints.
We leverage the OpenTelemetry Collector to allow the Vamp Release Agents to stream release metrics to any open-source or commercial backend that supports OpenTelementry metrics. This greatly simplifies the configuration.
If you are sending metrics to a commercial backend like Lightstep or New Relic, then the vendor specific configuration such as access tokens is done in one place, the OpenTelementry Collector.
There are two primary methods of deploying the OpenTelemetry Collector on Kubernetes:
Agent: one Collector instance per Node running as a Kubernetes DaemonSet plus one Collector instance acting as an egress gateway. In this configuration, the Vamp Release Agents connect to the local agent.
Standalone Gateway: one Collector instance per cluster acting as an egress gateway. In this configuration, the Vamp Release Agents connect directly to the gateway Collector.
The Release Agents' OpenTelementry support is enabled by setting the
OTLP_COLLECTOR_GRPC_ADDRESS environment variable to the endpoint of either the local Collector agent or a Collector gateway. This variable is not set by default.
If you want to send metrics via a local agent, then the Vamp Release Agents need to be configured with the Pod's own IP using the Kubernetes downward API.
- name: HOST_IPvalueFrom:fieldRef:apiVersion: v1fieldPath: status.hostIP- name: OTLP_COLLECTOR_GRPC_ADDRESSvalue: $(HOST_IP):55680
If you want to send metrics direct to a gateway Collector, then the Vamp Release Agents need to be configured to use the gateway's Service.
otel-collector is based on the OpenTelementry Kubernetes Collector deployment example.
- name: OTLP_COLLECTOR_GRPC_ADDRESSvalue: otel-collector.default:55680