Kubernetes annotation-based discovery for the OpenTelemetry Collector

In the world of containers and Kubernetes, observability is crucial. Users need to know the status of their workloads at any given time. In other words, they need observability into moving objects.

This is where the OpenTelemetry Collector and its receiver creator component come in handy. Users can set up fairly complex monitoring scenarios with a self-service approach, following the principle of least privilege at the cluster level.

The self-service approach is great, but how much self-service can it actually be? In this blog post, we will explore a newly added feature of the Collector that makes dynamic workload discovery even easier, providing a seamless experience for both administrators and users.

Automatic discovery for containers and pods

Applications running on containers and pods become moving targets for the monitoring system. With automatic discovery, monitoring agents like the Collector can track changes at the container and pod levels and dynamically adjust the monitoring configuration.

Today, the Collector—and specifically the receiver creator—can provide such an experience. Using the receiver creator, observability users can define configuration “templates” that rely on environment conditions. For example, as an observability engineer, you can configure your Collectors to enable the NGINX receiver when a NGINX pod is deployed on the cluster. The following configuration can achieve this:

receivers:
  receiver_creator:
    watch_observers: [k8s_observer]
    receivers:
      nginx:
        rule: type == "port" && port == 80 && pod.name matches "(?i)nginx"
        config:
          endpoint: 'http://`endpoint`/nginx_status'
          collection_interval: '15s'

The previous configuration is enabled when a pod is discovered via the Kubernetes API that exposes port 80 (the known port for NGINX) and its name matches the nginx keyword.

This is great, and as an SRE or Platform Engineer managing an observability solution, you can rely on this to meet your users’ needs for monitoring NGINX workloads. However, what happens if another team wants to monitor a different type of workload, such as Apache servers? They would need to inform your team, and you would need to update the configuration with a new conditional configuration block, take it through a pull request and review process, and finally deploy it. This deployment would require the Collector instances to restart for the new configuration to take effect. While this process might not be a big deal for some teams, there is definitely room for improvement.

So, what if, as a Collector user, you could simply enable automatic discovery and then let your cluster users tell the Collector how their workloads should be monitored by annotating their pods properly? That sounds awesome, and it’s not actually something new. OpenTelemetry already supports auto-instrumentation through the Kubernetes operator, allowing users to instrument their applications automatically just by annotating their pods. In addition, this is a feature that other monitoring agents in the observability industry already support, and users are familiar with it.

All this motivation led the OpenTelemetry community (GitHub issue) to create a similar feature for the Collector. We are happy to share that autodiscovery based on Kubernetes annotations is now supported in the Collector (GitHub issue)!

A solution

The solution is built on top of the existing functionality provided by the Kubernetes observer and receiver creator.

The K8s observer notifies the receiver creator about the objects appearing in the K8s cluster and provides all the information about them. In addition to the K8s object metadata, the observer supplies information about the discovered endpoints that the collector can connect to. This means that each discovered endpoint can potentially be used by a particular scraping receiver to fetch metrics data.

Each scraping receiver has a default configuration with only one required field: endpoint. Given that the endpoint information is provided by the Kubernetes observer, the only information that the user needs to provide explicitly is which receiver/scraper should be used to scrape data from a discovered endpoint. That information can be configured on the Collector, but as mentioned before, this is inconvenient. A much more convenient place to define which receiver can be used to scrape telemetry from a particular pod is the pod itself. Pod’s annotations is the natural place to put that kind of detail. Given that the receiver creator has access to the annotations, it can instantiate the proper receiver with the receiver’s default configuration and discovered endpoint.

The following annotation instructs the receiver creator that this particular pod runs NGINX, and the NGINX receiver can be used to scrape metrics from it:

io.opentelemetry.discovery.metrics/scraper: nginx

Apart from that, the discovery on the pod needs to be explicitly enabled with the following annotation:

io.opentelemetry.discovery.metrics/enabled: 'true'

In some scenarios, the default receiver’s configuration is not suitable for connecting to a particular pod. In that case, it’s possible to define custom configuration as part of another annotation:

io.opentelemetry.discovery.metrics/config: |
  endpoint: "http://`endpoint`/nginx_status"
  collection_interval: '20s'
  initial_delay: '20s'
  read_buffer_size: '10'

It’s important to mention that the configuration defined in the annotations cannot point the receiver creator to another pod. The Collector will reject such configurations.

In addition to the metrics scraping, the annotation-based discovery also supports log collection with filelog receiver. The following annotation can be used to enable log collection on a particular pod:

io.opentelemetry.discovery.logs/enabled: 'true'

Similar to metrics, an optional configuration can be provided in the following form:

io.opentelemetry.discovery.logs/config: |
  max_log_size: "2MiB"
  operators:
  - type: container
    id: container-parser
  - type: regex_parser
    regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'

If the set of filelog receiver operators needs to be changed, the full list, including the default container parser, has to be redefined because list config fields are entirely replaced when merged into the default configuration struct.

The discovery functionality has to be explicitly enabled in the receiver creator by adding the following configuration field:

receivers:
  receiver_creator:
    watch_observers: [k8s_observer]
    discovery:
      enabled: true

Give it a try

If you are an OpenTelemetry Collector user on Kubernetes, and you find this new feature interesting, see Receiver Creator configuration section to learn more.

Give it a try and let us know what you think via the #otel-collector channel of the CNCF Slack workspace.

• ←Previous
• Next→