Kubernetes Prometheus Operator Hands-On

At first glance, Prometheus may seem quite a complicated product, but like any well-designed system, it consists of clearly expressed functional components and, in fact, does only three things:

  1. collects metrics
  2. executes rules
  3. saves results to the database time series data.

The article is devoted not so much to Prometheus itself, as to the integration of this system with Kubernetes, for which we actively use an auxiliary tool called Prometheus Operator. But it’s still necessary to begin with Prometheus …

First of all prom8s has 2 main functions:

  • Collect metrics from each monitoring target with defined scrape interval.
  • With defined evaluation_interval based on recording rules metrics are recording as a new metrics, based on alerting rules are firing alerts.

How to configure Prometheus?

Prometheus server has config and rule files (file with rules).

In config defined following sections:

  • scrape_configs - settings for searching monitoring targets
  • rule_files - list of rules files or directories where rules files located

    - /etc/prometheus/rules/rules-0/\*  
    - /etc/prometheus/rules/rules-1/\*
  • alerting - settings for searching alertmanagers to where alerts are sending.

Where is the list of targets coming from?

Algorithm of Prometheus looks like:

  1. Prom8s reads section from scrape_config and based on that he tweaks his own mechanism of Service Discovery (SD).
  2. Mechanism of SD is communicating with K8s API to discover service endpoints.
  3. Based on inputs from K8s SD mechanism prometheus updates the list of targets.

In scrape_configs defined list of scrape jobs, each of them has following definition:

- job_name: kube-prometheus/custom/0 # Name of scrape job'а  
                                       # shown in section Service Discovery  
  scrape_interval:30s                  # How often metrics are scraping  
  scrape_timeout:10s # Timeout on request  
  metrics_path: /metrics                # path to metrics  
  scheme: http                          # http or https  
 # Service Discovery settings  
  kubernetes_sd_configs: # means that we are getting targets from Kubernetes  
  - api_server:null # Use address of API-server from env  
                                       # variables  
    role:endpoints # targets are getting from endpoints  

# search endpoints only in following namespaces  
      - foo  
      - baz  

# Filtering settings what endpoints to add 
# What labels to add and remove for collected metrics  
 # Filtering based on value of label prometheus_custom_target

  # received from service binded with endpoint

  - source_labels: [__meta_kubernetes_service_label_prometheus_custom_target]  
    regex:.+ # fit any not empty label  
    action: keep  
    # Filtering based on port name  
  - source_labels: [__meta_kubernetes_endpoint_port_name]  
    regex:http-metrics # fit if port named http-metrics  
    action: keep  
 # Adding job label with using value of label prometheus_custom_target

 # from service, and after that adding prefix "custom-" to value of label 

 # job label using for grouping in Prometheus. He defines name of group,  
 # which will shown as a target in targets page, also he will added to each metric  
 # scraped from this target for ability to filter in rules and dashboards  
  - source_labels: [__meta_kubernetes_service_label_prometheus_custom_target]  
    regex: (.*)  
    target_label: job  
    replacement: custom-$1  
    action: replace  
    # Adding namespace label  
  - source_labels: [__meta_kubernetes_namespace]  
    regex: (.*)  
    target_label: namespace  
    replacement: $1  
    action: replace  
    # Adding service label  
  - source_labels: [__meta_kubernetes_service_name]  
    regex: (.*)  
    target_label: service  
    replacement: $1  
    action: replace  
    # Adding instance label there we will see pod name  
  - source_labels: [__meta_kubernetes_pod_name]  
    regex: (.*)  
    target_label: instance  
    replacement: $1  
    action: replace

By this method Prometheus automatically tracks:

  • Adding and removing pods (When K8s adding/removing pods endpoints are changing and Prom8s see that and adding/removing targets)
  • Adding and removing services (targets) in defined namespaces.

Configuration changes are required in following cases:

  • Addition of new scrape config (new kind of service that should be monitored)
  • Namespace changing

With basic knowledge of Prometheus we can move to Prometheus Operator - Operators in general are design pattern by coreos built for Kubernetes in the remaining part of this article we will see how it makes the management of Prometheus in Kubernetes a breeze.

Prometheus Operator: what is it?

To simplify let’s descrive in highlevel what the promethues operator offers:

Firstly as many opertators the Prometheus Operator introduces CRD’s (Custom Resource Definitions) the three CRD’s are:

  1. prometheus - define installation of Prometheus cluster.
  2. servicemonitor - define how to monitor set of services.
  3. alertmanager - define alertmanagers cluster.

Secondndly, the operator tracks resources of prometheus and create for each of them following resources:

  1. A StatefulSet (with Prom8s itself)
  2. A Kubernetes Secret with the pormetheus.yaml (config of Prometheus) and configmaps.json (config for prometheus-config-reloader).

And finally, the operator tracks resources of servicemonitor and ConfigMaps with rules and based on those rules updates the prometheus.yaml and configmaps.json configurations (eventhough they are keeped in “secrets”).

So what happens with Prometheus in the POD?

Pod consist from two containers:

  1. prometheus - with prometheus binary
  2. prometheus-config-reloader - a side-car container which tracks for changes in prometheus.yaml and if necessary calls reload configuration of Prometheus whilst tracking ConfigMap changes according to certain conditions, then if necessary updates them and restart Prometheus.

Pod uses three volumes:

  1. Config - mounted secret (two files: prometheus.yaml and configmap.json). Binded to both containers.
  2. Rules - emptyDir to where prometheus-config-reloader writes files and prometheus reads. Binded to both but Prom8s only in read mode
  3. Data - prometheus tsdb mounted only to Prometheus container.

How Service Monitor works?

  1. Prometheus Operator checks Service Monitors (and track their adding/removing/changes). Service monitor selector defined in prometheus resource. (see details in docs)
  2. For each Service Monitor, if namespaces is not defined then Prometheus Operator identifies (with calls to k8s API ) list of namespaces where services exists with selected labels in Service Monitor.
  3. Prometheus Operator reads service monitor resources and identified namespaces for generating part of configuration (section scrape_configs) and save configuration to secret.
  4. Data comes in pod from secrets rely on Kubernetes functionality and prometheus.yaml file is updating.
  5. When prometheus.yaml file is changed prometheus-config-reloader send http request to prometheus for.
  6. Prometheus reads configuration file and applies changes in scrape_configs section.

How is handling ConfigMaps with rules?

  1. Prometheus Operator tracks ConfigMaps, matched by ruleSelector, defined in prometheus resource.
  2. If ConfigMap is created/deleted, Prometheus Operator updates prometheus.yaml and following Service Monitors logic starts.
  3. If content of ConfigMap is changing then Prometheus Operator updates file configmap.json (in this file defined list of ConfiMaps and their checksum).
  4. Data comes in pod from secrets rely on Kubernetes functionality and configmap.json file is updating.
  5. When configmap.json file is changed prometheus-config-reloader is downloading ConfigMaps in rules directory(emptyDir) and send http request to prometheus for reloading.
  6. Prometheus reads configuration file and applies changes.

Practical Example

In this part we will focus on how easy it is to add new targets to prometheus managed by the prometheus operator and make those targets discoverable by matching kubernetes labels.

In order to add new scrape target for prometheus first of all we should have a pod with an exporter that will collect metrics from some source (mongodb, rabbitmq, zookeeper or application endpoint, etc).

In kubernetes we can use a deployment, or a daemonset, depending on your use case, and type of exporter. In most cases you will use a deployment for collecting metrics from various endpoints accessible via kube-dns within the internal network of the cluster. In some usecases cases a daemonset should be used, as an example the node-exporter collects operational node metrics and it should be located on each node in the cluster.

In the following example we are going to collect metrics from rabbitmq:

Deployment of rabbitmq exporter:

apiVersion: extensions/v1beta1
kind: Deployment
  name: rabbitmq-exporter
  namespace: monitoring
  replicas: 1
        k8s-app: rabbitmq-exporter
      - env:
        - name: RABBIT_URL
          value: http://rabbitmq-management.tikal-io-minikube:15672
        - name: PUBLISH_PORT
          value: "9419"
        name: rabbitmq-exporter
        image: kbudde/rabbitmq-exporter
        - containerPor

In order to take advantage of Kubernetes built-in loadbalancing mechanisem we will create a service which “exposes” the exporter via labels to prometheus to scrapre, the service Resource Definition will remain between restarts of the deployment / pod’s hance in Prometheus the name will always remain the same rabbitmq-exporter and the ip might vary due to outages / rolling updates / rollbacks etc.

So let’s create a service to serve the rabbitmq-exporter:

Service of rabbitmq exporter:

apiVersion: v1
kind: Service
    k8s-app: rabbitmq-exporter
  name: rabbitmq-exporter
  namespace: monitoring
  - name: mqex
    port: 9419
    protocol: TCP
    targetPort: mqex
    k8s-app: rabbitmq-exporter

When we have service we can create ServiceMonitor CRD that will select services by matching labels and will add matched services to prometheus target list Automatically.

apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor 
      k8s-app: rabbitmq-exporter 
    name: rabbitmq-exporter 
    namespace: monitoring 
      - interval: 30s 
        port: mqex 
        - monitoring 
          k8s-app: rabbitmq-exporter

The above ServiceMonitor will render in the ui to somthing simalr to the following:


Prometheus Operator can help boost your monitoring capabilities in your Kubernetes environment. Comments / Finding are welcome in the form of comments below.

Thank you for reading Stas

Thank you for your interest!

We will contact you as soon as possible.

Want to Know More?

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com
Thank you for your interest!

We will contact you as soon as possible.

Let's talk

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com