Observability

Configuring Prometheus to Scrap Container Metrics (cAdvisor)

Ganesh Kumar reddy's photo
Ganesh Kumar reddy
·

3 min read

What is Prometheus ?

Prometheus is an Open-Source monitoring tool that helps to collect metric data and provide tools to visualize the collected data.

It allows us to create alert when the metric reaches its defined threshold value. It collects metrics by scraping the targets who exposed metrics through https endpoint.

The scraped metrics will be stored in time series database which later can be quired using PromQL.

Architecture:

Monitoring Will Helps

  • Gives better insights about internal system

  • Speedup Troubleshooting

  • Monitor performance of application

Monitoring Containers

As most of the applications are deploying in containerized environments its important to understand how we are configuring and monitoring alerts in the system.

Here we have two things

1. Monitoring Docker Engine Metrics

2. Monitoring Container Metrics

Configuring Docker Engine Metrics

Lets check how many containers are present in the docker host

docker ps

By default, Docker does not export any metrics, so configure it to export metrics on port 9323. Restart Docker service after making the required changes.

vi /etc/docker/daemon.json

Add below given lines in it: (here specify ip and port that you want to expose your metric endpoint

{
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true
}

Restart docker service:

systemctl restart docker

Verify if docker is exporting the metrics now:

curl localhost:9323/metrics

Create a new job in Prometheus called docker and add the Docker host (i.e. localhost) as a target. Make sure to restart the Prometheus service after making the required changes.

vi /etc/prometheus/prometheus.yml

- job_name: "docker"
  static_configs:
    - targets: ["localhost:9323"]

Restart the prometheus service

systemctl restart prometheus.service

In the expression search box, run the following query to report the number of containers and their state.

engine_daemon_container_states_containers

Configuring Container Metrics

To get the container-level metrics, a cAdvisor container needs to run on the docker host. The cAdvisor can collect and expose metrics.

 vi cAdvisor.yml

version: '3.4'
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor
    container_name: cadvisor
    privileged: true
    devices:
      - "/dev/kmsg:/dev/kmsg"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - 8070:8080

docker compose up -d

Create a new job in Prometheus called cadvisor and add localhost:8070 as a target. Restart the Prometheus service after making the required changes.

Edit /etc/prometheus/prometheus.yml file:

vi /etc/prometheus/prometheus.yml

Add below given lines under scrape_configs:

  - job_name: "cadvisor"
    static_configs:
      - targets: ["localhost:8070"]

Restart prometheus service:

systemctl restart prometheus

In the Expression search box, run the following query to find Bytes transmitted over the network by the container per second in the last minute

rate(container_network_transmit_bytes_total[1m])

Thank You 😃

DevopsobservabilityKubernetes#prometheusGrafana