Monitor Anything With a Python Exporter for Prometheus

What is a Python Exporter?

If you don't know yet, Prometheus is a monitoring tool that supports 2 mechanisms — pull and push — to pull data (metrics) from applications, usually combined with Grafana to visualize them as charts.

If you're already familiar with setting up database services, web servers, queues, etc., you've probably run into tools that expose metrics — it's easy to find ready-made tools for each application with the keyword "App name + exporter". But these ready-made tools get built for a general model to cover as many use cases as possible, so it's fairly common to miss out on metrics specific to your particular case. That's why Python Exporter was created, to solve this problem.

Prometheus's data types

Since Prometheus provides different data types to model different charts, we need to know which data type to use for each case. As of this writing, May 2022, Prometheus supports 6 main data types — let's go through each one along with how it's used:

Counter

This metric type acts as a counter — it can only go up, never down. When the exporter restarts from scratch, the counter's value resets to 0. When exposing a counter metric, its name automatically gets '_total' appended to it

from prometheus_client import Counter
c = Counter('my_failures', 'Description of counter')
c.inc()     # Increment by 1
c.inc(1.6)  # Increment by given value

Gauge

This metric type can go up or down — suitable for setting info that constantly increases or decreases, like requests/s

from prometheus_client import Gauge
g = Gauge('my_inprogress_requests', 'Description of gauge')
g.inc()      # Increment by 1
g.dec(10)    # Decrement by given value
g.set(4.2)   # Set to a given value

Summary

Summary is used to track a task's latency, how long it took to complete.

from prometheus_client import Summary
s = Summary('request_latency_seconds', 'Description of summary')
s.observe(4.7)    # Observe 4.7 (seconds in this case)

Histogram

This metric type tracks the size and count of events. For example, we want to find the percentage or count of requests to the Server that take more than 1s to respond.

from prometheus_client import Histogram
h = Histogram('request_latency_seconds', 'Description of histogram')
h.observe(4.7)    # Observe 4.7 (seconds in this case)

Info

This metric type is a key-value type, usually used to store info about a target

from prometheus_client import Info
i = Info('my_build_version', 'Description of info')
i.info({'version': '1.2.3', 'buildhost': 'foo@bar'})

Enum

This metric type is used to track a service or task's state

from prometheus_client import Enum
e = Enum('my_task_state', 'Description of enum',
        states=['starting', 'running', 'stopped'])
e.state('running')

Hands-on

Now that we understand Prometheus's data types, let's practice monitoring some info.

What we'll build: Monitor the number of players currently online via a ready-made API, running a web server on port 8192 to expose metrics for the Prometheus server. The Prometheus server will get configured to pull data from this web server. We'll use Grafana to visualize the data from the Prometheus Server.

Code

Import a few needed libraries

import requests, time
from prometheus_client import start_http_server, Gauge

Declare global variables — since we're monitoring a player count that increases and decreases, we choose the Gauge data type

keys = ["sbSdp7SUm7eDk5ey326bztEvMbjTAasdsIByF", "vfKIyG9GZDBqZE0I3D555555kzIftu6mIjCD", "vLLifd1524ORZYJyZQPHEpbfUai7878zQA"]

total_player = Gauge('total_players', 'All players in 3 servers')
pvp_player = Gauge('pvp_players', 'Number of players in PVP server')
pve_player = Gauge('pve_players', 'Number of players in PVE server')
vanilla_player = Gauge('vanilla_players', 'Number of players in Vanilla server')

Add a Server object

class Server:
    def __init__(self, name, player_online, max_players):
        self.name = name
        self.player_online = player_online
        self.max_players = max_players

Write a function to pull info from the API, setting the data fetched from the API into the player variables declared earlier

def get_server_info():

    data = requests.get("https://unturned-servers.net/api/?object=servers&element=detail&key=" + keys[0]).json()
    serverPVE = Server(data["name"], data["players"], data["maxplayers"])
    pve_player.set(serverPVE.player_online)

    data = requests.get("https://unturned-servers.net/api/?object=servers&element=detail&key=" + keys[1]).json()
    serverPVP = Server(data["name"], data["players"], data["maxplayers"])
    pvp_player.set(serverPVP.player_online)

    data = requests.get("https://unturned-servers.net/api/?object=servers&element=detail&key=" + keys[2]).json()
    serverVanilla = Server(data["name"], data["players"], data["maxplayers"])
    vanilla_player.set(serverVanilla.player_online)

    total_player.set(serverPVE.player_online + serverPVP.player_online + serverVanilla.player_online)

Start an http server on port 8192, and add a while loop to continuously update the new player count every 1 minute

def main():
    start_http_server(8192)
    while True:
        get_server_info()
        time.sleep(60)

main()

The full code has the following content:

import requests, time
from prometheus_client import start_http_server, Gauge

keys = ["sbSdp7SUm7eDk5ey326bztEvMbjTA15IByF", "vfKIyG9GZDBqZE0I3DwGoSLkzIftu6mIjCD", "vLLifd1524ORZYJyZQPHEpbfUaigppczQA"]

total_player = Gauge('total_players', 'All players in 3 servers')
pvp_player = Gauge('pvp_players', 'Number of players in PVP server')
pve_player = Gauge('pve_players', 'Number of players in PVE server')
vanilla_player = Gauge('vanilla_players', 'Number of players in Vanilla server')
class Server:
    def __init__(self, name, player_online, max_players):
        self.name = name
        self.player_online = player_online
        self.max_players = max_players

def get_server_info():

    data = requests.get("https://unturned-servers.net/api/?object=servers&element=detail&key=" + keys[0]).json()
    serverPVE = Server(data["name"], data["players"], data["maxplayers"])
    pve_player.set(serverPVE.player_online)

    data = requests.get("https://unturned-servers.net/api/?object=servers&element=detail&key=" + keys[1]).json()
    serverPVP = Server(data["name"], data["players"], data["maxplayers"])
    pvp_player.set(serverPVP.player_online)

    data = requests.get("https://unturned-servers.net/api/?object=servers&element=detail&key=" + keys[2]).json()
    serverVanilla = Server(data["name"], data["players"], data["maxplayers"])
    vanilla_player.set(serverVanilla.player_online)

    total_player.set(serverPVE.player_online + serverPVP.player_online + serverVanilla.player_online)

def main():
    start_http_server(8192)
    while True:
        get_server_info()
        time.sleep(60)

main()

Configuring Prometheus

In the prometheus.yml config file, we add a job to scrape data from the web server we ran, as shown.

Then we need to restart the Prometheus service to pick up the new job. You can check whether Prometheus has scraped the data by going to the dashboard, usually running on port 9090.

Charting on Grafana

Create a new dashboard if you don't have one yet, then choose Add Panel in the top-right corner

Next, fill in the metric you want to visualize, as shown:

Apply it to an existing Dashboard and it looks decent enough

Wrapping up

This post walked you through writing a python tool that exposes whatever info you want to monitor through Prometheus. Hope this post helps you out at work in some way.

References

github.com/prometheus/client_python

If you're running into technical challenges or need help with systems, DevOps tools, I'm confident I can help. Get in touch at hoangviet.io.vn

Thank you all! Have a nice day!