Prometheus Exporter - VR Healthcheck results

[summer-abacus]

From time to time we have the issue, that the health check of the virtual routers is failing. An alert is triggered and shows up

Name: Domain Router
Type: 9
Description: Health checks failed: 1 failing checks on router r-xxxx-VM / id

Is it somehow possible to add the result of the virtual router healthcheck to the metrics of the prometheus exporter?

[DaanHoogland]

certainly but it will require some (java) coding.

[summer-abacus]

Thank you for your answer. Something like the textfile collector would be a nice feature. I'll try to use the prometheus node exporter, the textfile collector and cmk to get those metric into our monitoring.

[DaanHoogland]

not sure I understand, but looking forward to it. sounds like it could be a more generic enrichment method of the prometheus exporter?

[summer-abacus]

It was easier to implement than expected. What I did to get the metrics into prometheus:

• Installed the prometheus node exporter on the cloudstack management host
• Configured the textfile collector (https://github.com/prometheus/node_exporter#textfile-collector)
• Wrote a small script to generate the metrics
• Execute the script via cronjob

Script:

#!/bin/bash

/usr/bin/cmk set output json

JSON=$(/usr/bin/cmk list routers listall=true filter=name,hostname,healthchecksfailed,)

COUNT=$(echo "$JSON" | jq '.count')

echo "# HELP router_count Total number of routers"
echo "# TYPE router_count gauge"
echo "router_count $COUNT"
echo ""
echo "# HELP router_healthchecks_failed Indicates if health checks failed for a router (0 = false, 1 = true)"
echo "# TYPE router_healthchecks_failed gauge"

echo "$JSON" | jq -c '.router[]' | while read -r router; do
  HEALTHCHECKS_FAILED=$(echo "$router" | jq '.healthchecksfailed')
  HOSTNAME=$(echo "$router" | jq -r '.hostname')
  NAME=$(echo "$router" | jq -r '.name')

  # Convert healthchecksfailed to 0 or 1
  if [ "$HEALTHCHECKS_FAILED" = "false" ]; then
    HEALTHCHECKS_FAILED=0
  else
    HEALTHCHECKS_FAILED=1
  fi

  echo "router_healthchecks_failed{hostname=\"$HOSTNAME\", name=\"$NAME\"} $HEALTHCHECKS_FAILED"
done

Cronjob:

* * * * * /opt/scripts/router_health_to_prom.sh > /opt/node_exporter/drop_zone/router.prom 2>&1

In prometheus metrics, it looks like this:

# HELP router_count Total number of routers
# TYPE router_count gauge
router_count 5
# HELP router_healthchecks_failed Indicates if health checks failed for a router (0 = false, 1 = true)
# TYPE router_healthchecks_failed gauge
router_healthchecks_failed{hostname="srv1-213",name="r-1063-VM"} 0
router_healthchecks_failed{hostname="srv1-213",name="r-981-VM"} 0
router_healthchecks_failed{hostname="srv1-214",name="r-1335-VM"} 0
router_healthchecks_failed{hostname="srv1-214",name="r-8-VM"} 0
router_healthchecks_failed{hostname="srv1-214",name="r-982-VM"} 0

This allwos me to use alertmanager to trigger notifications via teams, mail etc.