Health and Monitoring¶

Platform Health¶

To monitor the health of the platform from an external service, you have the option to enable a healthcheck route on one or more Portals. This route allows you to display a global status of the platform. It is accessible via the URL:

https://your-portal-url/api/healthcheck

Example of the JSON response:

{
    "status": "OK",
    "version": "2.14.4",
    "services": {
        "api":
        {
            "status": "OK",
            "version": "2.15.2",
            "services":
            {
                "db":
                {
                    "status": "OK"
                },
                "provision-api":
                {
                    "status": "OK",
                    "version": "2.7.0"
                },
                "provision-relay-api":
                {
                    "status": "OK",
                    "version": "1.3.0"
                },
                "container-providers": [
                    {
                        "status": "OK",
                        "name": "Reemo SWARM",
                        "type": "SWARM"
                    }
                ],
                "ws-relays": [
                    {
                        "status": "OK",
                        "name": "Global - WS_SWARM",
                        "type": "WS_SWARM"
                    }
                ]
            }
        },
        "signal": {
            "status": "OK",
            "version": "2.6.1"
        }
    }
}

Enabling the route¶

To enable this route on the Portal, you can use the options:

HEALTHCHECK_ENABLE: “true”
HEALTHCHECK_RESTRICT_IP: “192.168.1.1,192.168.10.0/24”

And to enable this route on the Admin Portal:

HEALTHCHECK_PORTALADMIN_ENABLE: “true”
HEALTHCHECK_PORTALADMIN_RESTRICT_IP: “192.168.1.1,192.168.10.0/24”

Prometheus Format¶

If you wish to obtain this information in Prometheus format, it is accessible on the route https://your-portal-url/api/healthcheck/prometheus

# HELP app_service_status Services status (1 = up, 0 = down)
# TYPE app_service_status gauge
# HELP app_service_items Services items numbers
# TYPE app_service_items gauge

app_service_status{service="db"} 1
app_service_status{service="provision-api"} 0
app_service_status{service="provision-relay-api"} 0
app_service_status{service="Mon provider - SWARM", parent="container-providers", type="SWARM"} 0
app_service_status{service="Mon websocket relay 1 - WS_SWARM", parent="ws-relays", type="WS_SWARM"} 0
app_service_status{service="Mon websocket relay 1 - Traefik status", parent="ws-relays", type="WS_SWARM"} 0

app_service_status: Indicates the status of each monitored service. A value of 1 means the service is operational. A value of 0 means it is unavailable.

Monitoring Example¶

Here is an example of setting up platform health monitoring using Nagios

Prerequisites¶

On the Nagios server: have the curl and jq commands available

Command¶

define command {
    command_name    reemo_healthcheck
    command_line    /usr/lib/nagios/plugins/reemo_healthcheck $ARG1$
}

Script¶

#!/bin/bash

# Check if an argument (URL) is provided
if [ -z "$1" ]; then
    echo "CRITICAL: No URL provided. Usage: $0 <URL>"
    exit 2
fi

HEALTHCHECK_URL="$1"

# Fetch JSON response
response=$(curl -s "$HEALTHCHECK_URL/api/healthcheck")

# Check if the request was successful
if [ $? -ne 0 ] || [ -z "$response" ]; then
    echo "CRITICAL: Unable to access the health check page ($HEALTHCHECK_URL)"
    exit 2
fi

# Extract service statuses using jq
status=$(echo "$response" | jq -r '.status')
errors=$(echo "$response" | jq -r '.. | objects | select(.status? and .status != "OK")')

# Check if the global status is OK
if [ "$status" != "OK" ]; then
    echo "CRITICAL: Global health check status is $status"
    exit 2
fi

# Check if any sub-services have issues
if [ -n "$errors" ]; then
    echo "WARNING: Some services are not OK:"
    echo "$errors"
    exit 1
fi

# Everything is OK
echo "OK: All services are operational"
exit 0

Service¶

define service{
    host_name                       < HOSTNAME >
    use                             generic-service         ; Name of service template to use
    service_description             Reemo Healthcheck
    check_command                   reemo_healthcheck!https://< URL >
}

Prometheus Monitoring¶

It is possible to enable Prometheus on the Traefik entry points; several options are available:

Enable Prometheus¶

To enable Prometheus, you can add the option TRAEFIK_PROMETHEUS_ENABLE: true in the inventory file.

Port¶

By default, the Prometheus page will be accessible on the same port as the Portal. Use the option TRAEFIK_PROMETHEUS_PORT: “<port number>” to change it.

URL¶

By default, the Prometheus page will be accessible on the Portal URL with the /metrics path. It is possible to specify one or more dedicated URLs:

TRAEFIK_PROMETHEUS_URLS:
    - "metrics1.domain.tld"
    - "metrics2.domain.tld"

Note

In the case of a 3-node cluster, it is recommended to set 3 URLs so that Prometheus can retrieve the metrics from all nodes.

IP Filtering¶

To restrict access to specific IP addresses, you can use the TRAEFIK_PROMETHEUS_RESTRICT_IP option.

Example:

TRAEFIK_PROMETHEUS_RESTRICT_IP: "1.1.1.1,2.2.2.2"