Health and Monitoring¶

Platform Health¶

To monitor the health of the platform from an external service, you have the option to enable a healthcheck route on one or more Portals. This route allows you to display a global status of the platform. It is accessible via the URL:

https://your-portal-url/api/healthcheck

Example of the JSON response:

{
    "status": "OK",
    "internalServicesStatus": "OK",
    "globalStatus": "WARNING",
    "certificatesStatus": "OK",
    "version": "2.15.0",
    "globalVersion": "2.15.0",
    "services": {
        "api": {
            "status": "OK",
            "internalServicesStatus": "OK",
            "version": "3.26.1",
            "services": {
                "db": { "status": "OK" },
                "provision-api": { "status": "OK", "version": "3.10.4" },
                "provision-relay-api": { "status": "OK", "version": "1.9.0" },
                "container-providers": [
                    { "status": "OK", "name": "Swarm cluster EU-West - SWARM", "type": "SWARM" }
                ],
                "ws-relays": [
                    { "status": "OK", "name": "Websocket relay EU-West - WS_SWARM", "type": "WS_SWARM" },
                    { "status": "OK", "name": "Websocket relay EU-West - Traefik status", "type": "WS_SWARM" }
                ]
            },
            "relayDiagnostics": [
                {
                    "name": "Websocket relay EU-West",
                    "type": "WS_SWARM",
                    "status": "ERROR",
                    "proxy": {
                        "reachable": true,
                        "url": "http://relay.example.com:8404/stats;json",
                        "backends": [
                            {
                                "name": "relay_swarm_back",
                                "status": "UP",
                                "active": 2,
                                "backup": 0,
                                "sessions": { "current": 2, "max": 4, "limit": 26209, "queue": 0 },
                                "errors": { "connection": 0, "response": 0, "retries": 0, "redispatches": 0 }
                            }
                        ]
                    },
                    "cluster": { "reachable": true, "error": null, "members": 3 },
                    "nodes": [
                        {
                            "address": "10.0.0.13",
                            "hostname": "node2",
                            "state": "NGINX_DOWN",
                            "detail": "no answer on the node address",
                            "role": "manager",
                            "proxyExpected": true,
                            "cluster": { "member": true, "state": "down", "availability": "active", "reachability": "unreachable" },
                            "proxy": [
                                { "backend": "relay_swarm_back", "port": 8375, "status": "DOWN", "check": "L4TOUT", "checkCode": null, "lastCheck": "", "failures": 3, "flaps": 2, "downtime": 154, "since": 154 }
                            ],
                            "engine": { "ping": null, "version": null, "swarm": null, "manager": null, "clusterId": null }
                        }
                    ]
                }
            ],
            "providerDiagnostics": [],
            "ndbDiagnostics": null
        },
        "signals": [
            { "status": "OK", "url": "https://signal.example.com", "certificateExpiration": "2027-04-14 17:29:00", "error": null }
        ]
    },
    "license": { "expiration": "2028-01-01 01:00:00" }
}

The response above is abridged: each list holds one entry per object of that kind.

Reading the aggregates¶

The response opens with four aggregates. An external supervision usually watches the last two:

status — the API and its database only.
internalServicesStatus — every internal service except the database.
globalStatus — the platform as a whole: ERROR when the API, one of its internal services or every signal server is down, WARNING when only some signal servers, a provider or the database cluster are degraded.
certificatesStatus — the TLS certificates, against the instance’s warning and error thresholds.

The route always answers 200, including during a total outage: read the verdict in the body rather than the HTTP code.

Node diagnostics¶

relayDiagnostics and providerDiagnostics report each cluster machine by machine. Each entry names the cluster, its aggregated status (OK, WARNING, ERROR or UNAVAILABLE), the load balancer it was read from, and one object per node.

Fields available for each node:

Field	Content
`state`	`OK`, `NGINX_DOWN`, `DOCKER_DOWN`, `SWARM_INACTIVE`, `NOT_IN_CLUSTER`, `NOT_BEHIND_PROXY` for a Swarm cluster; `OK`, `PROXY_DOWN`, `APISERVER_DOWN`, `NOT_READY`, `NOT_IN_CLUSTER`, `NOT_BEHIND_PROXY` for a Kubernetes cluster.
`detail`	One sentence naming what led to that state, or `null`.
`role`	`manager`, `worker` or `control-plane`.
`proxyExpected`	Whether the node’s role requires the load balancer to serve it. A node in a state other than `OK` turns the cluster verdict to `ERROR` only when this field is `true`.
`proxy`	One entry per load balancer backend serving this node, with the port and the state HAProxy gives it.
`engine` / `apiServer`	What the node itself answered: engine version and swarm state for Swarm, api server liveness for Kubernetes.

ndbDiagnostics carries the same kind of detail for a MySQL NDB Cluster: its verdict (ok, warn, crit, stale or unknown), the coded reasons behind it and the node groups. It is null on an instance with no cluster supervision enabled.

Note

These three fields hold host names and addresses of the infrastructure. They are part of the response on the anonymous route as well: restrict that route by IP.

Enabling the route¶

To enable this route on the Portal, you can use the options:

HEALTHCHECK_ENABLE: “true”
HEALTHCHECK_RESTRICT_IP: “192.168.1.1,192.168.10.0/24”

And to enable this route on the Admin Portal:

HEALTHCHECK_PORTALADMIN_ENABLE: “true”
HEALTHCHECK_PORTALADMIN_RESTRICT_IP: “192.168.1.1,192.168.10.0/24”

Prometheus Format¶

If you wish to obtain this information in Prometheus format, it is accessible on the route https://your-portal-url/api/healthcheck/prometheus

# HELP app_status API and database status (1 = up, 0 = down)
# TYPE app_status gauge
app_status 1

# HELP app_internal_services_status Internal services status (1 = all up, 0 = at least one down)
# TYPE app_internal_services_status gauge
app_internal_services_status 0

# HELP app_service_status Services status (1 = up, 0 = down)
# TYPE app_service_status gauge
app_service_status{service="db"} 1
app_service_status{service="provision-api"} 0
app_service_status{service="provision-relay-api"} 0
app_service_status{service="My provider - SWARM", parent="container-providers", type="SWARM"} 0
app_service_status{service="My websocket relay 1 - WS_SWARM", parent="ws-relays", type="WS_SWARM"} 0
app_service_status{service="My websocket relay 1 - Traefik status", parent="ws-relays", type="WS_SWARM"} 0

# HELP app_service_items Services items numbers
# TYPE app_service_items gauge
app_service_items{service="My provider - SWARM", parent="container-providers", type="SWARM"} 12

app_status: The API and its database. A value of 1 means both answer.
app_internal_services_status: The internal services, the database excluded. A value of 0 means at least one of them is down.
app_service_status: Indicates the status of each monitored service. A value of 1 means the service is operational. A value of 0 means it is unavailable.

An instance whose database runs on a MySQL NDB Cluster also exposes the cluster itself:

# HELP app_ndb_status NDB cluster verdict (2 = ok, 1 = warn, 0 = crit, stale, unknown or unreachable)
# TYPE app_ndb_status gauge
app_ndb_status 2

# HELP app_ndb_min_redundancy Fewest live nodes in any node group; 1 means the next incident is an outage
# TYPE app_ndb_min_redundancy gauge
app_ndb_min_redundancy 2

# HELP app_ndb_data_nodes Data nodes the cluster reports
# TYPE app_ndb_data_nodes gauge
app_ndb_data_nodes{state="live"} 4
app_ndb_data_nodes{state="total"} 4

The same route also carries app_ndb_data_memory_max_pct, app_ndb_redo_max_pct and app_ndb_snapshot_age_seconds. A snapshot that stops ageing means the supervision sidecar no longer collects, so app_ndb_status drops to 0.

Warning

The node-by-node diagnostic of the relays and container providers is not exposed in Prometheus format. app_service_status reports the cluster’s entry point only. Read the JSON route for the per-node detail.

Monitoring Example¶

Here is an example of setting up platform health monitoring using Nagios

Prerequisites¶

On the Nagios server: have the curl and jq commands available

Command¶

define command {
    command_name    reemo_healthcheck
    command_line    /usr/lib/nagios/plugins/reemo_healthcheck $ARG1$
}

Script¶

#!/bin/bash

# Check if an argument (URL) is provided
if [ -z "$1" ]; then
    echo "CRITICAL: No URL provided. Usage: $0 <URL>"
    exit 2
fi

HEALTHCHECK_URL="$1"

# Fetch JSON response
response=$(curl -s "$HEALTHCHECK_URL/api/healthcheck")

# Check if the request was successful
if [ $? -ne 0 ] || [ -z "$response" ]; then
    echo "CRITICAL: Unable to access the health check page ($HEALTHCHECK_URL)"
    exit 2
fi

# Extract service statuses using jq
status=$(echo "$response" | jq -r '.status')
errors=$(echo "$response" | jq -r '.. | objects | select(.status? and .status != "OK")')

# Check if the global status is OK
if [ "$status" != "OK" ]; then
    echo "CRITICAL: Global health check status is $status"
    exit 2
fi

# Check if any sub-services have issues
if [ -n "$errors" ]; then
    echo "WARNING: Some services are not OK:"
    echo "$errors"
    exit 1
fi

# Everything is OK
echo "OK: All services are operational"
exit 0

Service¶

define service{
    host_name                       < HOSTNAME >
    use                             generic-service         ; Name of service template to use
    service_description             Reemo Healthcheck
    check_command                   reemo_healthcheck!https://< URL >
}

Prometheus Monitoring¶

It is possible to enable Prometheus on the Traefik entry points; several options are available:

Enable Prometheus¶

To enable Prometheus, you can add the option TRAEFIK_PROMETHEUS_ENABLE: true in the inventory file.

Port¶

By default, the Prometheus page will be accessible on the same port as the Portal. Use the option TRAEFIK_PROMETHEUS_PORT: “<port number>” to change it.

URL¶

By default, the Prometheus page will be accessible on the Portal URL with the /metrics path. It is possible to specify one or more dedicated URLs:

TRAEFIK_PROMETHEUS_URLS:
    - "metrics1.domain.tld"
    - "metrics2.domain.tld"

Note

In the case of a 3-node cluster, it is recommended to set 3 URLs so that Prometheus can retrieve the metrics from all nodes.

IP Filtering¶

To restrict access to specific IP addresses, you can use the TRAEFIK_PROMETHEUS_RESTRICT_IP option.

Example:

TRAEFIK_PROMETHEUS_RESTRICT_IP: "1.1.1.1,2.2.2.2"

Datadog Monitoring¶

To send platform logs to Datadog, use the following options:

DATADOG_ENABLED: true
DATADOG_API_KEY: "<your_datadog_api_key>"
DATADOG_SITE: "datadoghq.eu"
DATADOG_SYSLOG_PREFIX: ""

DATADOG_ENABLED: enables the Datadog integration.
DATADOG_API_KEY: your Datadog API key, to replace with your own.
DATADOG_SITE: the Datadog site associated with your account (e.g. datadoghq.eu), provided by Datadog.
DATADOG_SYSLOG_PREFIX: optional prefix added to each forwarded log line.