.. _monitoring-en-checks:

Monitoring Checks¶

This page details, component by component, what must be monitored and how to check it. To learn which checks apply to your installation, start with Monitoring an all-in-one installation or Monitoring a Split Installation.

Monitoring Common to All Servers¶

System¶

CPU usage, memory, load average
Disk space, particularly:
- /var/lib/docker (images and containers);
- /opt/<INSTANCE_NAME> (data, including the database on infra_manager / api_manager);
- /opt/traefik (traefik configuration and certificates).
Time synchronization (NTP/chrony): a clock skew breaks TLS validation and ephemeral TURN credentials.
Docker daemon active: systemctl is-active docker.

Docker Swarm Cluster¶

On a manager of each cluster:

# All nodes must be Ready / Active
docker node ls

# Each service must have REPLICAS = x/x (e.g., 2/2)
docker service ls

Recommended probes:

Nodes: alert if a node is not Ready or not Active.
Services: alert if the number of replicas running is less than expected:
```
docker service ls --format '{{.Name}} {{.Replicas}}' \
  | grep -v -E '_(db|logapidb|mysqlbackup) ' \
  | awk '{split($2,r,"/"); if (r[1] != r[2]) {print "KO " $0; e=1}} END {exit e}'
```
Three services are excluded from this check, as it is normal for them to be 0/1:
- <INSTANCE_NAME>_db and <INSTANCE_NAME>_logapidb: one-shot tasks for creating and updating databases. They run on each deployment, then stop (restart_config: condition: none). Instead, verify that their last execution completed successfully (state Complete):
```
docker service ps reemo_db --format '{{.CurrentState}} {{.Error}}' | head -1
```
- <INSTANCE_NAME>_mysqlbackup: legacy backup system, replaced by <INSTANCE_NAME>_backup. It does not need monitoring.
Blocked updates: a state of paused or rollback_completed signals a failed deployment:
```
docker service inspect reemo_api --format '{{json .UpdateStatus}}'
```

Reemo containers include a Docker healthcheck (internal listening port). An unhealthy container is automatically restarted by Swarm. Repeated restarts should be monitored:

docker ps --filter health=unhealthy
docker service ps reemo_api --no-trunc   # task history and errors

Logs¶

All services log to syslog (unixgram:///dev/log) with a <INSTANCE_NAME>_<service> tag (e.g., reemo_api, reemo_traefik, reemo_turn1). You can:

centralize these logs (rsyslog, ELK, Graylog…);
or enable forwarding to Datadog with DATADOG_ENABLED=true.

Traefik¶

Traefik is the HTTPS entry point on all profiles except provisioning.

Service traefik: replicas x/x (one per manager).
Listening ports on the host (mode: host): verify TCP from outside.
Prometheus metrics (optional, profiles infra_manager / portal_manager): TRAEFIK_PROMETHEUS_ENABLE=true exposes /metrics on TRAEFIK_PROMETHEUS_PORT (443 by default). Restrict access with TRAEFIK_PROMETHEUS_RESTRICT_IP.
```
curl -s https://portail.example.com/metrics | head
```

Inbound HTTPS Certificates¶

Each public URL must have a probe that verifies:

the expiration date (warning recommended at 30 days, critical at 7 days);
the complete and valid certificate chain;
name matching (CN/SAN) with the URL.

Example check:

echo | openssl s_client -connect portail.example.com:443 \
     -servername portail.example.com 2>/dev/null \
  | openssl x509 -noout -subject -enddate

# or with the standard Nagios plugin
check_http -H portail.example.com -S --sni -C 30,7

Entry points to monitor based on configuration:

Entry Point	Default Port	Condition
Portal (`PORTAL_URL`)	443 (`TRAEFIK_SSL_PORT`)	Always (`infra_manager` / `portal_manager`)
Signal (`SIGNAL_URL`)	8443 (`TRAEFIK_SSL_SIGNALPORT`)	Always. Can be shared with the portal if `TRAEFIK_SSL_SIGNALPORT` = `TRAEFIK_SSL_PORT`.
Admin portal (`PORTALADMIN_URL`)	443, or `PORTALADMIN_URL_PORT`	`PORTALADMIN_URL` set
Workstation	8445 (`WORKSTATION_INIT_PORTALPORT`)	`WORKSTATION_ENABLED=true`
Appliance portal	8444 (`APPLIANCE_INIT_PORTALPORT`)	`APPLIANCE_ENABLED=true`
Credential portal	8446 (`CREDENTIAL_PORTAL_PORT`)	`CREDENTIAL_ENABLED=true`
Wake-on-LAN API (`WOLAPI_URL`)	443	`WOLAPI_ENABLE=true`
Internal API (`API_INTERNAL_URL`)	443	`API_INTERNAL_ENABLED=true` (mTLS)
API (`api_manager` profile)	443	Separate architecture (mTLS, see below)
Relay WebSocket	443 (`RELAYWS_TRAEFIK_SSL_PORT`)	`relayws_manager` profile

Note

Workstation, Appliance portal, and Credential portal ports are routed in TCP passthrough (HostSNI(`*`)). The certificate presented is that of the service itself, not traefik’s. Monitor it the same way.

Note

If you use Let’s Encrypt (traefik’s ACME resolver), renewal is automatic. The expiration probe remains essential to detect a failed renewal (port 443 filtered, DNS changed…).

Portal and Admin Portal¶

Portal Healthcheck¶

When HEALTHCHECK_ENABLE=true, the portal exposes the /api/healthcheck route. It returns the global platform health in JSON format: API, database, provision-api, provision-relay-api, container providers, WebSocket relays, and signal. Access is restricted to addresses in HEALTHCHECK_RESTRICT_IP: add your monitoring server’s IP there.

curl -fsS https://portail.example.com/api/healthcheck | jq .

Probe: HTTP 200, global status field at OK, and no sub-service in another state. The Nagios script reemo_healthcheck performs this check. The same information is available in Prometheus format on the /api/healthcheck/prometheus route.

Admin Portal Healthcheck¶

When PORTALADMIN_URL is set and HEALTHCHECK_PORTALADMIN_ENABLE=true:

curl -fsS https://admin.example.com/api/healthcheck
# if a dedicated port is configured
curl -fsS https://admin.example.com:${PORTALADMIN_URL_PORT}/api/healthcheck

Access is restricted by HEALTHCHECK_PORTALADMIN_RESTRICT_IP.

Signal¶

Signal is a WebSocket service. A standard HTTPS request (without WebSocket upgrade) receives an HTTP 426 status code with the message Upgrade Required: this response indicates the service is working. The check goes through traefik, on the signal port (8443 by default):

curl -sS -w '\nHTTP %{http_code}\n' https://signal.example.com:8443/
# expected:
# Upgrade Required
# HTTP 426

Probe: HTTP 426 status and the presence of Upgrade Required in the response. Checking the text verifies the response comes from signal, not an error page or maintenance page. If you have multiple signal instances (multiple SIGNAL_URL_TRAEFIK servers), monitor each URL.

API¶

All-in-One Architecture (`infra_manager`)¶

The API is not exposed by traefik (traefik.enable=false). The portal calls it directly on the internal Swarm network (https://reemo_api:8040). Monitoring relies on:

the status of the reemo_api service (replicas x/x, healthy containers);
the portal healthcheck, which fails if the portal is unusable.

Separate Architecture (`api_manager` + `portal_manager`)¶

In this mode, the API runs on a separate cluster. The portal accesses it via an haproxy proxy named reemo_api (deployed on portal_manager). This proxy sends requests to API_IP addresses, port 443 (PORTAL_HA_PORT).

The traefik of the api_manager requires a client certificate signed by the internal PKI (mTLS, RequireAndVerifyClientCert). The monitoring server has no internal PKI certificate: it cannot query the API directly, and this is not necessary.

To monitor:

Portal healthcheck: this is the functional check of the API. The /api/healthcheck request traverses the entire chain: portal → reemo_api proxy → API’s traefik (mTLS) → API. The JSON returned contains the API’s status (services.api.status) and its dependencies (database, provision-api, provision-relay-api). See Portal Healthcheck.
Port 443 open on each ``API_IP`` and mTLS required: without a client certificate, the TLS connection must be refused by the API. If it succeeds with an HTTP response, the API is exposed without mutual authentication and an alert must be raised. See the probe check_reemo_api_port.
```
curl -sk -o /dev/null https://<API_IP>/
# expected: TLS failure like
# "tlsv13 alert certificate required" or "alert bad certificate"
```
API server certificate: the expiration date can be read without a client certificate (probe check_reemo_cert with VERIF_CHAINE=0, as the internal CA is unknown to the monitoring server).
haproxy proxy on the portal side: service reemo_api on portal_manager (replicas x/x), via check_reemo_swarm_services.
Services on the ``api_manager``: reemo_api, reemo_proapi, reemo_prorelayapi, reemo_logapi, reemo_credentialapi, cron jobs, etc. Monitor them via check_reemo_swarm_services (NRPE) on that cluster.

Note

The portal → API flow (port 443 to each API_IP) must be open from portal_manager machines. The port probe, run from the monitoring server, does not test this specific flow. On a portal healthcheck alert, test it from a portal machine (see Alert: API (separate architecture)).

Workstation¶

Condition: WORKSTATION_ENABLED=true. The Workstation service runs on the portal profile (infra_manager or portal_manager).

Public entry point: port 8445 (WORKSTATION_INIT_PORTALPORT), in TCP passthrough to the reemo_workstation service.
- TCP and TLS verification, with the Workstation certificate’s expiration;
```
echo | openssl s_client -connect portail.example.com:8445 2>/dev/null \
  | openssl x509 -noout -subject -enddate
```
Service: reemo_workstation, replicas x/x and healthy containers.
Dependency on the API:
- in all-in-one mode, Workstation calls https://reemo_api:8040 on the internal network;
- in separate architecture, Workstation calls the API via the portal’s reemo_api proxy, then API_IP:443 in mTLS. The checks in Separate Architecture (api_manager + portal_manager) therefore also cover Workstation. If WORKSTATION_API_URL is customized, monitor that URL.
Workstation CA: if the Workstation CA is generated by the role (WORKSTATION_INITCA_ENABLED=true), monitor its certificate’s expiration and that of the certificates it issues.

Appliances¶

Condition: APPLIANCE_ENABLED=true.

Appliances connect to the platform via the appliance portal (APPLIANCE_INIT_PORTALURL, which is PORTAL_URL:8444 by default). Two services are deployed:

Service	Profile	Exposure
`reemo_applianceportal`	`infra_manager` / `portal_manager`	Public port 8444 (`APPLIANCE_INIT_PORTALPORT`), in TCP passthrough via traefik to the service’s port 8443.
`reemo_applianceapi`	`infra_manager` / `api_manager`	Not exposed (internal Swarm network, port 8160).

To monitor:

Port 8444 open on the portal’s public IPs:

check_tcp -H portail.example.com -p 8444 -w 2 -c 5

Appliance portal certificate: in passthrough, the certificate presented is that of the service, issued by the internal PKI. Check only its expiration (VERIF_CHAINE=0):
```
check_reemo_cert portail.example.com 8444 portail.example.com 30 7 0
```
Services reemo_applianceportal (portal cluster) and reemo_applianceapi (API cluster): replicas x/x, via check_reemo_swarm_services.
Network flow: port 8444 must be reachable from the sites where appliances are installed, not just from the monitoring server.

Credential Portal¶

Condition: CREDENTIAL_ENABLED=true and CREDENTIALPORTAL_ENABLED=true.

Service	Profile	Exposure
`reemo_credentialportal`	Portal (traefik `infra_manager` / `portal_manager`)	Public port 8446 (`CREDENTIAL_PORTAL_PORT`), in TCP passthrough via traefik to the service’s port 8443.
`reemo_credentialapi`	`infra_manager` / `api_manager`	Not exposed (internal Swarm network, port 8190).
`reemo_vault` / `reemo_vault<N>`	`infra_manager` / `api_manager`	Not exposed. See Vault.

To monitor:

Port 8446 open:

check_tcp -H portail.example.com -p 8446 -w 2 -c 5

Credential portal certificate (passthrough, internal PKI): check only its expiration:
```
check_reemo_cert portail.example.com 8446 portail.example.com 30 7 0
```
IP restriction: if CREDENTIAL_PORTAL_RESTRICT_IP is set, traefik cuts the TCP connection for any non-authorized IP. Add your monitoring server’s IP to this list, otherwise the two probes above fail permanently:
```
CREDENTIAL_PORTAL_RESTRICT_IP:
  - name: supervision
    ip: 192.0.2.50
  - name: siteA
    ip: 198.51.100.0/24
```
Services reemo_credentialportal and reemo_credentialapi: replicas x/x.
Vault unsealed: reemo_credentialapi does not work while Vault is sealed. The check_reemo_vault probe is essential as soon as the credential portal is enabled.

TURN¶

Condition: TURN_ENABLED=true.

The TURN server runs:

on the portal profile (infra_manager / portal_manager) in classic deployment;
on dedicated turn_manager machines when this group exists in the inventory.

Checks:

Services reemo_turn1 (and reemo_turn2 if TURN2_NODE is set): replicas 1/1.
Public ports: TURN_PORT (58200 by default), in TCP and UDP, on TURN1_IP / TURN2_IP. These ports are published in mode: host: check each IP individually from outside.
```
nc -vz  <TURN1_IP> 58200     # TCP
nc -vzu <TURN1_IP> 58200     # UDP (informational, UDP with no guaranteed response)
```

Functional test (recommended): actual allocation with turnutils_uclient (coturn package).

In TURN_AUTH_MODE=static mode: user TURN_USERNAME / password TURN_PASSWORD.
In TURN_AUTH_MODE=secret mode: ephemeral credentials computed from TURN_SECRET.

# secret mode: username = expiration timestamp, password = HMAC-SHA1 base64
u=$(( $(date +%s) + 3600 )):supervision
p=$(printf '%s' "$u" | openssl dgst -sha1 -hmac "$TURN_SECRET" -binary | base64)
turnutils_uclient -y -u "$u" -w "$p" -p 58200 <TURN1_IP>

Secret consistency (turn_manager): the TURN secret is declared in the inventory and must be identical on turn_manager and on api_manager / infra_manager. A mismatch results in TURN authentication failures, which the functional test above detects.
``turn_manager`` profile: traefik is also deployed (ports 80/443, TURN_TRAEFIK_PORT / TURN_TRAEFIK_SSL_PORT). Monitor the traefik service and, if applicable, the exposed certificate.

Note

Media traffic from relayed sessions passes through the TURN server. Also monitor the network bandwidth of TURN machines.

Relay WebSocket¶

Condition: relayws_manager profile (and RELAYS_IP set on the API side).

Traefik: ports 80/443 (RELAYWS_TRAEFIK_PORT / RELAYWS_TRAEFIK_SSL_PORT), with the public certificate’s expiration.
Nginx (API Docker exposed to prorelayapi): port 8443 (RELAYWS_NGINX_PORT), with mandatory client certificate (DN CN=reemo_prorelayapi). Verify:
- TCP reachability from api_manager / infra_manager machines;
- service status: systemctl is-active nginx.
Swarm cluster: docker node ls / docker service ls on the relayws. Relay containers are created on demand by prorelayapi.
On the API side: service reemo_prorelayapi (replicas x/x) and reemo_relayws proxy if deployed.
Portal healthcheck: the provision-relay-api and ws-relays entries in the JSON indicate whether the API can reach the relays.

Provisioning¶

Condition: provision_manager / provisionN_manager profiles.

Nginx: port 8443 (PROVISION_NGINX_PORT), with mandatory client certificate.
- TCP reachability from API machines;
- systemctl is-active nginx;
- nginx server certificate expiration.
Swarm cluster: nodes Ready / Active (managers and workers).
Portal healthcheck: the provision-api and container-providers entries in the JSON indicate whether the API can reach the container providers.
Capacity: CPU, RAM, and disk on workers, which host user containers. Also monitor disk space used by images (docker system df).

Database¶

MariaDB¶

Profiles infra_manager / api_manager with DB_DIALECT=mysql and local database.

Service reemo_mysql (MariaDB): replicas 1/1, healthy.
Disk space for /opt/<INSTANCE_NAME>/db.

SQL ping (from the container):

docker exec $(docker ps -q -f name=reemo_mysql) \
  sh -c 'MYSQL_PWD=$(cat $MYSQL_ROOT_PASSWORD_FILE) mysqladmin ping -u root'

If DB_SSL_REQUIRE=true: the database certificate’s expiration.

NDB Cluster¶

With DB_DIALECT=NDBCLUSTER:

all ndbd, mgmd, and mysqld nodes connected:

docker exec $(docker ps -q -f name=reemo_mysql-mgmd | head -1) ndb_mgm -e show

data and index memory (ndb_mgm -e "all report memory"), with an alert beyond 80%;
option NDBMONITOR_ENABLED=true: dedicated NDB monitoring service (reemo_ndbmonitor).

Vault¶

Condition: CREDENTIAL_ENABLED=true and VAULT_ENABLED=true.

Services reemo_vault (frontend) and reemo_vault<N> (replicas): replicas 1/1 each. Vault is provided by OpenBao (bao command).
Sealed state: after a restart, Vault starts locked. In VAULT_UNSEAL_MODE=manual mode (the default), intervention is required. An alert on sealed=true is essential:
```
# execute for each reemo_vault<N> replica
docker exec <vault_container> bao status -format=json | grep '"sealed"'
```
In raft mode (multiple replicas): verify all members are unsealed and the cluster has a leader.
Dependent services: reemo_credentialapi and, if enabled, reemo_credentialportal.

Backups¶

Condition: BACKUP_ENABLED=true.

Freshness: the most recent backup must be less than 24 hours old (alert beyond 25 hours, critical beyond 48 hours).
Size: alert on a backup abnormally small compared to previous ones.
Integrity: a periodic check can be run with --extra-vars "VERIFYBACKUP=true". Monitor its exit code.
Externalization: verify the copy to remote storage succeeded.
Restoration: regularly test a restoration in a staging environment.

Internal PKI Certificates¶

Internal exchanges (portal → API, traefik → signal, traefik → relayws, prorelayapi → relayws, database in TLS…) are protected by certificates issued by the internal PKI (INITCA_ENABLE). They are stored as Docker secrets (reemo_*_ssl_cert).

Monitor the CA’s expiration date and each service certificate’s expiration. An expired internal certificate causes service outage with no visible browser error.
Recommended method: check the PKI files (LOCAL_PATH directory or PKI git repository) from the administration workstation:
```
for c in pki/*.crt; do
  printf '%s ' "$c"; openssl x509 -in "$c" -noout -enddate
done
```
Traefik’s default certificate (/opt/traefik/config/default.crt) is self-signed and valid for 824 days. It is only used for requests with no matching SNI.

Summary of Flows to Monitor¶

Source	Destination	Port	Condition
Users / monitoring	Portal	443 (+ 80 for redirect)	Always
Users / monitoring	Signal	8443	Always
Users / monitoring	Workstation	8445	`WORKSTATION_ENABLED`
Appliances (customer sites) / monitoring	Appliance portal	8444	`APPLIANCE_ENABLED`
Authorized users / monitoring	Credential portal	8446	`CREDENTIALPORTAL_ENABLED`
Users / monitoring	TURN	58200 TCP/UDP	`TURN_ENABLED`
`portal_manager`	`api_manager` (`API_IP`)	443 (mTLS)	Separate architecture
`api_manager` / `infra_manager`	`relayws_manager`	8443 (mTLS)	Relay WebSocket
`api_manager` / `infra_manager`	`provisionN_manager`	8443 (mTLS)	Provisioning
Nodes in the same cluster	Nodes in the same cluster	2377/tcp, 7946/tcp+udp, 4789/udp (or `SWARM_DATA_PATH_PORT`)	Multi-node clusters

Monitoring Checks¶

Monitoring Common to All Servers¶

System¶

Docker Swarm Cluster¶

Logs¶

Traefik¶

Inbound HTTPS Certificates¶

Portal and Admin Portal¶

Portal Healthcheck¶

Admin Portal Healthcheck¶

Signal¶

API¶

All-in-One Architecture (infra_manager)¶

Separate Architecture (api_manager + portal_manager)¶

Workstation¶

Appliances¶

Credential Portal¶

TURN¶

Relay WebSocket¶

Provisioning¶

Database¶

MariaDB¶

NDB Cluster¶

Vault¶

Backups¶

Internal PKI Certificates¶

Summary of Flows to Monitor¶

All-in-One Architecture (`infra_manager`)¶

Separate Architecture (`api_manager` + `portal_manager`)¶