Alert Troubleshooting¶

This page lists, for each alert, the commands to run to identify the cause. For an initial global diagnosis, start with Quick status check.

This section lists, for each alert, the commands to run to identify the cause. They should be executed as root on a manager of the cluster in question, unless otherwise specified.

Common Toolbox¶

# Global state
docker node ls
docker service ls

# Task history for a service, with the complete error message
docker service ps reemo_portal --no-trunc

# Configuration and status of the latest update
docker service inspect reemo_portal --pretty
docker service inspect reemo_portal --format '{{json .UpdateStatus}}'

# Containers on the local node and Docker healthcheck status
docker ps -a --filter name=reemo_portal
docker inspect --format '{{json .State.Health}}' <CONTAINER_ID>

# Recent events (restarts, healthcheck failures, OOM…)
docker events --since 30m --until 0s --filter type=container

# Node resources
df -h /var/lib/docker /opt
free -m
dmesg -T | grep -i -E 'oom|killed process'

Logs: services log to syslog with the tag <INSTANCE_NAME>_<service>. You must read them on the node running the container (column NODE from docker service ps):

journalctl -t reemo_portal --since "1 hour ago"
# or, depending on rsyslog configuration:
grep reemo_portal /var/log/syslog      # Debian / Ubuntu
grep reemo_portal /var/log/messages    # RedHat / Rocky

# Docker >= 20.10 also keeps a local log cache:
docker logs --tail 200 <CONTAINER_ID>

Common corrective actions:

# Restart a service (recreates its containers)
docker service update --force reemo_portal

# Revert to the previous version after a failed update
docker service rollback reemo_portal

# Redeploy a component using the Ansible role (from the administration station)
ansible-playbook -i <inventory> <playbook> --limit <group> --tags portal

Common error messages in docker service ps --no-trunc:

Message	Probable cause
`no suitable node`	Impossible placement constraint: unavailable node, missing label, `replicas_max_per_node` reached, or insufficient resources.
`secret not found`	Missing Docker secret: rerun the role with the `secrets` tag (or `secrets_turn`, `traefik`…).
`No such image` / `manifest unknown`	Image missing from registry or not loaded (offline mode: rerun with `--tags load_image`).
`port is already in use` / `address already in use`	Another process (nginx, old container) occupies the published port: `ss -lntup \| grep <port>`.
`task: non-zero exit (1)` then restarts in a loop	Container starts then crashes: check its logs.
`unhealthy container`	Docker healthcheck fails: the service is not listening on its internal port. Check its logs.

Alert: Swarm node down¶

docker node ls
docker node inspect <NODE> --pretty          # status, reason, address
# on the node in question:
systemctl status docker
journalctl -u docker --since "1 hour ago"
docker info | sed -n '/Swarm/,/Node Address/p'
timedatectl                                   # clock skew?

Points to verify:

Docker daemon is running on the node;
Swarm flows between nodes are open: 2377/tcp, 7946/tcp+udp, and 4789/udp (or SWARM_DATA_PATH_PORT). Test: nc -vz <other_node> 2377;
manager quorum: with 3 managers, losing 2 of them blocks the cluster. Never remove a manager without verifying quorum.

Alert: Incomplete Swarm service (replicas x/y)¶

docker service ps <SERVICE> --no-trunc | head -20
docker service inspect <SERVICE> --format '{{json .Spec.TaskTemplate.Placement}}'
# on the node indicated in the NODE column:
journalctl -t <SERVICE> --since "30 min ago"

Special cases:

<INSTANCE_NAME>_db / <INSTANCE_NAME>_logapidb at 0/1: normal state for these unique database update tasks. An alert is only justified if their last execution failed (Failed). Then check journalctl -t reemo_db (or reemo_logapidb) on the node that ran it, then rerun the deployment with --tags db (or logapidb).
<INSTANCE_NAME>_mysqlbackup: legacy backup system, ignore it.

Refer to the frequent error message table in Common Toolbox. If the service restarts in a loop, the cause is almost always in its logs: unreachable database, invalid secret or certificate, configuration error.

Alert: Failed update (paused / rollback)¶

docker service inspect <SERVICE> --format '{{json .UpdateStatus}}'
docker service ps <SERVICE> --no-trunc | head -10

rollback_completed: the new version did not start correctly and Swarm reverted to the previous one. The service is working, but the update is not applied. Analyze the failed task’s logs before rerunning the deployment.
paused: the update or rollback is blocked; intervention is required: docker service rollback <SERVICE> or rerun the role.

Alert: HTTPS certificate near expiration or invalid¶

# Certificate actually presented and chain
openssl s_client -connect portail.example.com:443 -servername portail.example.com -showcerts </dev/null

# Traefik configuration (on managers)
ls -l /opt/traefik/config/
cat /opt/traefik/config/certificates.toml

# Traefik logs (loading errors, ACME)
journalctl -t reemo_traefik --since "1 day ago" | grep -i -E 'error|acme|certificate'

Certificate provided by the client (TRAEFIK_SSL_CERTS): replace the files on the administration station, then rerun --tags traefik_ssl. Traefik reloads the configuration hot, without restart.
Let’s Encrypt: the TLS-ALPN challenge goes through port 443. Verify that the port is reachable from the Internet and that DNS points to the platform.
Traefik presents a ``localhost`` certificate: this is the default certificate. No certificate matches the requested name (SNI): check the URL and the contents of certificates.toml.
Ports in passthrough (Workstation, Appliance, Credential portal): the certificate is from the service itself. Renew it on the service side (PKI / Workstation CA), then redeploy the service.

Alert: Portal / admin portal healthcheck down¶

curl -v https://portail.example.com/api/healthcheck
curl -s https://portail.example.com/api/healthcheck | jq '.. | objects | select(.status? and .status != "OK")'
docker service ps reemo_portal --no-trunc | head
journalctl -t reemo_portal --since "30 min ago"
journalctl -t reemo_traefik --since "30 min ago" | grep -i portal

HTTP code interpretation:

Code	Probable cause
200 with a `status` different from `OK`	A sub-service is failing: identify it in the JSON (`api`, `db`, `provision-api`, `provision-relay-api`, `container-providers`, `ws-relays`, `signal`) and follow the corresponding alert.
403	Monitoring IP missing from `HEALTHCHECK_RESTRICT_IP` (or from `HEALTHCHECK_PORTALADMIN_RESTRICT_IP`).
404	Healthcheck route not deployed (`HEALTHCHECK_ENABLE` is false) or wrong hostname (`PORTAL_URL`).
502 / 503	Traefik cannot reach the `reemo_portal` service: stopped, restarting, or unhealthy.
504	Portal not responding in time: overload, or slow/unreachable dependency (API).
TLS error	See Alert: HTTPS certificate near expiration or invalid.

In a separate architecture, a portal healthcheck error can come from the API: also follow Alert: API (separate architecture).

Alert: Signal down¶

curl -v https://signal.example.com:8443/            # expected: HTTP 426 "Upgrade Required"
docker service ps reemo_signal --no-trunc | head
journalctl -t reemo_signal --since "30 min ago"
ss -lntp | grep 8443                           # on portal managers

Response interpretation:

HTTP 426 “Upgrade Required”: signal is working. If sessions still fail, an intermediary (proxy, WAF, load balancer) is probably blocking the WebSocket upgrade (Upgrade / Connection headers).
HTTP 200, or 426 without this text: the response is not from signal (maintenance page, another traefik router). Check SIGNAL_URL and the port.
404: no traefik router matches the requested hostname.
502 / 504: traefik cannot reach reemo_signal (service stopped, or mTLS traefik > signal failed).

The signal port (TRAEFIK_SSL_SIGNALPORT) must be open inbound. Traefik reaches signal via mTLS: a 502 with TLS errors in traefik logs indicates an expired signal certificate or internal CA.

Alert: API (separate architecture)¶

Follow if there is an alert on the API port, or on the portal healthcheck in separate architecture.

From a portal machine (flow actually used by the portal):

nc -vz <API_IP> 443                             # portal -> API network flow
curl -sk -o /dev/null https://<API_IP>/         # should fail with "certificate required"
docker service ps reemo_api --no-trunc | head   # haproxy proxy on portal side
journalctl -t reemo_api --since "30 min ago"    # haproxy logs (portal node)
journalctl -t reemo_portal --since "30 min ago" | grep -i -E 'api|error|timeout'

On an ``api_manager`` manager:

docker service ls | grep -E 'reemo_(api|mysql|proapi|prorelayapi)|traefik'
docker service ps reemo_api --no-trunc | head
journalctl -t reemo_api --since "30 min ago"
journalctl -t reemo_traefik --since "30 min ago" | grep -i -E 'error|tls|api'

Symptom	Lead
`nc` fails from portal	Network filtering between portal and API, or traefik stopped on API.
Port probe reports “mTLS not required”	API traefik configuration changed (`/opt/traefik/config/certificates.toml`): redeploy with `--tags traefik`.
Port OK, portal healthcheck down, TLS errors in haproxy/portal logs	Portal client certificate or API server certificate expired, or different CA between the two clusters. Check the PKI (see Alert: Internal PKI certificate near expiration).
Port OK, portal healthcheck down, 502 / 504 in API traefik logs	Traefik cannot reach `reemo_api`: service stopped or unhealthy. Check API logs (database connection?).

Alert: Workstation down¶

nc -vz portail.example.com 8445
openssl s_client -connect portail.example.com:8445 </dev/null | openssl x509 -noout -subject -enddate
docker service ps reemo_workstation --no-trunc | head
journalctl -t reemo_workstation --since "30 min ago"

Port 8445 (WORKSTATION_INIT_PORTALPORT) closed: check firewall and traefik service.
Workstation starts but fails on API calls: in separate architecture, follow Alert: API (separate architecture). Otherwise, check WORKSTATION_API_URL and the reemo_api service.
Expired certificate: Workstation CA (WORKSTATION_INITCA_ENABLED), then redeploy with --tags workstation.

Alert: Appliance portal down¶

# From the monitoring server, or from an appliance site
nc -vz portail.example.com 8444
openssl s_client -connect portail.example.com:8444 </dev/null | openssl x509 -noout -subject -enddate

# On a portal manager
docker service ps reemo_applianceportal --no-trunc | head
journalctl -t reemo_applianceportal --since "30 min ago"
ss -lntp | grep 8444                            # is traefik publishing the port?

# On an API manager
docker service ps reemo_applianceapi --no-trunc | head
journalctl -t reemo_applianceapi --since "30 min ago"

Symptom	Lead
Port 8444 closed	Firewall, or traefik deployed without the `applianceportal` entrypoint (`APPLIANCE_ENABLED` false at traefik deployment): rerun with `--tags traefik`.
Port open but connection dropped immediately	`reemo_applianceportal` service stopped or unhealthy: check its logs.
Expired certificate	Renew it via PKI (see Alert: Internal PKI certificate near expiration), then redeploy with `--tags applianceportal`.
Appliances not connecting while probe is OK	Flow blocked from appliance site, or `APPLIANCE_INIT_PORTALURL` incorrect in appliance configuration.
API call errors in applianceportal / applianceapi logs	See `reemo_applianceapi` and, in separate architecture, Alert: API (separate architecture).

Alert: Credential portal down¶

# From the monitoring server (IP authorized in CREDENTIAL_PORTAL_RESTRICT_IP)
nc -vz portail.example.com 8446
openssl s_client -connect portail.example.com:8446 </dev/null | openssl x509 -noout -subject -enddate

# On a portal manager
docker service ps reemo_credentialportal --no-trunc | head
journalctl -t reemo_credentialportal --since "30 min ago"
docker service inspect reemo_credentialportal \
  --format '{{index .Spec.Labels "traefik.tcp.middlewares.reemo_credentialportal_ipallowlist.ipallowlist.sourcerange"}}'

# On an API manager
docker service ps reemo_credentialapi --no-trunc | head
journalctl -t reemo_credentialapi --since "30 min ago"

Symptom	Lead
`nc` OK but `openssl s_client` dropped without certificate	Source IP missing from `CREDENTIAL_PORTAL_RESTRICT_IP` (list shown by the `docker service inspect` command above).
Port 8446 closed	Firewall, or traefik deployed without the `credentialportal` entrypoint: rerun with `--tags traefik`.
Portal reachable but vault opening errors	`reemo_credentialapi` down, or Vault sealed (see Alert: Vault sealed).
Expired certificate	Renew it via PKI, then redeploy with `--tags credentialportal`.

Alert: TURN down¶

# On the node hosting TURN (TURN1_NODE / TURN2_NODE)
docker service ps reemo_turn1 --no-trunc | head
ss -lntup | grep 58200                          # listening on TCP and UDP
journalctl -t reemo_turn1 --since "30 min ago"
timedatectl                                     # time must be synchronized

# From outside
nc -vz  <TURN1_IP> 58200
nc -vzu <TURN1_IP> 58200

Symptom	Lead
Port unreachable from outside	Firewall or NAT: `TURN_PORT` must be open in TCP and UDP to `TURN1_IP` / `TURN2_IP`.
Service at 0/1, `no suitable node`	`TURN1_NODE` does not match the Swarm hostname of the node (`docker node ls`).
Authentication refused (401)	Secret differs between TURN and API, or clocks out of sync (ephemeral credentials).
Sessions not passing while probe is OK	`TURN1_IP` / `TURN2_IP` are not the actual public IPs reachable by clients (`externalip`).

Verify secret consistency: a Docker secret is not readable. Instead, compare the creation date on each side, and ensure TURN_SECRET has the same value in the inventory for turn_manager and for api_manager / infra_manager:

docker secret inspect reemo_TURN_SECRET --format '{{.CreatedAt}}'

The role creates the secret only if it does not exist. Changing TURN_SECRET in the inventory does not update an existing secret. To change it, you must delete and recreate the secret on each cluster. The secret is used by running services (TURN, API): perform this operation during a maintenance window, in coordination with Reemo support.

Alert: Vault sealed¶

docker ps --format '{{.ID}} {{.Names}}' | grep reemo_vault
docker exec <CONTAINER_ID> bao status
journalctl -t reemo_vault1 --since "1 hour ago"

In VAULT_UNSEAL_MODE=manual mode, Vault locks on each container restart (node restart, update…). To unlock it:

using the role (recommended): from the administration station with init.json (VAULT_INIT_FILE), rerun the role with --tags vault;

manually: on each replica, apply VAULT_KEY_THRESHOLD keys (3 by default):

docker exec -it <CONTAINER_ID> bao operator unseal     # repeat with 3 different keys

In raft mode, then check the cluster: docker exec <ID> bao operator raft list-peers (requires a token). The reemo_credentialapi / reemo_credentialportal services resume operation once Vault is unlocked.

Alert: MariaDB database¶

docker service ps reemo_mysql --no-trunc | head
journalctl -t reemo_mysql --since "1 hour ago"
df -h /opt/reemo/db                              # disk full?
id=$(docker ps -q --filter name=^reemo_mysql\.)
docker exec -it $id sh -c 'MYSQL_PWD=$(cat $MYSQL_ROOT_PASSWORD_FILE) mysqladmin -u root status processlist'

Disk full: free up space. The database restarts on its own once space is available; otherwise, restart the service.
Too many connections: check the process list (processlist) and identify the service responsible for the connections.
InnoDB errors at startup: do not delete any files. Contact support with the logs.

Alert: NDB Cluster¶

id=$(docker ps -q --filter name=reemo_mysql-mgmd | head -1)
docker exec $id ndb_mgm -e show
docker exec $id ndb_mgm -e "all status"
docker exec $id ndb_mgm -e "all report memory"
journalctl -t reemo_mysql-mgmd-1 --since "1 hour ago"

ndbd node not connected: check the reemo_mysql-ndbd-N service (docker service ps) and its logs (journalctl -t reemo_mysql-ndbd-N).
Data memory saturated: writes fail (table is full). Increase NDB memory (sizing), then perform a rolling restart with the role (--tags mysqlclusterrollingrestart).

Alert: Backup missing or outdated¶

docker service ps reemo_backup --no-trunc | head
journalctl -t reemo_backup --since "2 days ago"

# From the administration station
ansible-playbook ... --tags backup --extra-vars "LISTBACKUP=true"    # available backups
ansible-playbook ... --tags backup --extra-vars "BACKUP_NOW=true"    # immediate backup
ansible-playbook ... --tags backup --extra-vars "VERIFYBACKUP=true"  # integrity check

Common causes: backup server unreachable (SSH, key), full space on destination, encryption key or password changed, database unavailable at backup time.

Alert: Internal PKI certificate near expiration¶

The role automatically regenerates a service certificate when it expires within 30 days, when the PKI role is run (INITCA_ENABLE=true). To renew:

rerun the role with --tags initca and INITCA_ENABLE=true (git push if INITCA_GIT_ENABLE=true);
redeploy affected services so they pick up the new certificates (full deployment, or targeted tags: api, portal, signal, traefik…).

The CA itself expiration (INITCA_OWNCA_NOT_AFTER, 824 days by default) requires a planned operation: all certificates must be reissued. Plan ahead with Reemo support.

Alert: relayws / provisioning (nginx mTLS) down¶

# On the relayws or provision machine
systemctl status nginx
nginx -t
journalctl -u nginx --since "30 min ago"
tail -50 /var/log/nginx/error.log
ss -lntp | grep 8443

# From an API machine
nc -vz <RELAYWS_OR_PROVISION_IP> 8443

nc fails: check firewall. With RELAYWS_NGINX_FW_ENABLE / PROVISION_NGINX_FW_ENABLE, the API machine IPs must be in *_NGINX_FW_ALLOWED_IPS.
Error 400 No required SSL certificate / certificate verify failed in nginx logs: client certificate (prorelayapi) expired or different CA. See Alert: Internal PKI certificate near expiration.

Alert: Traefik down¶

docker service ps traefik --no-trunc | head
journalctl -t reemo_traefik --since "30 min ago"
ss -lntp | grep -E ':(80|443|8443) '             # port conflict?
ls -ld /opt/traefik /opt/traefik/config          # permissions root:docker 0750

If traefik is stopped, all HTTPS access to the cluster is cut. A port conflict (nginx or apache installed on the host, for example) prevents port publication in host mode.

Alert: Disk full¶

df -h
docker system df
du -sh /opt/* /var/lib/docker/* 2>/dev/null | sort -h | tail

Warning

docker image prune -a removes unused images, including previous versions, making rollback impossible. In offline installations, these images must be reloaded. Prefer docker image prune (orphaned images only) and docker container prune.