Quick status check¶

When a problem is reported (portal inaccessible, sessions failing, multiple alerts), this page helps you assess the platform status in a few minutes and identify the failing component. Then follow the corresponding troubleshooting sheet in Alert Troubleshooting.

Status check script¶

The following script summarizes a cluster’s status: nodes, incomplete services, failed updates, failing one-off tasks, unhealthy containers, node resources, Vault, and listening ports. Run it as root on each cluster’s manager. Each failing line starts with [KO].

#!/bin/bash
# reemo_etat_des_lieux [PREFIX]  (PREFIX = INSTANCE_NAME followed by _, default reemo_)
P="${1:-reemo_}"
ko=0
say() { echo "  $*"; case "$*" in "[KO]"*) ko=$((ko + 1));; esac; }

echo "== $(hostname) - $(date '+%F %T')"

echo "== Swarm Nodes"
if nodes=$(docker node ls --format '{{.Hostname}} {{.Status}} {{.Availability}} {{.ManagerStatus}}' 2>&1); then
  while read -r h s a m; do
    if [ "$s" = "Ready" ] && [ "$a" = "Active" ] && [ "$m" != "Unreachable" ]; then
      say "[OK] $h $s $a $m"
    else
      say "[KO] $h $s $a $m"
    fi
  done <<< "$nodes"
else
  say "[KO] docker node ls : $nodes (run the script on a manager)"
fi

echo "== Services"
while read -r n r _; do
  [[ "$n" == "$P"* || "$n" == "traefik" ]] || continue
  [[ "$n" =~ _mysqlbackup$ ]] && continue
  c=$(docker service inspect "$n" --format '{{.Spec.TaskTemplate.RestartPolicy.Condition}}')
  if [ "$c" = "none" ]; then
    st=$(docker service ps "$n" --format '{{.CurrentState}}' | head -1)
    case "$st" in Complete*|Running*) ;; *) say "[KO] $n one-off task : $st";; esac
    continue
  fi
  [ "${r%%/*}" != "${r#*/}" ] && say "[KO] $n replicas $r"
  u=$(docker service inspect "$n" --format '{{if .UpdateStatus}}{{.UpdateStatus.State}}{{end}}')
  case "$u" in paused|rollback_*) say "[KO] $n update : $u";; esac
done < <(docker service ls --format '{{.Name}} {{.Replicas}}')

echo "== Unhealthy containers (local node)"
while read -r line; do [ -n "$line" ] && say "[KO] $line"; done \
  < <(docker ps --filter health=unhealthy --format '{{.Names}} {{.Status}}')

echo "== Resources (local node)"
while read -r fs size used avail pct mnt; do
  if [ "${pct%\%}" -ge 90 ]; then say "[KO] disk $mnt $pct"; else say "[OK] disk $mnt $pct"; fi
done < <(df -hP / /var/lib/docker /opt 2>/dev/null | awk 'NR > 1' | sort -u -k6)
free -m | awk '/^Mem:/ {printf "  [info] available memory : %d Mo / %d Mo\n", $7, $2}'
oom=$(dmesg -T 2>/dev/null | grep -ci 'killed process')
[ "${oom:-0}" -gt 0 ] && say "[KO] $oom processes killed by kernel (OOM), see dmesg -T"

echo "== Vault (local replicas)"
for c in $(docker ps --format '{{.ID}}:{{.Names}}' | grep -E ":${P}vault[0-9]+\."); do
  docker exec "${c%%:*}" bao status >/dev/null 2>&1
  case $? in
    0) say "[OK] ${c#*:} unlocked";;
    2) say "[KO] ${c#*:} SEALED";;
    *) say "[KO] ${c#*:} bao status error";;
  esac
done

echo "== Listening ports (local node)"
ss -lntuH 2>/dev/null | awk '{print $1" "$5}' \
  | grep -E ':(80|443|8443|8444|8445|8446|58200)$' | sort -u | sed 's/^/  [info] /'

echo "== Result: $ko failing point(s)"
[ "$ko" -eq 0 ]

Workflow by installation type¶

From the monitoring server, check the portal healthcheck and display only the failing components:
```
curl -s https://portail.example.com/api/healthcheck \
  | jq '.. | objects | select(.status? and .status != "OK")'
```
No output: all components respond. Otherwise, the failing component is indicated (api, db, provision-api, ws-relays, signal…).
On an infra_manager manager, run reemo_etat_des_lieux.
If a node is failing, also run the script on that node (disk, unhealthy containers, and Vault are checked locally).

Check entry points from outside:

for p in 443 8443; do nc -vz -w 3 portail.example.com $p; done

If Reemo Containers or WebSocket sessions are failing, also run the script on the provisionN_manager and relayws_manager clusters.

From the monitoring server, check the portal healthcheck: it indicates whether the problem is from the portal or the API.
```
curl -s https://portail.example.com/api/healthcheck \
  | jq '.. | objects | select(.status? and .status != "OK")'
```
No response at all: start with portal_manager. Component api or db failing: start with api_manager.

On a portal_manager manager: run reemo_etat_des_lieux, then test the flow to the API:

nc -vz -w 3 <API_IP> 443
curl -sk -o /dev/null https://<API_IP>/   # expected: "certificate required" error

On an api_manager manager: run reemo_etat_des_lieux (API services, database, Vault).
On turn_manager (if WebRTC sessions are failing): run reemo_etat_des_lieux, then test the TURN port from outside:
```
nc -vz -w 3 <TURN1_IP> 58200
```
On relayws_manager / provisionN_manager (if WebSocket or Reemo Containers sessions are failing): run reemo_etat_des_lieux, then systemctl status nginx.

Reading the results¶

Finding	Troubleshooting sheet
Node `[KO]` (`Down`, `Unreachable`)	Alert: Swarm node down
Service `[KO] replicas x/y`	Alert: Incomplete Swarm service (replicas x/y)
`[KO] update : paused / rollback_...`	Alert: Failed update (paused / rollback)
`[KO] ... one-off task : Failed` (`_db`, `_logapidb`)	Alert: Incomplete Swarm service (replicas x/y)
Unhealthy container	Alert: Incomplete Swarm service (replicas x/y), then the sheet for the component
Disk `[KO]` or processes killed (OOM)	Alert: Disk full
Vault `SEALED`	Alert: Vault sealed
Expected port missing from listening ports list	Alert: Traefik down (ports 80/443/84xx) or Alert: TURN down (58200)
Healthcheck: `api` or `db` failing	Alert: API (separate architecture), Alert: MariaDB database or Alert: NDB Cluster
Healthcheck: `ws-relays` / `provision-relay-api` failing	Alert: relayws / provisioning (nginx mTLS) down
Healthcheck: `signal` failing	Alert: Signal down
Everything is `[OK]` but portal does not respond from outside	Alert: HTTPS certificate near expiration or invalid and Alert: Portal / admin portal healthcheck down (certificate, IP filtering, DNS)