Nagios Probes¶

This section provides examples of Nagios-compatible probes (plugins), Icinga, Centreon, Shinken, Naemon, and more generally any tool that understands Nagios return codes:

Code	Status	Meaning
0	OK	Everything is working properly
1	WARNING	Degraded, needs attention
2	CRITICAL	Service down, immediate action required
3	UNKNOWN	The probe could not run

These scripts are examples: adapt the names (reemo_ = INSTANCE_NAME), ports, and thresholds to your platform, then validate them in staging before deploying to production.

Prerequisites and Installation¶

Probes that query Docker (Swarm, services, Vault, database) run locally on a manager of each cluster, via NRPE (or via your tool’s agent). Network probes (certificates, HTTP, TURN, API port) run from the monitoring server.

Copy scripts to /usr/local/lib/nagios/plugins/ (permissions 0755, owner root).
Authorize the nagios user to run only these scripts with sudo, without a password. Avoid adding nagios to the docker group: this effectively grants root privileges.
```
# /etc/sudoers.d/nagios-reemo  (edit with visudo -f)
nagios ALL=(root) NOPASSWD: /usr/local/lib/nagios/plugins/check_reemo_*
```
Dependencies: bash, openssl, curl, awk, timeout (coreutils), and turnutils_uclient (package coturn-utils or coturn) for the TURN probe.

check_reemo_swarm_nodes¶

Run on a manager of each cluster. Verifies that all nodes are Ready / Active, no manager is unreachable, and a leader exists.

#!/bin/bash
# check_reemo_swarm_nodes - Docker Swarm node status
out=$(docker node ls --format '{{.Hostname}} {{.Status}} {{.Availability}} {{.ManagerStatus}}' 2>&1)
if [ $? -ne 0 ]; then
  echo "UNKNOWN - docker node ls : $out"; exit 3
fi
total=$(echo "$out" | wc -l)
bad=$(echo "$out" | awk '$2 != "Ready" || $3 != "Active" || $4 == "Unreachable"')
if [ -n "$bad" ]; then
  echo "CRITICAL - node(s) in fault : $(echo "$bad" | paste -sd ';') | nodes=$total"
  exit 2
fi
if ! echo "$out" | grep -q 'Leader'; then
  echo "CRITICAL - no Swarm leader | nodes=$total"; exit 2
fi
echo "OK - $total node(s) Ready/Active | nodes=$total"
exit 0

check_reemo_swarm_services¶

Run on a manager of each cluster. For each service, verifies that the number of running replicas matches the desired count, and reports failed updates (paused or rollback).

#!/bin/bash
# check_reemo_swarm_services [PREFIX] [EXCLUDE_REGEX]
#   PREFIX          : check only services starting with this prefix
#                      (e.g. reemo_). The traefik service is always checked.
#   EXCLUDE_REGEX   : services to ignore. Default: legacy backup system
#                      (_mysqlbackup).
# One-shot tasks (restart_config: condition none, e.g. _db and _logapidb) are
# not checked on their replica count, but on the state of their last run.
PREFIX="${1:-}"
EXCLUDE="${2:-_mysqlbackup$}"

list=$(docker service ls --format '{{.Name}} {{.Replicas}}' 2>&1)
if [ $? -ne 0 ]; then
  echo "UNKNOWN - docker service ls : $list"; exit 3
fi

crit=(); warn=(); n=0
while read -r name rep _; do
  [ -z "$name" ] && continue
  if [ -n "$PREFIX" ] && [[ "$name" != "$PREFIX"* ]] && [ "$name" != "traefik" ]; then
    continue
  fi
  [[ "$name" =~ $EXCLUDE ]] && continue
  n=$((n + 1))
  cond=$(docker service inspect "$name" --format '{{.Spec.TaskTemplate.RestartPolicy.Condition}}' 2>/dev/null)
  if [ "$cond" = "none" ]; then
    last=$(docker service ps "$name" --format '{{.CurrentState}}' 2>/dev/null | head -1)
    case "$last" in
      Complete*|Running*|Starting*|Preparing*|Assigned*|Pending*) ;;
      *) warn+=("$name one-shot : ${last:-unknown}") ;;
    esac
    continue
  fi
  cur=${rep%%/*}; want=${rep#*/}
  [ "$cur" != "$want" ] && crit+=("$name $cur/$want")
  st=$(docker service inspect "$name" --format '{{if .UpdateStatus}}{{.UpdateStatus.State}}{{end}}' 2>/dev/null)
  case "$st" in
    paused|rollback_started|rollback_paused|rollback_completed) warn+=("$name update=$st") ;;
  esac
done <<< "$list"

if [ ${#crit[@]} -gt 0 ]; then
  echo "CRITICAL - missing replicas : $(IFS=';'; echo "${crit[*]}") $( [ ${#warn[@]} -gt 0 ] && IFS=';' && echo "/ ${warn[*]}") | services=$n"
  exit 2
fi
if [ ${#warn[@]} -gt 0 ]; then
  echo "WARNING - $(IFS=';'; echo "${warn[*]}") | services=$n"
  exit 1
fi
echo "OK - $n service(s) with all replicas | services=$n"
exit 0

check_reemo_cert¶

Run from the monitoring server. Checks TLS certificate expiration, certification chain, and subject name. Works on TCP passthrough ports (Workstation, Appliance portal…) and mTLS ports: the server certificate is read before the client certificate is requested.

#!/bin/bash
# check_reemo_cert HOST PORT [SNI] [WARN_DAYS] [CRIT_DAYS] [VERIFY_CHAIN 1|0]
H="$1"; P="$2"; SNI="${3:-$1}"; W="${4:-30}"; C="${5:-7}"; VERIFY="${6:-1}"
if [ -z "$H" ] || [ -z "$P" ]; then
  echo "UNKNOWN - usage : $0 HOST PORT [SNI] [WARN] [CRIT] [VERIFY_CHAIN]"; exit 3
fi

out=$(echo | timeout 15 openssl s_client -connect "$H:$P" -servername "$SNI" \
            -verify_hostname "$SNI" 2>&1)
end=$(echo "$out" | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
if [ -z "$end" ]; then
  echo "CRITICAL - unable to read certificate on $H:$P"; exit 2
fi
subject=$(echo "$out" | openssl x509 -noout -subject 2>/dev/null | sed 's/^subject=//')
days=$(( ( $(date -d "$end" +%s) - $(date +%s) ) / 86400 ))
vcode=$(echo "$out" | grep -o 'Verify return code: [0-9]*' | head -1 | awk '{print $4}')
vmsg=$(echo "$out" | grep 'Verify return code' | head -1 | sed 's/.*(\(.*\))/\1/')

if [ "$days" -lt 0 ]; then
  echo "CRITICAL - certificate EXPIRED $(( -days )) days ago ($subject) | days=$days"; exit 2
fi
if [ "$VERIFY" = "1" ] && [ "$vcode" != "0" ]; then
  echo "CRITICAL - invalid chain/name : $vmsg ($subject), expires in $days days | days=$days"; exit 2
fi
if [ "$days" -le "$C" ]; then
  echo "CRITICAL - expires in $days days on $end ($subject) | days=$days"; exit 2
fi
if [ "$days" -le "$W" ]; then
  echo "WARNING - expires in $days days on $end ($subject) | days=$days"; exit 1
fi
echo "OK - expires in $days days on $end ($subject) | days=$days"
exit 0

Tip

For certificates issued by the internal PKI (API, relayws, provisioning), pass VERIFY_CHAIN=0: the internal CA is not known to the monitoring server. Only expiration is checked in this case.

HTTP Healthchecks (standard `check_http` plugin)¶

The check_http plugin (or check_curl) from the monitoring-plugins package is sufficient for the admin portal and signal. For the portal, the reemo_healthcheck script (see Health and Monitoring) additionally checks the status of each sub-service:

# Portal: prefer the reemo_healthcheck script, which parses the JSON
reemo_healthcheck https://portal.example.com

# Admin portal
check_http -H admin.example.com -S --sni -u /api/healthcheck -e 200 -w 3 -c 10

# Signal (signal port, 8443 by default)
# (expected response: HTTP 426 with text "Upgrade Required")
check_http -H signal.example.com -p 8443 -S --sni -u / -e 426 -s 'Upgrade Required' -w 3 -c 10

# HTTP > HTTPS redirect for the portal
check_http -H portal.example.com -p 80 -e 301,302,307,308

check_reemo_api_port¶

Run from the monitoring server, without a certificate. Reserved for the separated api_manager / portal_manager architecture. Verifies that the API’s port 443 responds and that the API requires a client certificate: connection without a certificate must be refused at the TLS level.

#!/bin/bash
# check_reemo_api_port API_IP [PORT]
IP="$1"; PORT="${2:-443}"
[ -z "$IP" ] && { echo "UNKNOWN - usage : $0 API_IP [PORT]"; exit 3; }

err=$(curl -sSk -o /dev/null -w '%{http_code}' --max-time 15 "https://$IP:$PORT/" 2>&1)
rc=$?

case $rc in
  0)
    echo "CRITICAL - API $IP:$PORT responds HTTP ${err: -3} WITHOUT client certificate : mTLS not required"
    exit 2 ;;
  6|7)
    echo "CRITICAL - API $IP:$PORT : connection failed (port closed or filtered)"; exit 2 ;;
  28)
    echo "CRITICAL - API $IP:$PORT : timeout"; exit 2 ;;
esac
if echo "$err" | grep -qi -E 'certificate required|bad certificate|certificate_required'; then
  echo "OK - API $IP:$PORT open, client certificate required (mTLS)"; exit 0
fi
echo "WARNING - API $IP:$PORT : unexpected TLS response (curl $rc) : $(echo "$err" | head -1)"
exit 1

Note

The message returned by curl depends on the TLS version and TLS library used (tlsv13 alert certificate required in TLS 1.3 with OpenSSL, alert bad certificate in TLS 1.2). Validate the probe on your platform and adapt the search pattern if needed.

check_reemo_turn¶

Run from the monitoring server (ideally from outside, like a real client). Makes a real TURN allocation with ephemeral credentials (TURN_AUTH_MODE=secret).

#!/bin/bash
# check_reemo_turn HOST [PORT] [SECRET_FILE]
#   SECRET_FILE : file containing TURN_SECRET (permissions 0400, owner nagios)
H="$1"; P="${2:-58200}"; F="${3:-/etc/nagios/reemo_turn_secret}"
if [ -z "$H" ] || [ ! -r "$F" ]; then
  echo "UNKNOWN - usage : $0 HOST [PORT] [SECRET_FILE] (secret readable required)"; exit 3
fi
command -v turnutils_uclient >/dev/null || { echo "UNKNOWN - turnutils_uclient missing"; exit 3; }

u="$(( $(date +%s) + 600 )):nagios"
p=$(printf '%s' "$u" | openssl dgst -sha1 -hmac "$(tr -d '\n' < "$F")" -binary | base64)

out=$(timeout 30 turnutils_uclient -y -n 5 -m 1 -u "$u" -w "$p" -p "$P" "$H" 2>&1)
recv=$(echo "$out" | grep -o 'tot_recv_msgs=[0-9]*' | tail -1 | cut -d= -f2)

if echo "$out" | grep -qi '401\|unauthorized\|wrong credentials'; then
  echo "CRITICAL - TURN $H:$P : authentication denied (different secret ?)"; exit 2
fi
if [ -n "$recv" ] && [ "$recv" -gt 0 ]; then
  echo "OK - TURN $H:$P allocation and relay OK ($recv messages received) | recv=$recv"; exit 0
fi
echo "CRITICAL - TURN $H:$P : no allocation/relay ($(echo "$out" | tail -1))"
exit 2

Note

The output format of turnutils_uclient may vary depending on the coturn version: validate the parsing (tot_recv_msgs) on your version. In TURN_AUTH_MODE=static mode, replace -u/-w with TURN_USERNAME / TURN_PASSWORD.

Warning

TURN_SECRET allows generating valid TURN credentials. Store it with restricted permissions (chmod 0400, owner nagios).

check_reemo_vault¶

Run on each node of the cluster hosting Vault. The probe checks replicas present on the local node. Return codes from bao status: 0 = unlocked, 2 = locked (sealed), 1 = error.

#!/bin/bash
# check_reemo_vault [SERVICE_PREFIX]
PREFIX="${1:-reemo_vault}"
cids=$(docker ps --format '{{.ID}} {{.Names}}' | awk -v p="^${PREFIX}[0-9]+\\\\." '$2 ~ p {print $1":"$2}')
if [ -z "$cids" ]; then
  echo "OK - no Vault replica on this node"; exit 0
fi
sealed=(); err=(); ok=0
for c in $cids; do
  id=${c%%:*}; name=${c#*:}; name=${name%%.*}
  docker exec "$id" bao status >/dev/null 2>&1
  case $? in
    0) ok=$((ok + 1));;
    2) sealed+=("$name");;
    *) err+=("$name");;
  esac
done
if [ ${#sealed[@]} -gt 0 ]; then
  echo "CRITICAL - Vault SEALED : ${sealed[*]}"; exit 2
fi
if [ ${#err[@]} -gt 0 ]; then
  echo "CRITICAL - Vault error : ${err[*]}"; exit 2
fi
echo "OK - $ok Vault replica(s) unlocked"
exit 0

check_reemo_mariadb¶

Run on infra_manager / api_manager nodes. Only the node hosting the database responds; on other nodes, the probe returns OK. Overall service status is covered by check_reemo_swarm_services.

#!/bin/bash
# check_reemo_mariadb [SERVICE_NAME]
SVC="${1:-reemo_mysql}"
id=$(docker ps -q --filter "name=^${SVC}\." | head -1)
if [ -z "$id" ]; then
  echo "OK - $SVC not hosted on this node"; exit 0
fi
out=$(docker exec "$id" sh -c 'MYSQL_PWD=$(cat "$MYSQL_ROOT_PASSWORD_FILE") mysqladmin -u root status' 2>&1)
if [ $? -ne 0 ]; then
  echo "CRITICAL - $SVC : $out"; exit 2
fi
threads=$(echo "$out" | grep -o 'Threads: [0-9]*' | awk '{print $2}')
echo "OK - $SVC : $out | threads=${threads:-0}"
exit 0

check_reemo_ndb¶

Run on a node hosting an mgmd (DB_DIALECT=NDBCLUSTER). Verifies that all NDB nodes are connected and checks memory usage.

#!/bin/bash
# check_reemo_ndb [WARN_%] [CRIT_%]
W="${1:-80}"; C="${2:-90}"
id=$(docker ps -q --filter "name=reemo_mysql-mgmd" | head -1)
[ -z "$id" ] && { echo "UNKNOWN - no mgmd on this node"; exit 3; }

show=$(docker exec "$id" ndb_mgm -e show 2>&1) || { echo "CRITICAL - ndb_mgm : $show"; exit 2; }
nc=$(echo "$show" | grep -c 'not connected')
if [ "$nc" -gt 0 ]; then
  echo "CRITICAL - $nc NDB node(s) not connected : $(echo "$show" | grep 'not connected' | awk '{print $1}' | paste -sd ' ')"
  exit 2
fi

mem=$(docker exec "$id" ndb_mgm -e "all report memory" 2>/dev/null \
      | grep -o '[A-Za-z]* usage is [0-9]*%' | awk '{gsub("%","",$4); print $1"="$4}')
max=$(echo "$mem" | cut -d= -f2 | sort -n | tail -1)
perf=$(echo "$mem" | sort -u | paste -sd ' ')
if [ -n "$max" ] && [ "$max" -ge "$C" ]; then
  echo "CRITICAL - NDB memory at ${max}% | $perf"; exit 2
fi
if [ -n "$max" ] && [ "$max" -ge "$W" ]; then
  echo "WARNING - NDB memory at ${max}% | $perf"; exit 1
fi
echo "OK - all NDB nodes connected, max memory ${max:-?}% | $perf"
exit 0

check_reemo_pki¶

Run on the Ansible admin workstation (where the PKI is located: LOCAL_PATH or a clone of the PKI git repository). Checks expiration of all internal PKI certificates.

#!/bin/bash
# check_reemo_pki DIRECTORY [WARN_DAYS] [CRIT_DAYS]
D="$1"; W="${2:-30}"; C="${3:-7}"
[ -d "$D" ] || { echo "UNKNOWN - directory $D not found"; exit 3; }
crit=(); warn=(); n=0
for f in "$D"/*.crt; do
  [ -e "$f" ] || continue
  n=$((n + 1))
  end=$(openssl x509 -in "$f" -noout -enddate 2>/dev/null | cut -d= -f2) || continue
  days=$(( ( $(date -d "$end" +%s) - $(date +%s) ) / 86400 ))
  if   [ "$days" -le "$C" ]; then crit+=("$(basename "$f")=${days}d")
  elif [ "$days" -le "$W" ]; then warn+=("$(basename "$f")=${days}d")
  fi
done
[ ${#crit[@]} -gt 0 ] && { echo "CRITICAL - ${crit[*]} ${warn[*]}"; exit 2; }
[ ${#warn[@]} -gt 0 ] && { echo "WARNING - ${warn[*]}"; exit 1; }
echo "OK - $n certificate(s) valid for more than $W days"
exit 0

check_reemo_backup_age¶

The backup service (reemo_backup) sends archives to the backup server. The most reliable check is to verify on the backup server the age of the most recent file:

#!/bin/bash
# check_reemo_backup_age BACKUP_DIRECTORY [WARN_SEC] [CRIT_SEC]
D="$1"; W="${2:-90000}"; C="${3:-172800}"      # 25 h / 48 h
last=$(ls -1t "$D" 2>/dev/null | head -1)
[ -z "$last" ] && { echo "CRITICAL - no backup in $D"; exit 2; }
exec /usr/lib/nagios/plugins/check_file_age -w "$W" -c "$C" -f "$D/$last"

Additionally, enable native notifications from the backup container on failure: BACKUP_MAIL_RECEIVER or BACKUP_WEBHOOK_ENABLED / BACKUP_WEBHOOK_URL, with BACKUP_NOTIFY_ON.

Configuration NRPE¶

Example of /etc/nagios/nrpe.d/reemo.cfg on a portal_manager node:

command[check_reemo_swarm_nodes]=/usr/bin/sudo /usr/local/lib/nagios/plugins/check_reemo_swarm_nodes
command[check_reemo_swarm_services]=/usr/bin/sudo /usr/local/lib/nagios/plugins/check_reemo_swarm_services reemo_
command[check_docker]=/usr/lib/nagios/plugins/check_procs -c 1: -C dockerd
command[check_disk_docker]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /var/lib/docker
command[check_disk_opt]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /opt

On an api_manager / infra_manager node, add:

command[check_reemo_vault]=/usr/bin/sudo /usr/local/lib/nagios/plugins/check_reemo_vault
command[check_reemo_mariadb]=/usr/bin/sudo /usr/local/lib/nagios/plugins/check_reemo_mariadb
# or, on NDB Cluster:
command[check_reemo_ndb]=/usr/bin/sudo /usr/local/lib/nagios/plugins/check_reemo_ndb 80 90

Nagios Definitions¶

Example commands and services on the Nagios server:

define command {
    command_name  check_reemo_cert
    command_line  $USER1$/check_reemo_cert $ARG1$ $ARG2$ $ARG3$ 30 7 $ARG4$
}
define command {
    command_name  check_reemo_https
    command_line  $USER1$/check_http -H $ARG1$ -p $ARG2$ -S --sni -u $ARG3$ -e 200 -w 3 -c 10
}
define command {
    command_name  check_reemo_tcp
    command_line  $USER1$/check_tcp -H $ARG1$ -p $ARG2$ -w 2 -c 5
}
define command {
    command_name  check_reemo_signal
    command_line  $USER1$/check_http -H $ARG1$ -p $ARG2$ -S --sni -u / -e 426 -s 'Upgrade Required' -w 3 -c 10
}
define command {
    command_name  check_reemo_api_port
    command_line  $USER1$/check_reemo_api_port $ARG1$ 443
}
define command {
    command_name  check_reemo_turn
    command_line  $USER1$/check_reemo_turn $ARG1$ $ARG2$ /etc/nagios/reemo_turn_secret
}

# --- Portal ----------------------------------------------------------
# The reemo_healthcheck command is defined on the "Health and Monitoring" page
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - portal certificate
    check_command        check_reemo_cert!portal.example.com!443!portal.example.com!1
    check_interval       720
}
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - portal healthcheck
    check_command        reemo_healthcheck!https://portal.example.com
}
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - signal
    check_command        check_reemo_signal!signal.example.com!8443
}
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - workstation certificate
    check_command        check_reemo_cert!portal.example.com!8445!portal.example.com!0
    check_interval       720
}
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - swarm services
    check_command        check_nrpe!check_reemo_swarm_services
}

# --- Appliances (APPLIANCE_ENABLED) ----------------------------------
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - appliance portal port 8444
    check_command        check_reemo_tcp!portal.example.com!8444
}
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - appliance portal certificate
    check_command        check_reemo_cert!portal.example.com!8444!portal.example.com!0
    check_interval       720
}

# --- Credential portal (CREDENTIAL_ENABLED + CREDENTIALPORTAL_ENABLED) ---
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - credential portal port 8446
    check_command        check_reemo_tcp!portal.example.com!8446
}
define service {
    use                  generic-service
    host_name            portal
    service_description  Reemo - credential portal certificate
    check_command        check_reemo_cert!portal.example.com!8446!portal.example.com!0
    check_interval       720
}

# --- API (separated architecture) ------------------------------------
define service {
    use                  generic-service
    host_name            api
    service_description  Reemo - API port 443 / mTLS required
    check_command        check_reemo_api_port!10.0.0.10
}
define service {
    use                  generic-service
    host_name            api
    service_description  Reemo - API certificate
    check_command        check_reemo_cert!10.0.0.10!443!reemo_api!0
    check_interval       720
}
define service {
    use                  generic-service
    host_name            api
    service_description  Reemo - Vault sealed
    check_command        check_nrpe!check_reemo_vault
}

# --- TURN ----------------------------------------------------------------
define service {
    use                  generic-service
    host_name            turn
    service_description  Reemo - TURN allocation
    check_command        check_reemo_turn!203.0.113.10!58200
}

Recommended frequencies:

healthchecks, Swarm services, TURN, Vault, API: every 1 to 5 minutes.
certificates and PKI: twice a day (check_interval 720).
backups: every hour.

Nagios Probes¶

Prerequisites and Installation¶

check_reemo_swarm_nodes¶

check_reemo_swarm_services¶

check_reemo_cert¶

HTTP Healthchecks (standard check_http plugin)¶

check_reemo_api_port¶

check_reemo_turn¶

check_reemo_vault¶

check_reemo_mariadb¶

check_reemo_ndb¶

check_reemo_pki¶

check_reemo_backup_age¶

Configuration NRPE¶

Nagios Definitions¶

HTTP Healthchecks (standard `check_http` plugin)¶