Additional Options¶
The reemo-infra Ansible role that installs the INFRA environment provides extra options. For readability, these options are detailed in this document.
Syslog logging¶
By default each service logs to the node’s local Syslog with facility=daemon.
Syslog Facility¶
You can change the facility using the following options:
SYSLOG_FACILITY: "daemon"
SYSLOG_FACILITY_API: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_APICRON: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_DB: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_EXIM4: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_MYSQL: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PORTAL: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PORTALADMIN: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_SIGNAL: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PROAPI: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_STUN: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PRORELAYAPI: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PROCLOUDAPI: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_APICRONLOG: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_APICRONLDAP: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_APICRONCLOUD: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_HAPROXY: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_TRAEFIK: "{{SYSLOG_FACILITY}}"
Tip
All logs are written to Syslog with the reemo_ prefix.
Useful commands to search Reemo logs¶
To search for errors in logs with Ansible:
Infra Manager
ansible infra_manager -i inventory.yml -m shell -a "grep _traefik /var/log/syslog | grep '\" 5[0-9][0-9] [0-9]'"
ansible infra_manager -i inventory.yml -m shell -a "grep ' 500 [0-9]' /var/log/syslog"
API Manager
ansible api_manager -i inventory.yml -m shell -a "grep _traefik /var/log/syslog | grep '\" 5[0-9][0-9] [0-9]'"
ansible api_manager -i inventory.yml -m shell -a "grep ' 500 [0-9]' /var/log/syslog"
Portal Manager
ansible portal_manager -i inventory.yml -m shell -a "grep _traefik /var/log/syslog | grep '\" 5[0-9][0-9] [0-9]'"
ansible portal_manager -i inventory.yml -m shell -a "grep ' 500 [0-9]' /var/log/syslog"
Platform Health¶
To monitor platform health from an external service, you can enable a healthcheck route on the Reemo Portal. This route returns a global status and is available at:
https://your-portal-url/api/healthcheck
Example JSON response:
{
"status": "OK",
"version": "2.14.4",
"services": {
"api":
{
"status": "OK",
"version": "2.15.2",
"services":
{
"db":
{
"status": "OK"
},
"provision-api":
{
"status": "OK",
"version": "2.7.0"
},
"provision-relay-api":
{
"status": "OK",
"version": "1.3.0"
},
"container-providers": [
{
"status": "OK",
"name": "Reemo SWARM",
"type": "SWARM"
}
],
"ws-relays": [
{
"status": "OK",
"name": "Global - WS_SWARM",
"type": "WS_SWARM"
}
]
}
},
"signal": {
"status": "OK",
"version": "2.6.1"
}
}
}
Enabling the route¶
To enable this route on the User Portal, use:
HEALTHCHECK_ENABLE: “true”
HEALTHCHECK_RESTRICT_IP: “192.168.1.1,192.168.10.0/24”
To enable it on the Admin Portal, use:
HEALTHCHECK_PORTALADMIN_ENABLE: “true”
HEALTHCHECK_PORTALADMIN_RESTRICT_IP: “192.168.1.1,192.168.10.0/24”
Prometheus Format¶
If you want to obtain this information in Prometheus format, it is available at the route https://your-portal-url/api/healthcheck/prometheus
# HELP app_service_status Services status (1 = up, 0 = down)
# TYPE app_service_status gauge
# HELP app_service_items Services items numbers
# TYPE app_service_items gauge
app_service_status{service="db"} 1
app_service_status{service="provision-api"} 0
app_service_status{service="provision-relay-api"} 0
app_service_status{service="Mon provider - SWARM", parent="container-providers", type="SWARM"} 0
app_service_status{service="Mon websocket relay 1 - WS_SWARM", parent="ws-relays", type="WS_SWARM"} 0
app_service_status{service="Mon websocket relay 1 - Traefik status", parent="ws-relays", type="WS_SWARM"} 0
app_service_statusIndicates the status of each monitored service. The value
1means the service is operational. The value0means it is unavailable.
Monitoring example¶
Example of health monitoring with Nagios.
Prerequisites¶
On the Nagios server: have curl and jq available.
Command¶
define command {
command_name reemo_healthcheck
command_line /usr/lib/nagios/plugins/reemo_healthcheck $ARG1$
}
Script¶
#!/bin/bash
# Check if an argument (URL) is provided
if [ -z "$1" ]; then
echo "CRITICAL: No URL provided. Usage: $0 <URL>"
exit 2
fi
HEALTHCHECK_URL="$1"
# Fetch JSON response
response=$(curl -s "$HEALTHCHECK_URL/api/healthcheck")
# Check if the request was successful
if [ $? -ne 0 ] || [ -z "$response" ]; then
echo "CRITICAL: Unable to access the health check page ($HEALTHCHECK_URL)"
exit 2
fi
# Extract service statuses using jq
status=$(echo "$response" | jq -r '.status')
errors=$(echo "$response" | jq -r '.. | objects | select(.status? and .status != "OK")')
# Check if the global status is OK
if [ "$status" != "OK" ]; then
echo "CRITICAL: Global health check status is $status"
exit 2
fi
# Check if any sub-services have issues
if [ -n "$errors" ]; then
echo "WARNING: Some services are not OK:"
echo "$errors"
exit 1
fi
# Everything is OK
echo "OK: All services are operational"
exit 0
Service¶
define service{
host_name < HOSTNAME >
use generic-service ; Name of service template to use
service_description Reemo Healthcheck
check_command reemo_healthcheck!https://< URL >
}
Prometheus Monitoring¶
You can enable Prometheus on Traefik entrypoints. Several options are available:
Enable Prometheus¶
To enable Prometheus, add TRAEFIK_PROMETHEUS_ENABLE: true in the inventory.
Port¶
By default the Prometheus metrics are exposed on the same port as the Portal. Use TRAEFIK_PROMETHEUS_PORT: “<port number>” to change it.
URL¶
By default the Prometheus page is exposed under the Portal URL at /metrics. You can specify one or more dedicated URLs:
TRAEFIK_PROMETHEUS_URLS:
- "metrics1.domain.tld"
- "metrics2.domain.tld"
Note
In a 3-node cluster, set 3 URLs so Prometheus can scrape metrics from all nodes.
IP filtering¶
To restrict access by IP addresses, use TRAEFIK_PROMETHEUS_RESTRICT_IP.
Example:
TRAEFIK_PROMETHEUS_RESTRICT_IP: "1.1.1.1,2.2.2.2"
Email sending¶
If you want the Reemo platform to send emails to users, enable the mail service included in the reemo-infra role:
API_MAIL_ACTIVE: "true"
API_MAIL_BASEURL: "https://< URL of the user login portal >"
EXIM_SMARTHOST: "< SMTP server URL >::587"
EXIM_PASSWORD: "< SMTP server URL >:< SMTP user >:< SMTP password>"
Maintenance mode¶
During maintenance, you can replace the Portal with a maintenance page.
Enable¶
ansible-playbook -i inventory.yml playbooks/reemo-infra.yml --tags maintenance --extra-vars "MAINTENANCE_ENABLE=true"
Disable¶
ansible-playbook -i inventory.yml playbooks/reemo-infra.yml --tags portal
Alternate LDAP¶
In a multi-API architecture it is possible to use two FQDNs for the same LDAP server.
Add the following option on the API server that must use the alternate LDAP URL:
API_useAlternateLDAPHost: "true"
Combine with the RELAYWS environment¶
By default the INFRA environment uses WebRTC between users and resources. If you want to use WebSocket for HTTPS-only connections, add a RELAYWS environment and provide the IP addresses in the inventory as well as the user connection URL for RELAYWS.
Example:
all:
vars:
API_LICENSE: "ewogICAg ... Uw5NXhGVDF0NFU2TkxOdjQvZU53PT0iCiAgICC9Cn0="
infra_manager:
vars:
PORTAL_URL: "url.domain.tld"
PORTALADMIN_URL: "urladmin.domain.ltd"
PORTALADMIN_URL_RESTRICT_IP: "10.3.1.2,10.3.2.0/24"
TRAEFIK_SSL_CERTS:
- cert_file: "/localpath/to/cert.crt"
key_file: "/localpath/to/key.key"
RELAYS_IP:
- ip: "10.10.0.1"
- ip: "10.10.0.2"
- ip: "10.10.0.3"
RELAYS_URL: "relayws.domain.tld"
hosts:
infra_manager1:
ansible_host: "10.0.0.1"
infra_manager2:
ansible_host: "10.0.0.2"
infra_manager3:
ansible_host: "10.0.0.3"
Combine with the PROVISION environment¶
If you want to use a PROVISION environment to leverage Reemo Containers, you must interconnect with a PROVISION environment.
Tip
One INFRA environment can drive as many PROVISION environments as you need.
The connection between INFRA and PROVISION is HTTPS from INFRA to PROVISION via a URL resolvable by INFRA. Two choices:
With DNS¶
Without DNS¶
If you do not use DNS, specify in the inventory the IP addresses of the Nginx services in PROVISION, only in the api_manager and infra_manager groups.
Also add PROVISION_SIGNAL_IP, the IP addresses of the Signaling servers where containers will register, usually the IPs of Portal or INFRA servers.
Example:
all:
vars:
API_LICENSE: "ewogICAg ... Uw5NXhGVDF0NFU2TkxOdjQvZU53PT0iCiAgICC9Cn0="
infra_manager:
vars:
PORTAL_URL: "url.domain.tld"
PORTALADMIN_URL: "urladmin.domain.ltd"
PORTALADMIN_URL_RESTRICT_IP: "10.3.1.2,10.3.2.0/24"
TRAEFIK_SSL_CERTS:
- cert_file: "/localpath/to/cert.crt"
key_file: "/localpath/to/key.key"
PROVISION_IP:
- ip: "10.10.0.1"
- ip: "10.10.0.2"
- ip: "10.10.0.3"
PROVISION_SIGNAL_IP:
- ip: "10.0.0.1"
- ip: "10.0.0.2"
- ip: "10.0.0.3"
hosts:
infra_manager1:
ansible_host: "10.0.0.1"
infra_manager2:
ansible_host: "10.0.0.2"
infra_manager3:
ansible_host: "10.0.0.3"