Additional Options¶

The Ansible role reemo-infra responsible for the installation has additional options. For ease of reading, these options are detailed in this document.

Syslog Logging¶

By default, the logs of each service are sent to the local Syslog of each node with facility=daemon by default.

Syslog Facility¶

You can modify the facility value using the following list of options:

SYSLOG_FACILITY: "daemon"
SYSLOG_FACILITY_API: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_APICRON: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_DB: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_EXIM4: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_MYSQL: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PORTAL: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PORTALADMIN: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_SIGNAL: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PROAPI: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_STUN: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PRORELAYAPI: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_PROCLOUDAPI: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_APICRONLOG: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_APICRONLDAP: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_APICRONCLOUD: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_HAPROXY: "{{SYSLOG_FACILITY}}"
SYSLOG_FACILITY_TRAEFIK: "{{SYSLOG_FACILITY}}"

Tip

All logs are written to the Syslog with the prefix reemo_

Useful commands for searching Reemo logs¶

To search for errors in the logs using Ansible:

Infra Manager

ansible infra_manager -i inventory.yml -m shell -a "grep _traefik /var/log/syslog | grep '\" 5[0-9][0-9] [0-9]'"
ansible infra_manager -i inventory.yml -m shell -a "grep ' 500 [0-9]' /var/log/syslog"

API Manager

ansible api_manager -i inventory.yml -m shell -a "grep _traefik /var/log/syslog | grep '\" 5[0-9][0-9] [0-9]'"
ansible api_manager -i inventory.yml -m shell -a "grep ' 500 [0-9]' /var/log/syslog"

Portal Manager

ansible portal_manager -i inventory.yml -m shell -a "grep _traefik /var/log/syslog | grep '\" 5[0-9][0-9] [0-9]'"
ansible portal_manager -i inventory.yml -m shell -a "grep ' 500 [0-9]' /var/log/syslog"

Security Logs¶

The platform allows the management and centralization of security events via the logapi service. The following configuration options are available:

LOGAPI_ENABLED: true (default). Allows you to globally enable or disable the logapi service.
LOGAPI_SYSLOG_ACTIVE: false. Defines whether security logs should be sent to the host’s local Syslog service.
API_LOGAPI_ANONYMIZE: true. Allows you to enable anonymization of sensitive data within the security logs.

Platform Health¶

To monitor the health of the platform from an external service, you have the option to enable a healthcheck route on one or more Portals. This route allows you to display a global status of the platform. It is accessible via the URL:

https://your-portal-url/api/healthcheck

Example of the JSON response:

{
    "status": "OK",
    "version": "2.14.4",
    "services": {
        "api":
        {
            "status": "OK",
            "version": "2.15.2",
            "services":
            {
                "db":
                {
                    "status": "OK"
                },
                "provision-api":
                {
                    "status": "OK",
                    "version": "2.7.0"
                },
                "provision-relay-api":
                {
                    "status": "OK",
                    "version": "1.3.0"
                },
                "container-providers": [
                    {
                        "status": "OK",
                        "name": "Reemo SWARM",
                        "type": "SWARM"
                    }
                ],
                "ws-relays": [
                    {
                        "status": "OK",
                        "name": "Global - WS_SWARM",
                        "type": "WS_SWARM"
                    }
                ]
            }
        },
        "signal": {
            "status": "OK",
            "version": "2.6.1"
        }
    }
}

Enabling the route¶

To enable this route on the Portal, you can use the options:

HEALTHCHECK_ENABLE: “true”
HEALTHCHECK_RESTRICT_IP: “192.168.1.1,192.168.10.0/24”

And to enable this route on the Admin Portal:

HEALTHCHECK_PORTALADMIN_ENABLE: “true”
HEALTHCHECK_PORTALADMIN_RESTRICT_IP: “192.168.1.1,192.168.10.0/24”

Prometheus Format¶

If you wish to obtain this information in Prometheus format, it is accessible on the route https://your-portal-url/api/healthcheck/prometheus

# HELP app_service_status Services status (1 = up, 0 = down)
# TYPE app_service_status gauge
# HELP app_service_items Services items numbers
# TYPE app_service_items gauge

app_service_status{service="db"} 1
app_service_status{service="provision-api"} 0
app_service_status{service="provision-relay-api"} 0
app_service_status{service="Mon provider - SWARM", parent="container-providers", type="SWARM"} 0
app_service_status{service="Mon websocket relay 1 - WS_SWARM", parent="ws-relays", type="WS_SWARM"} 0
app_service_status{service="Mon websocket relay 1 - Traefik status", parent="ws-relays", type="WS_SWARM"} 0

app_service_status: Indicates the status of each monitored service. A value of 1 means the service is operational. A value of 0 means it is unavailable.

Monitoring Example¶

Here is an example of setting up platform health monitoring using Nagios

Prerequisites¶

On the Nagios server: have the curl and jq commands available

Command¶

define command {
    command_name    reemo_healthcheck
    command_line    /usr/lib/nagios/plugins/reemo_healthcheck $ARG1$
}

Script¶

#!/bin/bash

# Check if an argument (URL) is provided
if [ -z "$1" ]; then
    echo "CRITICAL: No URL provided. Usage: $0 <URL>"
    exit 2
fi

HEALTHCHECK_URL="$1"

# Fetch JSON response
response=$(curl -s "$HEALTHCHECK_URL/api/healthcheck")

# Check if the request was successful
if [ $? -ne 0 ] || [ -z "$response" ]; then
    echo "CRITICAL: Unable to access the health check page ($HEALTHCHECK_URL)"
    exit 2
fi

# Extract service statuses using jq
status=$(echo "$response" | jq -r '.status')
errors=$(echo "$response" | jq -r '.. | objects | select(.status? and .status != "OK")')

# Check if the global status is OK
if [ "$status" != "OK" ]; then
    echo "CRITICAL: Global health check status is $status"
    exit 2
fi

# Check if any sub-services have issues
if [ -n "$errors" ]; then
    echo "WARNING: Some services are not OK:"
    echo "$errors"
    exit 1
fi

# Everything is OK
echo "OK: All services are operational"
exit 0

Service¶

define service{
    host_name                       < HOSTNAME >
    use                             generic-service         ; Name of service template to use
    service_description             Reemo Healthcheck
    check_command                   reemo_healthcheck!https://< URL >
}

Prometheus Monitoring¶

It is possible to enable Prometheus on the Traefik entry points; several options are available:

Enable Prometheus¶

To enable Prometheus, you can add the option TRAEFIK_PROMETHEUS_ENABLE: true in the inventory file.

Port¶

By default, the Prometheus page will be accessible on the same port as the Portal. Use the option TRAEFIK_PROMETHEUS_PORT: “<port number>” to change it.

URL¶

By default, the Prometheus page will be accessible on the Portal URL with the /metrics path. It is possible to specify one or more dedicated URLs:

TRAEFIK_PROMETHEUS_URLS:
    - "metrics1.domain.tld"
    - "metrics2.domain.tld"

Note

In the case of a 3-node cluster, it is recommended to set 3 URLs so that Prometheus can retrieve the metrics from all nodes.

IP Filtering¶

To restrict access to specific IP addresses, you can use the TRAEFIK_PROMETHEUS_RESTRICT_IP option.

Example:

TRAEFIK_PROMETHEUS_RESTRICT_IP: "1.1.1.1,2.2.2.2"

Email Sending¶

If you want the Reemo platform to be able to send emails to users, you can enable the messaging service included in the reemo-infra Ansible role:

API_MAIL_ACTIVE: "true"
API_MAIL_BASEURL: "https://< URL du portal de connexion pour les utilisateurs >"
EXIM_SMARTHOST: "< URL du serveur SMTP >::587"
EXIM_PASSWORD: "< URL du serveur SMTP >:< Utilisateur SMTP >:< Mot de passe de connexion SMTP>"

Maintenance Mode¶

During a maintenance operation, you have the option to replace the Portal with a maintenance page.

Enable¶

ansible-playbook -i inventory.yml playbooks/reemo-infra.yml --tags maintenance --extra-vars "MAINTENANCE_ENABLE=true"

Disable¶

ansible-playbook -i inventory.yml playbooks/reemo-infra.yml --tags portal

CRON¶

Several CRON tasks are declared in a Reemo platform. It is imperative that they are unique. When using multiple infra or api servers/clusters, you must choose the server/cluster that will handle these services and disable them on the others using the CRON_ENABLE: “false” option.

Gestion générale des tâches

Option disponible:

API_cronGhostContainersMinutes: Temps d’expiration des conteneurs qui continuent à être UP dans l’environnement Provision mais qui n’ont plus d’existance connue par l’interface

Alternative LDAP¶

In the case of a multi-API architecture, it is possible to use 2 FQDNs for the same LDAP server.

To do this, you can add the following option to the configuration of the API server that needs to use the alternative URL to contact the LDAP server:

API_useAlternateLDAPHost: "true"