NDB Cluster Node Failure

Restore a MySQL NDB cluster after the failure of a Swarm node or of an mgmd service, using the reemo-infra Ansible role.

Note

This procedure is a workaround: it brings the cluster back into service on the remaining hosts, with reduced redundancy, until the missing node returns.

Warning

  • Never use INIT_DB=true on an existing cluster.

  • Don’t use docker service scale reemo_mysql-ndbd-X=0 to stop or test a data node: ndbd ignores SIGTERM and keeps running for 300 s. Use ndb_mgm <id> STOP.

  • Don’t rerun the role when 2 out of 3 hosts are lost: it refuses, by design.

  • Don’t set --nowait-nodes by hand in the image or in config.ini: the role sets and removes it.

NDB identifiers match the host’s position in the host list: mgmd 1 to 3, ndbd 21 to 23, and mysqld 51 to 53.

The deployment commands below use the PLAY variable. Define it from the administration workstation, replacing <inventory> and <playbook> with the inventory and playbook of your infrastructure:

PLAY='ansible-playbook -i <inventory> <playbook> -u root -e INSTALL_DOCKER=true'

Diagnose the Failure

Run the following commands from a Swarm leader node. The diagnosis takes about 1 minute.

# Swarm node in Down state?
docker node ls

# MySQL services at 0/1 replicas?
docker service ls -f name=reemo_mysql-

# Data nodes in started state?
docker exec -i $(docker ps -qf name=reemo_mysql-mgmd | head -n1) ndb_mgm -e "all status"

# ndbmonitor alerts (silence means a healthy cluster)
docker logs --tail 3 $(docker ps -qf name=reemo_ndbmonitor. | head -1)

Compare the result with this table to choose what to do next:

What you observe

Case

The cluster responds (ndb_mgm shows started nodes) and one node or one mgmd is down.

Case A

The cluster is stuck at startup: the mgmd logs show error 4012, the ndbd logs show Waiting for non partitioned start, and a host is missing.

Case B

The host is permanently lost (VM destroyed, disk wiped).

Case B, then case C

Case A: The Cluster Is Running

No special option is needed: the remaining nodes keep running without --nowait-nodes. The cluster runs in degraded mode.

In the following commands, reemo_mysql-mgmd-x stands for the affected service (reemo_mysql-ndbd-x for a data node).

  1. Identify the state of the failed node.

    • If the host is running but the container is stopped, read the error with docker service ps reemo_mysql-mgmd-x. Swarm restarts the container on its own.

    • If the service is at 0 replicas, run docker service scale reemo_mysql-mgmd-x=1.

    • If the host is powered off, restart the VM.

  2. If you restarted the VM, run the following command on the host:

    systemctl start docker.socket docker
    

    The node rejoins Swarm and its services restart.

  3. Wait until the data node is started (or the mgmd is connected) in the output of the following command:

    docker exec -i $(docker ps -qf name=reemo_mysql-mgmd | head -n1) ndb_mgm -e "all status"
    
  4. Rerun the role to return to the nominal state:

    $PLAY
    

    The deployment must end with rc=0 and ndbmonitor must stay silent.

Case B: The Cluster Is Stuck at Startup

This case occurs on a cold start (hosts rebooted, Docker stopped) while a host is missing. The symptoms are:

  • The mgmd logs show error 4012 and wait for the configuration to be confirmed.

  • The ndbd logs show Waiting for non partitioned start, missing: 23 and stay in phase 0.

Note

If the missing data node is on a host whose Swarm node is still ready (stopped container, running host), the ndbd services don’t receive the option: detection only looks at the Swarm state. Bring the container back (case A) rather than forcing it.

  1. Check that the majority of hosts respond (2 out of 3).

    If they don’t, stop here: the role refuses to continue when 2 out of 3 hosts are lost. Recover a second host first.

  2. Run the role:

    $PLAY
    

    The role detects the missing host and sets --nowait-nodes=<missing ids> on the remaining nodes. It waits up to 300 s (MYSQL_CLUSTER_MGMD_START_TIMEOUT) for a missing mgmd to return, then deploys.

    The deployment ends with rc=4: the host is unreachable, but no task failed.

    To avoid waiting 300 s when you know which mgmd is missing, run the following command instead, with the identifier of the missing mgmd (1 to 3, mgmd identifiers only, never ndbd ones):

    $PLAY --tags mysqlclustermgmd -e MYSQL_CLUSTER_MGMD_NOWAIT_NODES=<missing-mgmd-id>
    
  3. Check the cluster state:

    docker exec -i $(docker ps -qf name=reemo_mysql-mgmd | head -n1) ndb_mgm -e "all status"
    

    2 data nodes must be started.

  4. Check that the option is set on a remaining ndbd (reemo_mysql-ndbd-1 if host 1 is not the missing one):

    docker service inspect reemo_mysql-ndbd-1 --format '{{json .Spec.TaskTemplate.ContainerSpec.Args}}'
    

    The output contains --nowait-nodes.

The cluster then runs without redundancy: don’t run any other operation on it until the node returns. When the node is back, follow Return to Normal After Case B.

Case C: The Host Is Permanently Lost

Apply case B first to restart the cluster on the remaining hosts, then add a new host.

  1. From the leader node, check in the output of docker node ls that <old-node> is the node in Down state.

    A mistake on a running manager would break the Swarm quorum.

  2. From the leader node, demote the old node:

    docker node demote <old-node>
    
  3. Remove the old Swarm entry:

    docker node rm --force <old-node>
    
  4. Install Docker on the new host, from the administration workstation (infra_managerX is the inventory group of the new host):

    $PLAY --limit infra_managerX --tags install_docker
    
  5. On the Swarm leader, display the join command:

    docker swarm join-token manager
    
  6. On the new host, run the docker swarm join command displayed in the previous step.

  7. On the new host, create the data directories before rerunning the role:

    mkdir -p /opt/reemo/mysqlcluster/{ndb_mgmd/db,ndb_mgmd/cluster,ndbd/db,mysqld/db,mysqld/files}
    

    Without these directories, the role detects an ndbd that is starting up and refuses to continue.

  8. Wait for Node xx: started in the output of the following command:

    docker exec -i $(docker ps -qf name=reemo_mysql-mgmd | head -n1) ndb_mgm -e "all status"
    
  9. Rerun the role from the administration workstation:

    $PLAY
    

    The deployment must end with rc=0, 27 services, and Args set to ["/bootstrap.sh"] everywhere. This deployment is the final one: it removes --nowait-nodes (see Return to Normal).

Return to Normal After Case B

The final deployment removes --nowait-nodes and restarts the remaining nodes one at a time, with a graceful shutdown (ndb_mgm <id> STOP, then waiting for the node to rejoin). This restart goes through a window without redundancy.

Warning

Run this deployment only when the returned node is started: otherwise, 2 data nodes can be missing at the same time.

  1. Check that all data nodes are started:

    docker exec -i $(docker ps -qf name=reemo_mysql-mgmd | head -n1) ndb_mgm -e "all status"
    
  2. Rerun the role:

    $PLAY
    

    The deployment must end with rc=0.

Fix the ENOTFOUND reemo_api Error After a Node Returns

When a VM returns, containers created in the same second on the node that rejoins Swarm can miss the VIP of a service created at the same moment. The portal then shows the error getaddrinfo ENOTFOUND reemo_api.

  1. Test name resolution from the portal container (docker ps -qf name=reemo_portal gives its identifier):

    docker exec <portal-container> getent hosts reemo_api
    

    An empty answer indicates the problem.

  2. Recreate the affected containers:

    docker service update --force reemo_portal
    

    This command restarts the portal.

If the Expected Result Is Not Obtained

If a node stays missing from ndb_mgm -e "all status" or if a deployment fails, gather the output of the 4 diagnostic commands, the logs of the affected mgmd and ndbd services, and the last Ansible output, then contact Reemo support.

See also

Database — Enabling and prerequisites of MySQL NDBCLUSTER.