A MariaDB Galera Cluster allows you to run a multi-primary database cluster where you can both read and write to any node in the cluster. The benefits of this over traditional primary/secondary clustering are:
- No secondary lag
- No lost transactions
- Read scalability
- Smaller client latencies
If a Galera cluster is not restarted in a particular delayed sequence it can lead to the cluster crashing. Reclaim Cloud supports sequential restarting of a cluster with a predefined delay. The default is 10 seconds between each node but for Galera clusters we recommend upping this to 30 seconds to avoid issues.
If you find that your Galera cluster has crashed the best way to troubleshoot and get it back online is restart each node individually from the web terminal. Enter one of MariaDB nodes (Ideally the one with the highest uptime, but if uptime is the same, then the primary MariaDB node)
Change the value of safe_to_bootstrap from 0 to 1 in /var/lib/mysql/grastate.dat only on this one node. Here is an example command that can be run from terminal.
sed -i 's/safe_to_bootstrap: 0/safe_to_bootstrap: 1/g' /var/lib/mysql/grastate.dat
Restart MariaDB on this node.
/etc/init.d/mysql restart
Perform the restart of the other two MariaDB nodes
/etc/init.d/mysql restart
You can confirm that the nodes are online by entering the MariaDB console with the following command
mysql -u admin -padmin -h 127.1 -P 6032
and run this command select * from runtime_mysql_servers where status='ONLINE';
Example output:
+--------------+----------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 2 | node3550 | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 3 | node3552 | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 3 | node3551 | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 3 | node3550 | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | node3552 | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | node3551 | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+----------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.01 sec)