Yesterday, three blogs I'm hosting suddenly went offline and alerts went off.

They all had the same error that they couldn't connect to their SQL database, and it seemed that the container recently was auto-updated.

Docker logs also showed (which may or may not have been related):

2024-02-21  7:30:29 97 [Warning] Aborted connection 97 to db: 'unconnected' user: 'unauthenticated' host: '192.168.200.4' (This connection closed normally without authentication)

This was odd. I started the usual troubleshooting:

• As these containers have their own network, maybe the IPs changed and as the SQL users can only access from a certain IP range, something broke there -- nothing seemed to have changed and I could connect from the nginx container to the other just fine
• Password somehow changed -- nah, connections were accepted
• Roll back to a previous version of MariaDB -- didn't do anything, odd
• This was not actually a MariaDB problem but a WordPress problem? Was there an auto-update? Didn't make sense because one of the blogs is running a very old WP version
• Worth noting that I also host two Ghosts blogs using the same setup (nginx+mariadb in container) and they were unaffected and humming along just fine...
• At this point I was starting to think something was massively corrupt and this was gonna be a long day.

I then continued and created a php (testconnection.php) file on the nginx:

<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);

// Connection Details
$hostname = "container-mariadb";  // Your MariaDB hostname 
$username = "your_username"; // Your MariaDB username
$password = "your_password"; // Your MariaDB password
$database = "your_database_name"; // The database

// Create the connection
$conn = new mysqli($hostname, $username, $password, $database);

// Check for errors
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

// Print Debug Information
echo "Connected successfully\n"; 
echo "Host information: " . $conn->host_info . "\n";
echo "Client information: " . $conn->client_info . "\n";

// Close the connection
$conn->close();
?>

This threw the following error... We were getting somewhere:

Fatal error: Uncaught mysqli_sql_exception: Server sent charset (0) unknown to the client. Please, report to the developers in /var/www/domain/testconnection.php:13 Stack trace: #0 /var/www/domain/testconnection.php(13): mysqli->__construct() #1 {main} thrown in /var/www/domain/testconnection.php on line 13

The charset I was using was utf8mb4 with a collate of utf8mb4_general_ci:

SHOW CREATE DATABASE your_database_name;

My wp-config.php file already had DB_CHARSET, but DB_COLLATE was not set. I added that in the config as well:

define( 'DB_CHARSET', 'utf8mb4' );
define( 'DB_COLLATE', 'utf8mb4_general_ci' );

Sadly, that didn't fix it either. Still saying it can't connect to the database. Grmbl.

I then went ahead and checked /etc/mysql/mariadb.conf and the files in /etc/mariadb.conf.d/. It included character-set-server = utf8mb4, but not collation-server = utf8mb4_general_ci. The line was specifically in /etc/mariadb.conf.d/50-server.cnf.

Adding the latter solved the problem... 2hrs of my life gone.

As this file (likely) is managed in the container and changed by the OS (it's apparently based on Ubuntu, ewwww) or the MariaDB version, the easiest way was just creating another file: /etc/mariadb.conf.d/51-<something>.cnf.

This is my very dirty hack (as I'm not using Docker Compose for this):

docker exec -it container-mariadb bash -c "echo -e '[mysqld]\ncollation-server = utf8mb4_general_ci'> /etc/mysql/mariadb.conf.d/51-yeri.cnf"
docker restart container-mariadb

It now works fine again...