Fixing the 'authentication plugin not loaded' issue on Bitnami's MySQL Helm Chart

When you upgrade an existing installation of the Bitnami MySQL Helm Chart from a version such as 10.x to an 11.x version, you might see this error message and have trouble starting your Pods:

SQLSTATE[HY000] [1524] Plugin 'mysql_native_password' is not loaded

The root cause

This is because version 11.x of the Helm Chart uses MySQL 8.4 under the hood, which has removed this plugin and switched to caching_sha2_password instead.

The problem is that each MySQL user has an authentication plugin defined in the database that's used to verify their identity. If the user still has the old plugin configured, they won't be able to log in anymore.

Fixing is not straightforward

To fix this issue, we just need to log into the MySQL database server and change the authentication plugin defined for the user:

I have no name!@mysql-0:/$ mysql -h mysql -u root -p

mysql> SELECT user, plugin FROM mysql.user;
+------------------+-----------------------+
| user             | plugin                |
+------------------+-----------------------+
| root             | mysql_native_password |
| mysql.infoschema | caching_sha2_password |
| mysql.session    | caching_sha2_password |
| mysql.sys        | caching_sha2_password |
+------------------+-----------------------+
4 rows in set (0.01 sec)

As you can see, the user root still has the old authentication plugin defined. But how can we change the authentication plugin, if we can't log in to MySQL anymore?

The workaround

To get this sorted, you need to add the following to your MySQL configuration:

[mysqld]
mysql_native_password=ON

This will re-enable the authentication plugin, so you can log in to fix the problem permanently. However, there's no direct way to add this attribute to the configuration in 11.x of the Helm Chart, so we have to overwrite the whole configuration.

Step 1: Get the current configuration

To get started, you'll need to authenticate against your Kubernetes cluster. Then, run the following command in the namespace where MySQL is installed:

kubectl get configmap mysql -o yaml

The output should look something like this:

apiVersion: v1
kind: ConfigMap
...
data:
  my.cnf: |-
    [mysqld]
    authentication_policy='* ,,'
    skip-name-resolve
    explicit_defaults_for_timestamp
    basedir=/opt/bitnami/mysql
    plugin_dir=/opt/bitnami/mysql/lib/plugin
    port=3306
    mysqlx=0
    mysqlx_port=33060
    socket=/opt/bitnami/mysql/tmp/mysql.sock
    datadir=/bitnami/mysql/data
    tmpdir=/opt/bitnami/mysql/tmp
    max_allowed_packet=16M
    bind-address=*
    pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
    log-error=/opt/bitnami/mysql/logs/mysqld.log
    character-set-server=UTF8
    slow_query_log=0
    long_query_time=10.0

    [client]
    port=3306
    socket=/opt/bitnami/mysql/tmp/mysql.sock
    default-character-set=UTF8
    plugin_dir=/opt/bitnami/mysql/lib/plugin

    [manager]
    port=3306
    socket=/opt/bitnami/mysql/tmp/mysql.sock
    pid-file=/opt/bitnami/mysql/tmp/mysqld.pid

Then copy the whole configuration. In this example, you'll want to copy everything from [mysqld] to pid-file=/opt/bitnami/mysql/tmp/mysqld.pid.

Step 2: Use the new configuration

Next, add mysql_native_password=ON to the [mysqld] section of the current configuration. Then set the new configuration as the value of the primary.configuration attribute of the Helm Chart. If you are using a values.yaml (which I strongly recommend you do), this would look like this:

primary:
  configuration: |-
    [mysqld]
    mysql_native_password=ON
    authentication_policy='* ,,'
    skip-name-resolve
    explicit_defaults_for_timestamp
    basedir=/opt/bitnami/mysql
    plugin_dir=/opt/bitnami/mysql/lib/plugin
    port=3306
    mysqlx=0
    mysqlx_port=33060
    socket=/opt/bitnami/mysql/tmp/mysql.sock
    datadir=/bitnami/mysql/data
    tmpdir=/opt/bitnami/mysql/tmp
    max_allowed_packet=16M
    bind-address=*
    pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
    log-error=/opt/bitnami/mysql/logs/mysqld.log
    character-set-server=UTF8
    slow_query_log=0
    long_query_time=10.0

    [client]
    port=3306
    socket=/opt/bitnami/mysql/tmp/mysql.sock
    default-character-set=UTF8
    plugin_dir=/opt/bitnami/mysql/lib/plugin

    [manager]
    port=3306
    socket=/opt/bitnami/mysql/tmp/mysql.sock
    pid-file=/opt/bitnami/mysql/tmp/mysqld.pid

Now, go ahead and reinstall the Helm Chart in the desired 11.x version using the adaptedvalues.yaml file:

helm upgrade mysql -f values.yaml

Step 3: Adjust the authentication plugin of the user

Once the Pod is restarted with the new configuration, connect to the MySQL Pod:

kubectl exec mysql-0 -it -- /bin/bash

Login with the root user and password:

I have no name!@mysql-0:/$ mysql -h mysql -u root -p

Adjust the configuration of the root user in the database table:

mysql> ALTER USER 'root' IDENTIFIED WITH caching_sha2_password BY '<ROOT_PASSWORD>';

Repeat the last query for each user who has the old authentication plugin configured:

mysql> SELECT user, plugin FROM mysql.user;
+------------------+-----------------------+
| user             | plugin                |
+------------------+-----------------------+
| root             | caching_sha2_password |
| mysql.infoschema | caching_sha2_password |
| mysql.session    | caching_sha2_password |
| mysql.sys        | caching_sha2_password |
+------------------+-----------------------+
4 rows in set (0.01 sec)

Step 4: Cleanup

Finally, remove the primary.configuration section from your values.yaml and reinstall the Helm Chart. All Pods should now start up perfectly.