<?xml version="1.0" encoding="utf-8" ?>
<rss version="2.0"
     xmlns:dc="http://purl.org/dc/elements/1.1/"
     xmlns:content="http://purl.org/rss/1.0/modules/content/">

<channel>
  <title>Planet MySQL</title>
  <link>https://planet.mysql.com</link>
  <pubDate>Sun, 22 Nov 2026 17:51:22 +0000</pubDate>
  <language>en</language>
  <description>Planet MySQL - https://planet.mysql.com</description>

  <item>
    <title>November 2026 MySQL Contributor Summit: Advancing Community Collaboration </title>
    <guid isPermaLink="false">c8899f2ca08e7efdebc7bdc1cb74efc1</guid>
    <link>https://blogs.oracle.com/mysql/november-2026-mysql-contributor-summit-advancing-community-collaboration</link>
    <description>On November 4–5, the MySQL community gathered online for the third MySQL Contributor Summit of 2026. Across two days, contributors shared proposals, demonstrated progress, and discussed how to make MySQL easier to extend, operate, and contribute to.  The summit marked another step in the community engagement plan announced in January. Progress since then includes new […]</description>
    <pubDate>Thu, 19 Nov 2026 21:33:11 +0000</pubDate>
    <dc:creator>Oracle MySQL Group</dc:creator>
    <category>MySQL</category>
    <category>MySQL Community</category>
    <category>News</category>
    <category>mysql</category>
    <category>mysqlcommunity</category>
  </item>

  <item>
    <title>Announcing VillageSQL Server 0.0.8</title>
    <guid isPermaLink="false">6afb4b61a5b39900015d2623</guid>
    <link>https://villagesql.com/blog/008/</link>
    <description>VillageSQL Server 0.0.8 is now available. This release is about using extensions from applications: through prepared statements, with text in other character sets, and with the server protected when an extension function throws an exception.
Highlights of this release include the following:

Functions that take custom types work in prepared statements.
Extensions built with the 0.0.8 SDK can tell raw bytes from text.
When a function in an extension built with the 0.0.8 SDK throws an exception, only that statement fails, and the server keeps running.

Custom types in prepared statements
Many applications keep values out of the SQL text. They send the query once as a prepared statement and pass the values separately, and most MySQL connectors can work this way. In 0.0.8, functions that take custom types work in prepared statements:
mysql&amp;gt; PREPARE re FROM 'SELECT complex_real(?) AS re';
mysql&amp;gt; SET @v = '(1.5,-2.5)';
mysql&amp;gt; EXECUTE re USING @v;
+------+
| re   |
+------+
|  1.5 |
+------+

Vector search works the same way: a nearest-neighbor query with its search value passed as a parameter uses the vector index. Custom indexes are still a preview feature, behind vsql_allow_preview_extensions=ON.
Running a prepared statement many times also costs less. For many extension functions, the server no longer prepares a statement that calls them again at every EXECUTE. For a prepared nearest-neighbor query, that repeat preparation took about 10% of the server's time.
Text and raw bytes
A password hash is raw bytes, and an email address is text. MySQL keeps the two apart, and extensions built with the 0.0.8 SDK can too. A hash reaches the extension byte for byte, and text reaches it in one encoding, whatever character set your client uses.
vsql-crypto 0.0.8 uses the new BINARY type. Its hashes come back as binary data, ready to store in a VARBINARY column:
mysql&amp;gt; SELECT CHARSET(digest('hello', 'sha256')) AS charset;
+---------+
| charset |
+---------+
| binary  |
+---------+

mysql&amp;gt; CREATE TABLE hashes (id INT PRIMARY KEY, h VARBINARY(32));
mysql&amp;gt; INSERT INTO hashes VALUES (1, digest('hello', 'sha256'));
mysql&amp;gt; SELECT id, HEX(h) FROM hashes;
+----+------------------------------------------------------------------+
| id | HEX(h)                                                           |
+----+------------------------------------------------------------------+
|  1 | 2CF24DBA5FB0A30E26E83B2AC5B9E29E1B161E5C1FA7425E73043362938B9824 |
+----+------------------------------------------------------------------+

Custom types read text in other character sets too. Here the COMPLEX type from an example extension reads values from a ucs2 column:
mysql&amp;gt; CREATE TABLE src (id INT PRIMARY KEY, s VARCHAR(40) CHARACTER SET ucs2);
mysql&amp;gt; INSERT INTO src VALUES (2, '(3.0,4.0)');
mysql&amp;gt; CREATE TABLE points (id INT PRIMARY KEY, c COMPLEX);
mysql&amp;gt; INSERT INTO points SELECT id, s FROM src;
Query OK, 1 row affected (0.00 sec)

mysql&amp;gt; SELECT id, c FROM points;
+----+-------+
| id | c     |
+----+-------+
|  2 | (3,4) |
+----+-------+

When an extension throws an exception
Extensions run inside the server process. When a function in an extension built with the 0.0.8 SDK throws a C++ exception, VillageSQL ends that one statement with an error, and the server keeps running. This test extension throws on purpose:
mysql&amp;gt; SELECT vsql_exceptions_test.exception_add(10, 20);
ERROR 3200 (HY000): VDF error in function 'exception_add': VDF threw an exception (vsql_exceptions_test::exception_add)

mysql&amp;gt; SELECT 'server still up' AS status;
+-----------------+
| status          |
+-----------------+
| server still up |
+-----------------+

Crash-safe extension installs
If the server crashes in the middle of INSTALL EXTENSION, it does not leave a half-written extension behind. If an extension file is cut short, INSTALL EXTENSION rejects it. And if an extension update fails at restart, the server rolls back only that extension.
For extension authors
These features come from protocol 5 of the interface between the server and its extensions, which is the stable protocol in 0.0.8. If you built an extension with the development SDK of 0.0.4 through 0.0.7, rebuild it with the stable 0.0.8 SDK.

Function arguments and results can be BINARY, raw bytes that the server never converts, and STRING arguments always arrive as utf8mb4.
A function's prerun check can see the values of constant arguments, including a value bound to a ? in a prepared statement, so a function can accept a NULL literal without accepting every string.
A C++ exception that escapes a function becomes a SQL error on the statement.
A custom type can register a from_binary function beside from_string, to accept values in a binary form, such as a vector of packed floats. The server uses it for a _binary string, a VARBINARY column, or a hex literal.
A function like tvector_dim(?) takes its type parameters from the value bound at EXECUTE.

Upgrading from 0.0.7
0.0.8 does not load extensions built for protocol 4, so if your 0.0.7 server has one installed, the server will not start after you upgrade. To check whether any of your installed extensions use protocol 4, run this query on your 0.0.7 server:
SELECT EXTENSION_NAME FROM INFORMATION_SCHEMA.EXTENSION_REGISTRATION WHERE NEGOTIATED_PROTOCOL = 4;
To upgrade, dump your data with mysqldump, install your extensions on a new 0.0.8 data directory, and then load the dump.
Bundled extensions
The install script, the Docker image, and the release tarballs include these extensions with VillageSQL Server 0.0.8. To use one, run INSTALL EXTENSION with its name.

vsql-ai 0.0.8
vsql-boolean 0.0.3
vsql-crypto 0.0.8
vsql-cube 0.0.8
vsql-fuzzystrmatch 0.0.2
vsql-http 0.0.4
vsql-network-address 0.0.8
vsql-oauth2 0.0.5
vsql-rest 0.0.11
vsql-trgm 0.0.5
vsql-uuid 0.0.9

You can find a list of all available extensions here. To build a new extension or port an existing PostgreSQL extension, you can use an agent skill.
Special thanks
Thanks to @kash2104, who made the server print &amp;lt;error decoding value&amp;gt; for a custom value it cannot decode, and cleaned up how the server reads raw custom values.
Thanks to @jobala, whose work lets an extension function's prerun check tell a NULL literal from other values, and to @askdba, who added a MyVector profile to vector-dev-bench, our tool for measuring in-database vector indexes.
There's more in this release than fits here. The release notes carry the per-PR detail. Our roadmap is published on GitHub, and we'd like to hear what you're building and what we should work on next, either on GitHub or on Discord. If you want to support VillageSQL, please star the repo.</description>
    <content:encoded><![CDATA[<img src="https://storage.ghost.io/c/db/c7/dbc78ca4-dcfe-468b-b2fa-030396d2a98e/content/images/2026/11/IMG_9645.jpg" alt="Announcing VillageSQL Server 0.0.8"><p>VillageSQL Server <a href="https://github.com/villagesql/villagesql-server">0.0.8 is now available</a>. This release is about using extensions from applications: through prepared statements, with text in other character sets, and with the server protected when an extension function throws an exception.</p>
<p>Highlights of this release include the following:</p>
<ul>
<li>Functions that take custom types work in prepared statements.</li>
<li>Extensions built with the 0.0.8 SDK can tell raw bytes from text.</li>
<li>When a function in an extension built with the 0.0.8 SDK throws an exception, only that statement fails, and the server keeps running.</li>
</ul>
<h2>Custom types in prepared statements</h2>
<p>Many applications keep values out of the SQL text. They send the query once as a prepared statement and pass the values separately, and most MySQL connectors can work this way. In 0.0.8, functions that take custom types work in prepared statements:</p>
<pre><code class="language-sql">mysql&gt; PREPARE re FROM 'SELECT complex_real(?) AS re';
mysql&gt; SET @v = '(1.5,-2.5)';
mysql&gt; EXECUTE re USING @v;
+------+
| re   |
+------+
|  1.5 |
+------+
</code></pre>
<p>Vector search works the same way: a nearest-neighbor query with its search value passed as a parameter uses the vector index. Custom indexes are still a preview feature, behind <code>vsql_allow_preview_extensions=ON</code>.</p>
<p>Running a prepared statement many times also costs less. For many extension functions, the server no longer prepares a statement that calls them again at every <code>EXECUTE</code>. For a prepared nearest-neighbor query, that repeat preparation took about 10% of the server's time.</p>
<h2>Text and raw bytes</h2>
<p>A password hash is raw bytes, and an email address is text. MySQL keeps the two apart, and extensions built with the 0.0.8 SDK can too. A hash reaches the extension byte for byte, and text reaches it in one encoding, whatever character set your client uses.</p>
<p><a href="https://github.com/villagesql/vsql-crypto">vsql-crypto</a> 0.0.8 uses the new <code>BINARY</code> type. Its hashes come back as binary data, ready to store in a <code>VARBINARY</code> column:</p>
<pre><code class="language-sql">mysql&gt; SELECT CHARSET(digest('hello', 'sha256')) AS charset;
+---------+
| charset |
+---------+
| binary  |
+---------+

mysql&gt; CREATE TABLE hashes (id INT PRIMARY KEY, h VARBINARY(32));
mysql&gt; INSERT INTO hashes VALUES (1, digest('hello', 'sha256'));
mysql&gt; SELECT id, HEX(h) FROM hashes;
+----+------------------------------------------------------------------+
| id | HEX(h)                                                           |
+----+------------------------------------------------------------------+
|  1 | 2CF24DBA5FB0A30E26E83B2AC5B9E29E1B161E5C1FA7425E73043362938B9824 |
+----+------------------------------------------------------------------+
</code></pre>
<p>Custom types read text in other character sets too. Here the <code>COMPLEX</code> type from an example extension reads values from a <code>ucs2</code> column:</p>
<pre><code class="language-sql">mysql&gt; CREATE TABLE src (id INT PRIMARY KEY, s VARCHAR(40) CHARACTER SET ucs2);
mysql&gt; INSERT INTO src VALUES (2, '(3.0,4.0)');
mysql&gt; CREATE TABLE points (id INT PRIMARY KEY, c COMPLEX);
mysql&gt; INSERT INTO points SELECT id, s FROM src;
Query OK, 1 row affected (0.00 sec)

mysql&gt; SELECT id, c FROM points;
+----+-------+
| id | c     |
+----+-------+
|  2 | (3,4) |
+----+-------+
</code></pre>
<h2>When an extension throws an exception</h2>
<p>Extensions run inside the server process. When a function in an extension built with the 0.0.8 SDK throws a C++ exception, VillageSQL ends that one statement with an error, and the server keeps running. This test extension throws on purpose:</p>
<pre><code class="language-sql">mysql&gt; SELECT vsql_exceptions_test.exception_add(10, 20);
ERROR 3200 (HY000): VDF error in function 'exception_add': VDF threw an exception (vsql_exceptions_test::exception_add)

mysql&gt; SELECT 'server still up' AS status;
+-----------------+
| status          |
+-----------------+
| server still up |
+-----------------+
</code></pre>
<h2>Crash-safe extension installs</h2>
<p>If the server crashes in the middle of <code>INSTALL EXTENSION</code>, it does not leave a half-written extension behind. If an extension file is cut short, INSTALL EXTENSION rejects it. And if an extension update fails at restart, the server rolls back only that extension.</p>
<h2>For extension authors</h2>
<p>These features come from protocol 5 of the interface between the server and its extensions, which is the stable protocol in 0.0.8. If you built an extension with the development SDK of 0.0.4 through 0.0.7, rebuild it with the stable 0.0.8 SDK.</p>
<ul>
<li>Function arguments and results can be <code>BINARY</code>, raw bytes that the server never converts, and <code>STRING</code> arguments always arrive as <code>utf8mb4</code>.</li>
<li>A function's <code>prerun</code> check can see the values of constant arguments, including a value bound to a ? in a prepared statement, so a function can accept a <code>NULL</code> literal without accepting every string.</li>
<li>A C++ exception that escapes a function becomes a SQL error on the statement.</li>
<li>A custom type can register a <code>from_binary</code> function beside <code>from_string</code>, to accept values in a binary form, such as a vector of packed floats. The server uses it for a _binary string, a VARBINARY column, or a hex literal.</li>
<li>A function like <code>tvector_dim(?)</code> takes its type parameters from the value bound at <code>EXECUTE</code>.</li>
</ul>
<h2>Upgrading from 0.0.7</h2>
<p>0.0.8 does not load extensions built for protocol 4, so if your 0.0.7 server has one installed, the server will not start after you upgrade. To check whether any of your installed extensions use protocol 4, run this query on your 0.0.7 server:</p>
<p><code>SELECT EXTENSION_NAME FROM INFORMATION_SCHEMA.EXTENSION_REGISTRATION WHERE NEGOTIATED_PROTOCOL = 4;</code></p>
<p>To upgrade, dump your data with <code>mysqldump</code>, install your extensions on a new 0.0.8 data directory, and then load the dump.</p>
<h2>Bundled extensions</h2>
<p>The install script, the Docker image, and the release tarballs include these extensions with VillageSQL Server 0.0.8. To use one, run <code>INSTALL EXTENSION</code> with its name.</p>
<ul>
<li><a href="https://github.com/villagesql/vsql-ai/releases/tag/0.0.8"><strong>vsql-ai</strong></a> 0.0.8</li>
<li><a href="https://github.com/villagesql/vsql-boolean/releases/tag/0.0.3"><strong>vsql-boolean</strong></a> 0.0.3</li>
<li><a href="https://github.com/villagesql/vsql-crypto/releases/tag/0.0.8"><strong>vsql-crypto</strong></a> 0.0.8</li>
<li><a href="https://github.com/villagesql/vsql-cube/releases/tag/0.0.8"><strong>vsql-cube</strong></a> 0.0.8</li>
<li><a href="https://github.com/villagesql/vsql-fuzzystrmatch/releases/tag/0.0.2"><strong>vsql-fuzzystrmatch</strong></a> 0.0.2</li>
<li><a href="https://github.com/villagesql/vsql-http/releases/tag/0.0.4"><strong>vsql-http</strong></a> 0.0.4</li>
<li><a href="https://github.com/villagesql/vsql-network-address/releases/tag/0.0.8"><strong>vsql-network-address</strong></a> 0.0.8</li>
<li><a href="https://github.com/villagesql/vsql-oauth2/releases/tag/0.0.5"><strong>vsql-oauth2</strong></a> 0.0.5</li>
<li><a href="https://github.com/villagesql/vsql-rest/releases/tag/0.0.11"><strong>vsql-rest</strong></a> 0.0.11</li>
<li><a href="https://github.com/villagesql/vsql-trgm/releases/tag/0.0.5"><strong>vsql-trgm</strong></a> 0.0.5</li>
<li><a href="https://github.com/villagesql/vsql-uuid/releases/tag/0.0.9"><strong>vsql-uuid</strong></a> 0.0.9</li>
</ul>
<p>You can find a list of <a href="https://villagesql.com/docs/en/extensions/catalog">all available extensions here</a>. To build a new extension or port an existing PostgreSQL extension, you can <a href="https://villagesql.com/agent-skills">use an agent skill</a>.</p>
<h2>Special thanks</h2>
<p>Thanks to <a href="https://github.com/kash2104">@kash2104</a>, who made the server print <code>&lt;error decoding value&gt;</code> for a custom value it cannot decode, and cleaned up how the server reads raw custom values.</p>
<p>Thanks to <a href="https://github.com/jobala">@jobala</a>, whose work lets an extension function's prerun check tell a NULL literal from other values, and to <a href="https://github.com/askdba">@askdba</a>, who added a MyVector profile to <a href="https://github.com/villagesql/vector-dev-bench">vector-dev-bench</a>, our tool for measuring in-database vector indexes.</p>
<p>There's more in this release than fits here. The <a href="https://github.com/villagesql/villagesql-server/blob/main/Docs/release_notes/release_note_0_0_8.md">release notes</a> carry the per-PR detail. Our roadmap is published on <a href="https://github.com/orgs/villagesql/projects/1/views/4">GitHub</a>, and we'd like to hear what you're building and what we should work on next, either on <a href="https://github.com/villagesql/villagesql-server/issues">GitHub</a> or on <a href="https://discord.gg/KSr6whd3Fr">Discord</a>. If you want to support VillageSQL, <a href="https://github.com/villagesql/villagesql-server">please star the repo</a>.</p>]]></content:encoded>
    <pubDate>Thu, 19 Nov 2026 14:00:00 +0000</pubDate>
    <dc:creator>VillageSQL</dc:creator>
    <category>VillageSQL</category>
    <category>MySQL</category>
    <category>Open Source</category>
    <category>AI</category>
  </item>

  <item>
    <title>Modernize on Your Terms: How Nutanix and AWS Accelerate Migrations, Simplify Operations, and Enable AI</title>
    <guid isPermaLink="false">https://www.nutanix.com/blog/aws-re-invent-2026-accelerate-migrations-and-ai</guid>
    <link>https://www.nutanix.com/blog/aws-re-invent-2026-accelerate-migrations-and-ai</link>
    <description>Learn how Nutanix and AWS help organizations migrate workloads without refactoring, simplify hybrid cloud operations, and build a flexible foundation for AI.</description>
    <pubDate>Thu, 19 Nov 2026 07:56:00 +0000</pubDate>
    <dc:creator>Nutanix</dc:creator>
    <category>Nutanix Cloud Infrastructure (NCI)</category>
    <category>Nutanix Enterprise AI (NAI)</category>
    <category>Nutanix Cloud Clusters (NC2)</category>
    <category>TOF - Awareness</category>
    <category>Hybrid Multicloud</category>
    <category>VMware Alternative   Broadcom Compete</category>
    <category>Migrate Apps to Cloud</category>
    <category>Nutanix Database Service (NDB)</category>
    <category>Private Cloud</category>
    <category>Unified Storage</category>
    <category>Blog Cat</category>
  </item>

  <item>
    <title>Why MySQL OR is slower than IN</title>
    <guid isPermaLink="false">https://www.percona.com/?p=56086</guid>
    <link>https://www.percona.com/blog/why-mysql-or-is-slower-than-in/</link>
    <description>Queries with OR conditions are commonly used to implement business logic. 
However, it might be slower than we expected. 
Although the query below scans only 1 million rows, it runs for 6,7254 seconds.# User@Host: msandbox[msandbox] @ localhost [127.0.0.1]  Id:   188
# Schema:   Last_errno: 1160  Killed: 0
# Query_time: 67254.081455  Lock_time: 0.000006  Rows_sent: 1  Rows_examined: 1000000  Rows_affected: 0  Bytes_sent: 0
...
SELECT COUNT(*) FROM db1.sbtest1 WHERE id = 1 OR id = 2 OR id = 3 OR id = 4 OR id = 5 OR id = 6 OR id = 7 OR id = 8 OR id = 9 ...
...The CPU profile indicates that 99% of CPU time was spent in 
			Item_cond_or::val_int() on MySQL 8.4.connection
└ mysql_execute_command → Sql_cmd_dml::execute → Query_expression::ExecuteIteratorQuery
  └ AggregateIterator::Read                                99.80%
    └ FilterIterator::Read                                 99.80%
      ├ Item_cond_or::val_int                              98.96%
      │ └ Item::val_bool → Item_func_eq::val_int           91.27%
      │   └ Arg_comparator::compare_int_signed             58.46%
      │     ├ Item_field::val_int                          16.18%
      │     └ (literal) val_int                            10.04%
      └ IndexScanIterator::Read → row_search_mvcc           0.02%
How MySQL evaluates OR conditions
MySQL turns 
			id = 1 OR id = 2 OR ... OR id = N into one 
			Item_cond_or holding N 
			Item_func_eq  elements.longlong Item_cond_or::val_int() {
  assert(fixed);
  List_iterator_fast&amp;lt;Item&amp;gt; li(list);
  Item *item;
  null_value = false;
  while ((item = li++)) {
    if (item-&amp;gt;val_bool()) {
      null_value = false;
      return 1;
    }
    if (item-&amp;gt;null_value) null_value = true;
    if (current_thd-&amp;gt;is_error()) return error_int();
  }
  return 0;
}It checks each 
			OR  element in the list until it finds a match.

If a row matches term k, it costs k.
If a row matches nothing, it costs N.

The cost of the linear search is 
			O(N).
How MySQL evaluates IN conditions
bool in_longlong::find_item(Item *item) {
  if (m_used_size == 0) return false;
  packed_longlong result;
  val_item(item, &amp;amp;result);
  if (item-&amp;gt;null_value) return false;
  return std::binary_search(base.begin(), base.begin() + m_used_size, result,
                            Cmp_longlong());
}The cost of the binary search is 
			O(log N).
The cost of OR conditions
MySQL evaluates OR conditions in a nested loop.
The outer loop walks the rows from the table, and the inner loop walks the OR list:for each row r:                 -- outer loop: R rows
    for each term o:            -- inner loop: up to N terms
        if r == o:
            break               -- exit inner loop; continue with next rowAssume that

N is the number of values in the OR list
R is the number of rows the filter evaluates
T is the number of rows in the table

A row that matches term k leaves the inner loop after k checks, which is about N/2 on average.
A row that matches nothing runs all N.
The OR list costs 
			O(R × N).
R depends on the query’s access type.
How access type impacts the cost
Range access: O(N²)
When the optimizer chooses 
			range access type, it reads only the matching keys from the index:mysql &amp;gt; explain for connection 11\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: sbtest1
   partitions: NULL
         type: range
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 8
          ref: NULL
         rows: 34753
     filtered: 100.00
        Extra: Using where; Using index
1 row in set (0.00 sec)The range has already done the filtering, but the server checks each row again, term by term.
R grows with N, and each row checks about N/2 terms: R × N/2 ≈ N²/2.
R is equal to N when all OR conditions in the table exist.
In general, the cost is O(N²).
Since the table size T does not appear in this cost, the increasing number of rows does not increase the execution time.
The same query on tables of 100k, 500k, and 1M rows takes about the same time:

Although the number of rows in the table grows 10 times from 100k to 1M, the execution time is still close.
Range access reads only the N matching rows, so only N matters.
However, R is not always equal to N.
R is less than N when only part of the list exists in the table.
R is greater than N when each value matches several rows.

In this case, the cost is O(p × N²), where p is the percentage of rows matched from the table.
Index scan: O(T × N)
When the optimizer chooses 
			index access type, it reads all records from the index:mysql &amp;gt; explain for connection 11\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: sbtest1
   partitions: NULL
         type: index
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 8
          ref: NULL
        rows: 1000000
     filtered: 100.00
        Extra: Using where; Using index
1 row in set (0.00 sec)Now R = T. Every row in the table goes through the inner loop, and the T − N rows that match nothing run all N terms.
The cost is O(T × N). T does not change as N grows, so the cost is linear in N.
Linear does not mean cheaper, because T is much larger than N.
At N = 40,000 on this table of about 1M rows, range access checks about N²/2 = 0.8 billion terms.
The index scan checks about T × N = 40 billion, roughly 50 times more.

Going from 34,753 OR conditions to 34,754 OR conditions pushed the ranges past the limit.
The OR query went from 37 s to 2,907 s, which is 79 times longer.
 
In the 
			index  access type, the table size T is part of this cost, and the test shows it.

The same 40,000-value OR query with an index scan took 3,406 s on a 1M-row table, 1,752 s on a 500k-row table, and 265 s on a 100k-row table.
Full table scan: O(T × N)
This is similar to the index scan, which occurs when no suitable index is available for the query.mysql&amp;gt; explain for connection 9\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: sbtest1
   partitions: NULL
         type: ALL
possible_keys: NULL
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 1000000
     filtered: 100.00
        Extra: Using where
1 row in set (0.01 sec)
Why does the optimizer switch from range to index
Reason 1: the 
			range_optimizer_max_mem_size  (8 MB by default) limit is exceeded.mysql &amp;gt; show warnings limit 1\G
*************************** 1. row ***************************
  Level: Warning
   Code: 3170
Message: Memory capacity of 8388608 bytes for 'range_optimizer_max_mem_size' exceeded. Range optimization was not done for this query.
1 row in set (0.00 sec)In this case, the server raises warning 3170 and does not use range access.
Please read more blogs about it
Low Value in range_optimizer_max_mem_size Might Cause Full Table Scan
Why Are Queries with Many IN Values More Expensive After Upgrading to MySQL 8.x?
Reason 2: the cost model thinks the index scan is cheaper.
The optimizer trace indicates it does not choose the range scan because of its cost.&quot;range_scan_alternatives&quot;: [
      {
        &quot;index&quot;: &quot;PRIMARY&quot;,
        &quot;ranges&quot;: [
          &quot;id = 1&quot;,
          ...
          &quot;id = 9998&quot;,
          &quot;id = 9999&quot;,
          &quot;id = 10000&quot;
        ],
        &quot;index_dives_for_eq_ranges&quot;: false,
        &quot;rowid_ordered&quot;: true,
        &quot;using_mrr&quot;: false,
        &quot;index_only&quot;: true,
        &quot;in_memory&quot;: 0.000709723,
        &quot;rows&quot;: 10000,
        &quot;cost&quot;: 11006.7,
        &quot;chosen&quot;: false,
        &quot;cause&quot;: &quot;cost&quot;
      }
The cost of IN conditions
In general, the cost of evaluating the entire IN condition is O(R log N). 
Similar to OR conditions, R depends on the query’s access type.
After rewriting the query with IN, the execution time is below.

Conclusion
Since the time complexity of OR conditions is more expensive than that of IN, the query with OR conditions is slower than the query with IN conditions.
 
The post Why MySQL OR is slower than IN appeared first on Percona.</description>
    <content:encoded><![CDATA[<p><span><span data-d-streaming-word="">Queries</span> with <code data-d-component="code" class="x1ghz6dp x1717udv x1wty727 x4gjf66 xdj266r x15eiy0f x1lziwak x15472z4 x15bjb6t xt0psk2 x1bng3s8 x11xefk2 x109r6dy xptor0k x1gojbwh xk50ysn x17etxam x1cwmb0u xlzvbmf"><span data-d-streaming-word="">OR</span></code> <span data-d-streaming-word="">conditions</span> <span data-d-streaming-word="">are</span> <span data-d-streaming-word="">commonly</span> <span data-d-streaming-word="">used</span> <span data-d-streaming-word="">to</span> <span data-d-streaming-word="">implement</span> <span data-d-streaming-word="">business</span> <span data-d-streaming-word="">logic. </span></span></p>
<p><span><span data-d-streaming-word="">However, it might be slower than we expected. </span></span></p>
<p>Although the query below scans only 1 million rows, it runs for 6,7254 seconds.</p><pre class="urvanov-syntax-highlighter-plain-tag"># User@Host: msandbox[msandbox] @ localhost [127.0.0.1]  Id:   188
# Schema:   Last_errno: 1160  Killed: 0
# Query_time: 67254.081455  Lock_time: 0.000006  Rows_sent: 1  Rows_examined: 1000000  Rows_affected: 0  Bytes_sent: 0
...
SELECT COUNT(*) FROM db1.sbtest1 WHERE id = 1 OR id = 2 OR id = 3 OR id = 4 OR id = 5 OR id = 6 OR id = 7 OR id = 8 OR id = 9 ...
...</pre><p><span>The CPU profile indicates that </span><span>99%</span><span> of CPU time was spent in 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-v">Item_cond_or</span><span class="crayon-o">::</span><span class="crayon-e">val_int</span><span class="crayon-sy">(</span><span class="crayon-sy">)</span></span></span></span><span> on MySQL 8.4</span><span>.</span></p><pre class="urvanov-syntax-highlighter-plain-tag">connection
└ mysql_execute_command → Sql_cmd_dml::execute → Query_expression::ExecuteIteratorQuery
  └ AggregateIterator::Read                                99.80%
    └ FilterIterator::Read                                 99.80%
      ├ Item_cond_or::val_int                              98.96%
      │ └ Item::val_bool → Item_func_eq::val_int           91.27%
      │   └ Arg_comparator::compare_int_signed             58.46%
      │     ├ Item_field::val_int                          16.18%
      │     └ (literal) val_int                            10.04%
      └ IndexScanIterator::Read → row_search_mvcc           0.02%</pre><p></p>
<h2><span>How MySQL evaluates OR conditions</span></h2>
<p><span>MySQL turns 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-v">id</span><span class="crayon-h"> </span><span class="crayon-o">=</span><span class="crayon-h"> </span><span class="crayon-cn">1</span><span class="crayon-h"> </span><span class="crayon-st">OR</span><span class="crayon-h"> </span><span class="crayon-v">id</span><span class="crayon-h"> </span><span class="crayon-o">=</span><span class="crayon-h"> </span><span class="crayon-cn">2</span><span class="crayon-h"> </span><span class="crayon-st">OR</span><span class="crayon-h"> </span><span class="crayon-sy">.</span><span class="crayon-sy">.</span><span class="crayon-sy">.</span><span class="crayon-h"> </span><span class="crayon-st">OR</span><span class="crayon-h"> </span><span class="crayon-v">id</span><span class="crayon-h"> </span><span class="crayon-o">=</span><span class="crayon-h"> </span><span class="crayon-v">N</span></span></span> </span><span>into one 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-v">Item_cond_or</span></span></span> </span><span>holding </span><span>N</span> 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-v">Item_func_eq</span></span></span> <span> elements.</span></p><pre class="urvanov-syntax-highlighter-plain-tag">longlong Item_cond_or::val_int() {
  assert(fixed);
  List_iterator_fast&lt;Item&gt; li(list);
  Item *item;
  null_value = false;
  while ((item = li++)) {
    if (item-&gt;val_bool()) {
      null_value = false;
      return 1;
    }
    if (item-&gt;null_value) null_value = true;
    if (current_thd-&gt;is_error()) return error_int();
  }
  return 0;
}</pre><p><span>It checks each 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-st">OR</span></span></span> </span><span> element in the list until it finds a match.</span></p>
<ul>
<li aria-level="1"><span>If a row matches term </span><span>k</span><span>, it costs </span><span>k</span><span>.</span></li>
<li aria-level="1"><span>If a row matches nothing, it costs </span><span>N</span><span>.</span></li>
</ul>
<p><span>The cost of the linear search is 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-e">O</span><span class="crayon-sy">(</span><span class="crayon-v">N</span><span class="crayon-sy">)</span></span></span>.</span></p>
<h2><span>How MySQL evaluates IN conditions</span></h2>
<p></p><pre class="urvanov-syntax-highlighter-plain-tag">bool in_longlong::find_item(Item *item) {
  if (m_used_size == 0) return false;
  packed_longlong result;
  val_item(item, &amp;result);
  if (item-&gt;null_value) return false;
  return std::binary_search(base.begin(), base.begin() + m_used_size, result,
                            Cmp_longlong());
}</pre><p><span>The cost of the binary search is 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-e">O</span><span class="crayon-sy">(</span><span class="crayon-i">log</span><span class="crayon-h"> </span><span class="crayon-v">N</span><span class="crayon-sy">)</span></span></span>.</span></p>
<h2><span>The cost of OR conditions</span></h2>
<p><span>MySQL evaluates </span><span>OR</span><span> conditions in a nested loop.</span></p>
<p><span>The outer loop walks the rows from the table, and the inner loop walks the </span><span>OR</span><span> list:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">for each row r:                 -- outer loop: R rows
    for each term o:            -- inner loop: up to N terms
        if r == o:
            break               -- exit inner loop; continue with next row</pre><p><span>Assume that</span></p>
<ul>
<li aria-level="1"><span>N</span><span> is the number of values in the </span><span>OR</span><span> list</span></li>
<li aria-level="1"><span>R</span><span> is the number of rows the filter evaluates</span></li>
<li aria-level="1"><span>T</span><span> is the number of rows in the table</span></li>
</ul>
<p><span>A row that matches term </span><span>k</span><span> leaves the inner loop after </span><span>k</span><span> checks, which is about </span><span>N/2</span><span> on average.</span></p>
<p><span>A row that matches nothing runs all </span><span>N</span><span>.</span></p>
<p><span>The OR list costs 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-e">O</span><span class="crayon-sy">(</span><span class="crayon-i">R</span><span class="crayon-h"> </span>×<span class="crayon-h"> </span><span class="crayon-v">N</span><span class="crayon-sy">)</span></span></span>.</span></p>
<p><span>R depends on the query’s access type.</span></p>
<h2><span>How access type impacts the cost</span></h2>
<h3><span>Range access: O(N²)</span></h3>
<p><span>When the optimizer chooses 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-v">range</span></span></span></span><span> access type, it reads only the matching keys from the index:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">mysql &gt; explain for connection 11\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: sbtest1
   partitions: NULL
         type: range
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 8
          ref: NULL
         rows: 34753
     filtered: 100.00
        Extra: Using where; Using index
1 row in set (0.00 sec)</pre><p><span>The range has already done the filtering, but the server checks each row again, term by term.</span></p>
<p><span>R</span><span> grows with </span><span>N</span><span>, and each row checks about N/2 terms: R × N/2 ≈ N²/2.</span></p>
<p><span>R</span><span> is equal to </span><span>N</span><span> when all OR conditions in the table exist.</span></p>
<p><span>In general, the cost is </span><span>O(N²)</span><span>.</span></p>
<p><span>Since the table size </span><span>T</span><span> does not appear in this cost, the increasing number of rows does not increase the execution time.</span></p>
<p><span>The same query on tables of 100k, 500k, and 1M rows takes about the same time:</span></p>
<p><img fetchpriority="high" decoding="async" class="alignnone size-full wp-image-56088" src="https://www.percona.com/wp-content/uploads/2026/11/ON%C2%B2.png" alt="" width="600" height="371" srcset="https://www.percona.com/wp-content/uploads/2026/11/ON².png 600w, https://www.percona.com/wp-content/uploads/2026/11/ON²-300x186.png 300w" sizes="(max-width: 600px) 100vw, 600px"></p>
<p><span>Although the number of rows in the table grows 10 times from 100k to 1M, the execution time is still close.</span></p>
<p><span>Range access reads only the </span><span>N</span><span> matching rows, so only </span><span>N</span><span> matters.</span></p>
<p><span>However, </span><span>R</span><span> is not always equal to </span><span>N</span><span>.</span></p>
<p><span>R</span><span> is less than </span><span>N</span><span> when only part of the list exists in the table.</span></p>
<p><span>R</span><span> is greater than </span><span>N</span><span> when each value matches several rows.</span></p>
<p><img decoding="async" class="alignnone size-large wp-image-56089" src="https://www.percona.com/wp-content/uploads/2026/11/R_percentage-1024x571.png" alt="" width="1024" height="571" srcset="https://www.percona.com/wp-content/uploads/2026/11/R_percentage-1024x571.png 1024w, https://www.percona.com/wp-content/uploads/2026/11/R_percentage-300x167.png 300w, https://www.percona.com/wp-content/uploads/2026/11/R_percentage-768x429.png 768w, https://www.percona.com/wp-content/uploads/2026/11/R_percentage.png 1222w" sizes="(max-width: 1024px) 100vw, 1024px"></p>
<p><span>In this case, the cost is O(p × N²), where p is the percentage of rows matched from the table.</span></p>
<h3><span>Index scan: O(T × N)</span></h3>
<p><span>When the optimizer chooses 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-v">index</span></span></span> </span><span>access type, it reads all records from the index:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">mysql &gt; explain for connection 11\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: sbtest1
   partitions: NULL
         type: index
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 8
          ref: NULL
        rows: 1000000
     filtered: 100.00
        Extra: Using where; Using index
1 row in set (0.00 sec)</pre><p><span>Now R = T. Every row in the table goes through the inner loop, and the T − N rows that match nothing run all N terms.</span></p>
<p><span>The cost is O(T × N). T does not change as N grows, so the cost is linear in N.</span></p>
<p><span>Linear does not mean cheaper, because T is much larger than N.</span></p>
<p><span>At N = 40,000 on this table of about 1M rows, range access checks about N²/2 = 0.8 billion terms.</span></p>
<p><span>The index scan checks about T × N = 40 billion, roughly 50 times more.</span></p>
<p><img decoding="async" class="alignnone size-large wp-image-56090" src="https://www.percona.com/wp-content/uploads/2026/11/OTN-1024x658.png" alt="" width="1024" height="658" srcset="https://www.percona.com/wp-content/uploads/2026/11/OTN-1024x658.png 1024w, https://www.percona.com/wp-content/uploads/2026/11/OTN-300x193.png 300w, https://www.percona.com/wp-content/uploads/2026/11/OTN-768x493.png 768w, https://www.percona.com/wp-content/uploads/2026/11/OTN.png 1404w" sizes="(max-width: 1024px) 100vw, 1024px"></p>
<p><span>Going from 34,753 OR conditions to 34,754 OR conditions pushed the ranges past the limit.</span></p>
<p><span>The OR query went from 37 s to 2,907 s, which is 79 times longer.</span></p>
<p><span> </span></p>
<p><span>In the 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-v">index</span></span></span> </span><span> access type, the table size </span><span>T</span><span> is part of this cost, and the test shows it.</span></p>
<p><img loading="lazy" decoding="async" class="alignnone size-full wp-image-56091" src="https://www.percona.com/wp-content/uploads/2026/11/TN.png" alt="" width="600" height="371" srcset="https://www.percona.com/wp-content/uploads/2026/11/TN.png 600w, https://www.percona.com/wp-content/uploads/2026/11/TN-300x186.png 300w" sizes="auto, (max-width: 600px) 100vw, 600px"></p>
<p><span>The same 40,000-value OR query with an index scan took </span><span>3,406</span><span> s on a 1M-row table, </span><span>1,752</span><span> s on a 500k-row table, and </span><span>265</span><span> s on a 100k-row table.</span></p>
<h3><span>Full table scan: O(T × N)</span></h3>
<p><span>This is similar to the index scan, which occurs when no suitable index is available for the query.</span></p><pre class="urvanov-syntax-highlighter-plain-tag">mysql&gt; explain for connection 9\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: sbtest1
   partitions: NULL
         type: ALL
possible_keys: NULL
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 1000000
     filtered: 100.00
        Extra: Using where
1 row in set (0.01 sec)</pre><p></p>
<h2><span data-d-streaming-word="">Why</span> does the <span data-d-streaming-word="">optimizer</span> <span data-d-streaming-word="">switch from </span><span data-d-streaming-word="">range </span><span data-d-streaming-word="">to </span><span data-d-streaming-word="">index</span></h2>
<p><span>Reason 1: the 
			<span class="urvanov-syntax-highlighter-syntax urvanov-syntax-highlighter-syntax-inline  crayon-theme-classic crayon-theme-classic-inline urvanov-syntax-highlighter-font-monaco"><span class="crayon-pre urvanov-syntax-highlighter-code"><span class="crayon-v">range_optimizer_max_mem_size</span></span></span> </span><span> (8 MB by default) limit is exceeded.</span></p><pre class="urvanov-syntax-highlighter-plain-tag">mysql &gt; show warnings limit 1\G
*************************** 1. row ***************************
  Level: Warning
   Code: 3170
Message: Memory capacity of 8388608 bytes for 'range_optimizer_max_mem_size' exceeded. Range optimization was not done for this query.
1 row in set (0.00 sec)</pre><p><span>In this case, the server raises warning </span><span>3170</span><span> and does not use range access.</span></p>
<p><span>Please read more blogs about it</span></p>
<p><a href="https://www.percona.com/blog/low-value-in-range_optimizer_max_mem_size-might-cause-full-table-scan/">Low Value in range_optimizer_max_mem_size Might Cause Full Table Scan</a></p>
<p><a href="https://www.percona.com/blog/why-are-queries-with-many-in-values-more-expensive-after-upgrading-to-mysql-8-x/">Why Are Queries with Many IN Values More Expensive After Upgrading to MySQL 8.x?</a></p>
<p><span>Reason 2: the cost model thinks the index scan is cheaper.</span></p>
<p><span>The optimizer trace indicates it does not choose the </span><span>range</span><span> scan because of its cost.</span></p><pre class="urvanov-syntax-highlighter-plain-tag">"range_scan_alternatives": [
      {
        "index": "PRIMARY",
        "ranges": [
          "id = 1",
          ...
          "id = 9998",
          "id = 9999",
          "id = 10000"
        ],
        "index_dives_for_eq_ranges": false,
        "rowid_ordered": true,
        "using_mrr": false,
        "index_only": true,
        "in_memory": 0.000709723,
        "rows": 10000,
        "cost": 11006.7,
        "chosen": false,
        "cause": "cost"
      }</pre><p></p>
<h2><span>The cost of IN conditions</span></h2>
<p><span>In general, the cost of evaluating the entire </span><span>IN</span><span> condition is O(R log N). </span></p>
<p><span>Similar to </span><span>OR</span><span> conditions, R depends on the query’s access type.</span></p>
<p><span>After rewriting the query with IN, the execution time is below.</span></p>
<p><img loading="lazy" decoding="async" class="alignnone size-large wp-image-56092" src="https://www.percona.com/wp-content/uploads/2026/11/IN-1024x483.png" alt="" width="1024" height="483" srcset="https://www.percona.com/wp-content/uploads/2026/11/IN-1024x483.png 1024w, https://www.percona.com/wp-content/uploads/2026/11/IN-300x142.png 300w, https://www.percona.com/wp-content/uploads/2026/11/IN-768x362.png 768w, https://www.percona.com/wp-content/uploads/2026/11/IN.png 1518w" sizes="auto, (max-width: 1024px) 100vw, 1024px"></p>
<h2><span>Conclusion</span></h2>
<p><span>Since the time complexity of OR conditions is more expensive than that of IN, the query with OR conditions is slower than the query with IN conditions.</span></p>
<p> </p>
<p>The post <a href="https://www.percona.com/blog/why-mysql-or-is-slower-than-in/">Why MySQL OR is slower than IN</a> appeared first on <a href="https://www.percona.com/">Percona</a>.</p>]]></content:encoded>
    <pubDate>Thu, 19 Nov 2026 00:22:44 +0000</pubDate>
    <dc:creator>MySQL Performance Blog</dc:creator>
    <category>MySQL</category>
    <category>SQL</category>
    <category>MySQL Optimizer</category>
  </item>

  <item>
    <title>Building PlanetScale with Insights Agent</title>
    <guid isPermaLink="false">https://planetscale.com/blog/building-planetscale-with-insights-agent</guid>
    <link>https://planetscale.com/blog/building-planetscale-with-insights-agent</link>
    <description>We've always used PlanetScale to build PlanetScale. So when we built Insights Agent, our own databases were the first place we ran it to see what kind of gains we could get. Insights Agent identifies problematic queries or schema using Query Insights, finds the code responsible for them, and opens a pull request with a fix. Our Rails API is a large, busy app, which makes it a good testing ground.We know a thing or two about databases over here, so our databases are already quite well optimized. Even so, Insights Agent has been able to find a bunch of wins for us, continuing our drive towards the most efficient database possible.Here are three (of many) examples of how we tested and used Insights Agent in the weeks leading up to launch, and how we used these experiences to make the product better.A feature flag check we didn't needOur Rails API checks feature flags with Flipper. On top of Flipper, we have an override that can turn a flag off for specific records. Checking the flag requires reading it from the database. Our code checked the override first, every time:Each lookup took about a millisecond, which doesn't raise any significant slow-query concerns at first look. However, when this happens ~28 million times per day, it added up to more than nine hours of database CPU time. Insights Agent found it by the aggregate time instead of individual execution time. The override can only turn a flag off, so Insights Agent moved it after the Flipper check.  def feature_enabled?(name)
-   if override_disables?(name)
-     false
-   else
-     Flipper.enabled?(name, actor)
-   end
+   Flipper.enabled?(name, actor) &amp;amp;&amp;amp; !override_disables?(name)
  end
If a flag is already off, there's nothing to override. The app behavior remains exactly the same as before, but now with far fewer lookups on these values:BeforeAfterCalls per day28.7 million8.3 millionDatabase time per day33,379 s9,625 sShare of all query time3.6%1.1%In this case, the traffic volume to the database did not change, but by careful inspection of both the database load and the codebase, the Insights Agent was able to significantly reduce load here.An index that made a query slowerNext, we asked Insights Agent to fix a slower query. This query looks up billing prices for an account, about 570,000 times a day (names simplified):select prices.* from prices
where prices.owner_id = ?
  and prices.metric in (&amp;lt;elements&amp;gt;)
  and prices.owner_type = ?
  and prices.region_id is not null
Insights showed this query reading about 122 rows for every returned row. This is often a tell that a query can be sped up. Insights Agent observed that the index MySQL used for this lookup had a range on region_id ahead of metric, and decided to add a new index with metric first. This might seem reasonable at first glance, but we merged it, and the query got slower:Before the indexAfter the indexAverage27.3 ms47.6 msp99129 ms222 msMySQL wasn't using the new index!Insights Agent was missing... somethingInsights Agent was working from a summary that left out a few important bits of context.First, almost none of these calls return a row, because most accounts have no prices to show. When the number of rows returned is zero (or near-zero), rows read per row returned looks terrible.Second, our Insights extension collapses IN lists when it normalizes SQL, which means &amp;lt;elements&amp;gt; was hiding 645 metric names, about 18 KB of SQL on every call. MySQL builds a lookup range for each of these values for every index that could use them. The new index gave it one more index to do that for.So we stopped sending the list of elements. The query now loads the few dozen regional prices an account has and filters them in Ruby:account.regional_prices.select do |price|
  price.region_id == region.id &amp;amp;&amp;amp; BILLABLE_METRICS.include?(price.metric)
end
Old queryNew queryDatabase time per day16,212 s1,129 sAverage28.5 ms1.95 msp99137 ms8.0 msThat's about 14 times less database time. The new query also uses the index Insights Agent added, so we kept it.Teaching Insights Agent what we learnedWe have updated Insights Agent to look at more evidence to be better at these types of investigations. For these, it must look at rows read and returned per call and the size of the SQL sent, including how many values are collapsed. And we're extending our eval harness to cover cases like this one, including the cases where the best option is to leave the query as-is. Sometimes no change is better than adding an index.More indexing workYet another indexing adventure here. A job in our Rails API runs every minute to find schema snapshots that need to be reconciled (query simplified):select snapshot.id from snapshot
join branch on branch.id = snapshot.branch_id
where snapshot.ready = ?
  and snapshot.failed_at is null
  and snapshot.created_at &amp;gt; ?
  and branch.ready = ?
order by snapshot.id asc
limit ?
The only index MySQL could use for this was one created on (ready, migration_applied_at). This index helps MySQL find unready snapshots. However, it does not help with the other filters because failed_at and created_at are not in it (as well as a few others which were removed). MySQL had to read every unready snapshot (ready = false) and check the additional columns on every row.There are a lot of unready snapshots in the database, but only a very small subset are both recent and not failed. MySQL had to find every match before it could sort by id and return the first few. Each call was reading 100s of thousands of rows to return, typically, less than 100 and was taking multiple seconds. The rows read / rows returned ratio on this query was really bad.Insights Agent suggested adding an index on (ready, failed_at, created_at):add_index :snapshot, [:ready, :failed_at, :created_at]
With ready and failed_at now included, MySQL can jump straight to the right range of created_at instead of reading every unready snapshot. Another query that counts unready snapshots for alerting had a similar issue, and was given a fix by the agent. In this case, the only change needed to the app-side code was a database migration.Both p50 and p95 dropped from seconds per call to millisecond-scale.WrapupWe spent weeks testing, improving, and optimizing the experience with Insights Agent in the weeks leading up to launch. While doing so, we used it to automate the discovery of a bunch of places where we could improve our code, our database schema, and our queries. If you want a faster database (and we know you do) go give Insights Agent a try on your database, today.If you prefer doing all these kinds of optimizations in your own agent harness, we have some docs to get you up and running on this as well.</description>
    <content:encoded><![CDATA[<p>We've always used PlanetScale to build PlanetScale. So when we built Insights Agent, our own databases were the first place we ran it to see what kind of gains we could get. Insights Agent identifies problematic queries or schema using Query Insights, finds the code responsible for them, and opens a pull request with a fix. Our Rails API is a large, busy app, which makes it a good testing ground.</p><p>We know a thing or two about databases over here, so our databases are already quite well optimized. Even so, Insights Agent has been able to find a bunch of wins for us, continuing our drive towards the most efficient database possible.</p><p>Here are three (of many) examples of how we tested and used Insights Agent in the weeks leading up to launch, and how we used these experiences to make the product better.</p><h2><a href="https://planetscale.com/blog/building-planetscale-with-insights-agent#a-feature-flag-check-we-didnt-need">A feature flag check we didn't need</a></h2><p>Our Rails API checks feature flags with Flipper. On top of Flipper, we have an override that can turn a flag off for specific records. Checking the flag requires reading it from the database. Our code checked the override first, every time:</p><p>Each lookup took about a millisecond, which doesn't raise any significant slow-query concerns at first look. However, when this happens ~28 million times per day, it added up to more than nine hours of database CPU time. Insights Agent found it by the aggregate time instead of individual execution time. The override can only turn a flag off, so Insights Agent moved it after the Flipper check.</p><div class="code-block" data-language="diff-ruby"><div class="max-w-full min-w-0"><pre class="shiki shiki-themes planetscale-light planetscale-dark" tabindex="0"><code><span class="line"><span>  </span><span>def</span><span> feature_enabled?</span><span>(</span><span>name</span><span>)</span></span>
<span class="line diff-remove"><span>- </span><span>  if</span><span> override_disables?</span><span>(</span><span>name</span><span>)</span></span>
<span class="line diff-remove"><span>- </span><span>    false</span></span>
<span class="line diff-remove"><span>- </span><span>  else</span></span>
<span class="line diff-remove"><span>- </span><span>    Flipper</span><span>.</span><span>enabled?</span><span>(</span><span>name</span><span>,</span><span> actor</span><span>)</span></span>
<span class="line diff-remove"><span>- </span><span>  end</span></span>
<span class="line diff-add"><span>+ </span><span>  Flipper</span><span>.</span><span>enabled?</span><span>(</span><span>name</span><span>,</span><span> actor</span><span>)</span><span> &amp;&amp;</span><span> !</span><span>override_disables?</span><span>(</span><span>name</span><span>)</span></span>
<span class="line"><span>  </span><span>end</span></span>
<span class="line"></span></code></pre></div></div><p>If a flag is already off, there's nothing to override. The app behavior remains exactly the same as before, but now with far fewer lookups on these values:</p><div class="table-overflow"><table><thead><tr><th></th><th>Before</th><th>After</th></tr></thead><tbody><tr><td>Calls per day</td><td>28.7 million</td><td>8.3 million</td></tr><tr><td>Database time per day</td><td>33,379 s</td><td>9,625 s</td></tr><tr><td>Share of all query time</td><td>3.6%</td><td>1.1%</td></tr></tbody></table></div><p>In this case, the traffic volume to the database did not change, but by careful inspection of both the database load and the codebase, the Insights Agent was able to significantly reduce load here.</p><h2><a href="https://planetscale.com/blog/building-planetscale-with-insights-agent#an-index-that-made-a-query-slower">An index that made a query slower</a></h2><p>Next, we asked Insights Agent to fix a slower query. This query looks up billing prices for an account, about 570,000 times a day (names simplified):</p><div class="code-block" data-language="sql"><div class="max-w-full min-w-0"><pre class="shiki shiki-themes planetscale-light planetscale-dark" tabindex="0"><code><span class="line"><span>select</span><span> prices.</span><span>*</span><span> from</span><span> prices</span></span>
<span class="line"><span>where</span><span> prices</span><span>.</span><span>owner_id</span><span> =</span><span> ?</span></span>
<span class="line"><span>  and</span><span> prices</span><span>.</span><span>metric</span><span> in</span><span> (</span><span>&lt;elements&gt;</span><span>)</span></span>
<span class="line"><span>  and</span><span> prices</span><span>.</span><span>owner_type</span><span> =</span><span> ?</span></span>
<span class="line"><span>  and</span><span> prices</span><span>.</span><span>region_id</span><span> is not null</span></span>
<span class="line"></span></code></pre></div></div><p>Insights showed this query reading about 122 rows for every returned row. This is often a tell that a query can be sped up. Insights Agent observed that the index MySQL used for this lookup had a range on <code>region_id</code> ahead of <code>metric</code>, and decided to add a new index with <code>metric</code> first. This might seem reasonable at first glance, but we merged it, and the query got slower:</p><div class="table-overflow"><table><thead><tr><th></th><th>Before the index</th><th>After the index</th></tr></thead><tbody><tr><td>Average</td><td>27.3 ms</td><td>47.6 ms</td></tr><tr><td>p99</td><td>129 ms</td><td>222 ms</td></tr></tbody></table></div><p>MySQL wasn't using the new index!</p><h2><a href="https://planetscale.com/blog/building-planetscale-with-insights-agent#insights-agent-was-missing-something">Insights Agent was missing... something</a></h2><p>Insights Agent was working from a summary that left out a few important bits of context.</p><p>First, almost none of these calls return a row, because most accounts have no prices to show. When the number of rows returned is zero (or near-zero), rows read per row returned looks terrible.</p><p>Second, our Insights extension collapses <code>IN</code> lists when it normalizes SQL, which means <code>&lt;elements&gt;</code> was hiding 645 metric names, about 18 KB of SQL on every call. MySQL builds a lookup range for each of these values for every index that could use them. The new index gave it one more index to do that for.</p><p>So we stopped sending the list of elements. The query now loads the few dozen regional prices an account has and filters them in Ruby:</p><div class="code-block" data-language="ruby"><div class="max-w-full min-w-0"><pre class="shiki shiki-themes planetscale-light planetscale-dark" tabindex="0"><code><span class="line"><span>account</span><span>.</span><span>regional_prices</span><span>.</span><span>select</span><span> do</span><span> |</span><span>price</span><span>|</span></span>
<span class="line"><span>  price</span><span>.</span><span>region_id</span><span> ==</span><span> region</span><span>.</span><span>id</span><span> &amp;&amp;</span><span> BILLABLE_METRICS</span><span>.</span><span>include?</span><span>(</span><span>price</span><span>.</span><span>metric</span><span>)</span></span>
<span class="line"><span>end</span></span>
<span class="line"></span></code></pre></div></div><div class="table-overflow"><table><thead><tr><th></th><th>Old query</th><th>New query</th></tr></thead><tbody><tr><td>Database time per day</td><td>16,212 s</td><td>1,129 s</td></tr><tr><td>Average</td><td>28.5 ms</td><td>1.95 ms</td></tr><tr><td>p99</td><td>137 ms</td><td>8.0 ms</td></tr></tbody></table></div><p>That's about 14 times less database time. The new query also uses the index Insights Agent added, so we kept it.</p><h3><a href="https://planetscale.com/blog/building-planetscale-with-insights-agent#teaching-insights-agent-what-we-learned">Teaching Insights Agent what we learned</a></h3><p>We have updated Insights Agent to look at more evidence to be better at these types of investigations. For these, it must look at rows read and returned per call <em>and</em> the size of the SQL sent, including how many values are collapsed. And we're extending our eval harness to cover cases like this one, including the cases where the best option is to leave the query as-is. Sometimes no change is better than adding an index.</p><h2><a href="https://planetscale.com/blog/building-planetscale-with-insights-agent#more-indexing-work">More indexing work</a></h2><p>Yet another indexing adventure here. A job in our Rails API runs every minute to find schema snapshots that need to be reconciled (query simplified):</p><div class="code-block" data-language="sql"><div class="max-w-full min-w-0"><pre class="shiki shiki-themes planetscale-light planetscale-dark" tabindex="0"><code><span class="line"><span>select</span><span> snapshot</span><span>.</span><span>id</span><span> from</span><span> snapshot</span></span>
<span class="line"><span>join</span><span> branch </span><span>on</span><span> branch</span><span>.</span><span>id</span><span> =</span><span> snapshot</span><span>.</span><span>branch_id</span></span>
<span class="line"><span>where</span><span> snapshot</span><span>.</span><span>ready</span><span> =</span><span> ?</span></span>
<span class="line"><span>  and</span><span> snapshot</span><span>.</span><span>failed_at</span><span> is</span><span> null</span></span>
<span class="line"><span>  and</span><span> snapshot</span><span>.</span><span>created_at</span><span> &gt;</span><span> ?</span></span>
<span class="line"><span>  and</span><span> branch</span><span>.</span><span>ready</span><span> =</span><span> ?</span></span>
<span class="line"><span>order by</span><span> snapshot</span><span>.</span><span>id</span><span> asc</span></span>
<span class="line"><span>limit</span><span> ?</span></span>
<span class="line"></span></code></pre></div></div><p>The only index MySQL could use for this was one created on <code>(ready, migration_applied_at)</code>. This index helps MySQL find unready snapshots. However, it does not help with the other filters because <code>failed_at</code> and <code>created_at</code> are not in it (as well as a few others which were removed). MySQL had to read every unready snapshot (<code>ready = false</code>) and check the additional columns on every row.</p><p>There are a lot of unready snapshots in the database, but only a very small subset are both <code>recent</code> and not failed. MySQL had to find every match before it could sort by <code>id</code> and return the first few. Each call was reading 100s of thousands of rows to return, typically, less than 100 and was taking multiple seconds. The rows read / rows returned ratio on this query was really bad.</p><p>Insights Agent suggested adding an index on <code>(ready, failed_at, created_at)</code>:</p><div class="code-block" data-language="ruby"><div class="max-w-full min-w-0"><pre class="shiki shiki-themes planetscale-light planetscale-dark" tabindex="0"><code><span class="line"><span>add_index </span><span>:</span><span>snapshot</span><span>,</span><span> [:</span><span>ready</span><span>,</span><span> :</span><span>failed_at</span><span>,</span><span> :</span><span>created_at</span><span>]</span></span>
<span class="line"></span></code></pre></div></div><p>With <code>ready</code> and <code>failed_at</code> now included, MySQL can jump straight to the right range of <code>created_at</code> instead of reading every unready snapshot. Another query that counts unready snapshots for alerting had a similar issue, and was given a fix by the agent. In this case, the only change needed to the app-side code was a database migration.</p><p><picture class="block"><source media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" srcset="https://planetscale-images.imgix.net/assets/graph-drop-lRYIW_8A.png?auto=compress%2Cformat"><source media="(prefers-color-scheme: dark)" srcset="https://planetscale-images.imgix.net/assets/graph-drop-darkmode-CHpajEiM.png?auto=compress%2Cformat"><img alt="p50 and p95 query latency drop from several seconds to about 0.1 seconds after the index ships" src="https://planetscale-images.imgix.net/assets/graph-drop-lRYIW_8A.png?auto=compress%2Cformat" width="2822" height="1217" loading="lazy"></picture></p><p>Both p50 and p95 dropped from seconds per call to millisecond-scale.</p><h2><a href="https://planetscale.com/blog/building-planetscale-with-insights-agent#wrapup">Wrapup</a></h2><p>We spent weeks testing, improving, and optimizing the experience with Insights Agent in the weeks leading up to launch. While doing so, we used it to automate the discovery of a bunch of places where we could improve our code, our database schema, and our queries. If you want a faster database (and we know you do) go give <a href="https://planetscale.com/docs/insights-agent">Insights Agent</a> a try on your database, today.</p><p>If you prefer doing all these kinds of optimizations in your own agent harness, we have some <a href="https://planetscale.com/docs/self-improving-database">docs to get you up and running</a> on this as well.</p>]]></content:encoded>
    <pubDate>Thu, 19 Nov 2026 00:00:00 +0000</pubDate>
    <dc:creator>PlanetScale</dc:creator>
    <category>engineering</category>
    <category>mysql</category>
  </item>

  <item>
    <title>Amazon RDS for MySQL LTS version 9.7 is now generally available</title>
    <guid isPermaLink="false">857e5579d00a3d6a46979e8b8344c816c31b9c81</guid>
    <link>https://aws.amazon.com/blogs/database/amazon-rds-for-mysql-lts-version-9-7-is-now-generally-available/</link>
    <description>Today, Amazon Relational Database Service (Amazon RDS) announced support for MySQL version 9.7, the latest Long-Term Support (LTS) major version from the MySQL community, in addition to MySQL 8.0 and 8.4. As with prior versions, you can bring your existing MySQL code, applications and tools to Amazon RDS. For customers who need more time to upgrade, Amazon RDS continues to offer earlier major versions under RDS Extended Support: MySQL 8.0 entered RDS Extended Support on August 1, 2026, and MySQL 5.7 remains available under RDS Extended Support through June 30, 2029. 
MySQL 9.7 is the second LTS Release from the community, following MySQL 8.4. It includes the features delivered across the 9.0 through 9.6 Innovation Releases and adds 9.7-specific capabilities. In this post, we explore notable features, list breaking changes and provide recommendations to ease the migration of your workloads to this version. 
Note for MySQL 5.7 customers: RDS Extended Support for MySQL 5.7 on Amazon Aurora MySQL-Compatible Edition and Amazon RDS for MySQL has been extended through June 30, 2029 (previously February 28, 2027), with no change to Extended Support pricing. During this period, AWS plans to provide critical security patches and bug fixes. If you’re running MySQL 5.7 workloads, you can use this additional time to plan and validate upgrades to MySQL 8.4 or to the latest 9.7 without disruption (follow the upgrade path). For details, see Amazon Aurora and RDS for MySQL expand Extended Support for MySQL 5.7 through June 2029. 
Before we turn to RDS for MySQL 9.7, we first revisit the MySQL versioning model. The community classifies every engine release as either an Innovation Release or a Long-Term Support (LTS) Release. 
 
 Innovation Releases: Released frequently for early and rapid experimentation, giving access to the latest features and changes. Each Innovation Release is supported only until the next Innovation Release, so support cycles are short. The MySQL community classified versions 9.0 through 9.6 as Innovation Releases. Because of their short lifecycle, RDS offers these versions in the Amazon RDS Database Preview Environment. 
 Long-Term Support (LTS) Releases: LTS Releases include stable, proven features, including functionality introduced in prior Innovation Releases. Features can be added or removed only in the first release of an LTS series (for example, 9.7.0) and there are no removals later in the series. Incremental releases within an LTS series (such as 9.7.1 and 9.7.2) contain only the fixes needed to reduce behavior-change risk and prioritize backward compatibility and security. MySQL 9.7 is an LTS series and follows MySQL 8.4 as the next LTS baseline. 
 
Note that MySQL 9.7 is the final release line to use the current sequential version numbers. Subsequent Innovation and LTS Releases move to calendar versioning in a YY.M format. For example, MySQL 26.7 is the July 2026 release. An LTS Release keeps its YY.M version fixed throughout that series’ support lifecycle. 
For more information, review MySQL Releases: Innovation and LTS. 
Upgrading your Amazon RDS for MySQL 8.4 database to MySQL 9.7 
Before planning your upgrade, review the MySQL community’s upgrade path guidelines and release model. MySQL supports upgrading to the next LTS series but not skipping one, so the path is LTS-to-LTS: MySQL 8.4 to MySQL 9.7. If you’re running MySQL 8.0, upgrade to 8.4 first, then to 9.7. 
You have multiple options when upgrading: use Amazon RDS Blue/Green deployments, read replicas, or an in-place upgrade. We recommend blue/green deployments to minimize risk and downtime. A blue/green deployment creates two environments, your current production environment (Blue) and a staging environment (Green), kept in sync with native logical replication. You can safely test your application against 9.7 on the Green environment and promote it only after you’re confident in compatibility, without impacting production. 
Before you upgrade, note the following requirements and limitations: 
 
 rdsproxyadmin becomes a reserved user. In MySQL 9.7, rdsproxyadmin is protected on all hosts, so you can no longer create, drop, rename, or change privileges for it. RDS Proxy creates and manages this user automatically when you register a proxy target, so if you use RDS Proxy, leave it in place. If you created the rdsproxyadmin user yourself, drop or rename it before you upgrade. 
 Blue/Green deployments are supported. RDS for MySQL supports Blue/Green deployments on all available 5.7, 8.0 and 8.4 versions. If you’re running MySQL 5.7, you can’t move directly to 9.7. Upgrade in steps, using a blue/green deployment for each hop: 5.7 to 8.0, then 8.0 to 8.4, then 8.4 to 9.7. 
 Replication from a MySQL 5.7 source to a 9.7 replica isn’t supported. Use a supported intermediate major version if you need to replicate between these versions. 
 Replicas running gtid_mode=OFF aren’t supported targets. RDS for MySQL 9.7 defaults to gtid_mode=ON_PERMISSIVE, which assigns global transaction identifiers (GTIDs) to new transactions. Raise any self-managed or external replica to at least OFF_PERMISSIVE before upgrading. 
 Multi-AZ DB clusters (Multi-AZ with two readable standbys) aren’t supported at launch. Amazon RDS for MySQL 9.7 doesn’t support Multi-AZ DB cluster deployments at GA and upgrades to 9.7 from an 8.4 Multi-AZ DB cluster with two readable standbys are blocked. Use a Single-AZ or Multi-AZ DB instance (single standby) deployment for 9.7. 
 
In the following sections, we discuss changes introduced with MySQL 9.7 that might be relevant for your applications. 
Changed parameter defaults 
Amazon RDS for MySQL offers the 9.7 LTS series at minor version 9.7.4. Because MySQL 9.7 is an LTS series, its incremental releases (9.7.1 through 9.7.4) contain only bug and security fixes. No features are added or removed and no parameter defaults change, after 9.7.0. Every default change in the following table was introduced earlier in the 9.x Innovation series (specifically MySQL 9.5.0) and is carried into the 9.7 LTS baseline that 9.7.4 ships. The 9.7.0 release itself introduced no default changes. We recommend thoroughly testing your applications with these defaults before upgrading. The following table shows the default changes, relative to MySQL 8.4, for parameters available on Amazon RDS for MySQL: 
 
  
   
   Parameter 
   MySQL 8.4 Default 
   MySQL 9.7 Default 
   
   
   explain_format 
   TRADITIONAL 
   TREE 
   
   
   explain_json_format_version 
   1 
   2 
   
   
   innodb_change_buffering 
   none 
   all 
   
   
   innodb_log_writer_threads 
   ON (dedicated log writer threads always enabled) 
   Binary-log aware: if log_bin=ON, ON when CPUs &amp;gt; 32 (otherwise OFF). If log_bin=OFF, ON when CPUs &amp;gt; 4 (otherwise OFF) 
   
   
   binlog_transaction_dependency_history_size 
   25,000 (max 1,000,000) 
   10,000,000 (max 100,000,000) 
   
  
 
The change to explain_format is especially visible: a bare EXPLAIN now returns TREE-format output instead of the traditional tabular format, which can affect tooling that parses EXPLAIN results. If you set any of these parameters explicitly in a parameter group today, review whether the new defaults better fit your workload before overriding them. 
In addition to the preceding community defaults, Amazon RDS for MySQL 9.7 applies the following default values changes (relative to MySQL 8.4): 
 
 require_secure_transport now defaults to ON, so client connections must use SSL/TLS. 
 gtid_mode now defaults to ON_PERMISSIVE. 
 group_replication_ssl_mode defaults to REQUIRED, enforcing SSL for Group Replication (active-active cluster) communication. 
 authentication_policy allows only caching_sha2_password (the removed mysql_native_password plugin is no longer accepted). 
 innodb_change_buffer_max_size now defaults to 5 (down from 25), as a percentage of the buffer pool. 
 back_log now defaults to 10000 (up from 151). 
 
Community MySQL 9.5.0 changed gtid_mode and enforce_gtid_consistency to ON. Amazon RDS for MySQL 9.7 keeps enforce_gtid_consistency=OFF and uses gtid_mode=ON_PERMISSIVE. See the following Amazon RDS for MySQL 9.7 enhancements section for details, including the note on active-active clusters (Group Replication is supported on all MySQL 9.7, 8.4 versions and on MySQL 8.0.35+). 
New capabilities in MySQL 9.7 
MySQL 9.7 adds several notable capabilities, highlighted in the following sections. 
Hypergraph Optimizer 
MySQL 9.7 makes the Hypergraph Optimizer available in the engine. It’s disabled by default and controlled through the optimizer_switch system variable, which you can set per session, globally, persisted, or per statement: 
 
 -- Session scope
SET optimizer_switch = 'hypergraph_optimizer=on';

-- Per-statement hint
SELECT /*+ SET_VAR(optimizer_switch = 'hypergraph_optimizer=on') */ ... ; 
 
The Hypergraph Optimizer can improve plan quality for complex queries with many joins. We recommend evaluating it against representative queries before enabling it broadly. On Amazon RDS for MySQL, the session and per-statement forms shown earlier work unchanged. To enable it for an entire instance, set optimizer_switch in a DB parameter group. 
JSON Duality Views 
MySQL 9.7 introduces JSON Duality Views, which you can use to access relational tables as JSON documents, with support for data manipulation language (DML), not just data definition language (DDL), and auto-increment. This provides a bridge for applications that work naturally with JSON while keeping data in normalized relational tables. 
Group Replication enhancements 
MySQL 9.7 adds several Group Replication components that improve operational visibility and resilience: 
 
 Replication Applier Metrics Component: extended statistics for lag, throughput, queued work and worker behavior. 
 Group Replication Flow Control Statistics Component: visibility into cluster throttling. 
 Group Replication Resource Manager Component: automatic eviction of unhealthy members and automatic rejoin. 
 Group Replication Primary Election Component: up-to-date-aware primary election that prefers the most current member. 
 
Security and authentication changes 
MySQL 9.7 changes how accounts authenticate and how connections are secured. Review the following changes before you upgrade. 
Removal of mysql_native_password 
The mysql_native_password authentication plugin has reached the end of its deprecation path. It was deprecated in MySQL 8.0.34, disabled by default in MySQL 8.4 and removed as of MySQL 9.0, so it doesn’t exist in MySQL 9.7. Accounts that still authenticate with mysql_native_password can’t connect after the upgrade. The server also rejects native-password authentication requests from older client programs that lack the CLIENT_PLUGIN_AUTH capability. 
Before upgrading, migrate any such accounts to caching_sha2_password (the default authentication plugin since MySQL 8.0) and update client drivers accordingly. To find affected accounts on your MySQL 8.4 instance, check for users whose plugin is still mysql_native_password: 
 
 SELECT user, host FROM mysql.user WHERE plugin = 'mysql_native_password'; 
 
On RDS for MySQL 9.7, the authentication_policy parameter can’t name mysql_native_password, because the plugin no longer exists. If you set that parameter in your current parameter group, make sure it specifies caching_sha2_password instead. 
PBKDF2 storage format for caching_sha2_password 
MySQL 9.7 adds a PBKDF2 storage format for caching_sha2_password credentials. On Amazon RDS for MySQL you select the format with caching_sha2_password_storage_format, which accepts CRYPT5 or PBKDF2_SHA512. The default remains CRYPT5. 
RDS for MySQL 9.7 also makes the related caching_sha2_password_proxy_users parameter modifiable. It adds proxy-user support for the plugin and takes effect only when check_proxy_users is enabled. 
Removed and deprecated variables 
Several variables and plugins present in MySQL 8.4 are removed in the 9.x series. Review and update any automation, monitoring, or parameter groups that reference them before upgrading, because setting a removed variable can cause startup or upgrade issues. 
 
  
   
   Variable / plugin 
   Replacement or behavior 
   
   
   mysql_native_password (plugin) 
   Removed. Migrate accounts to caching_sha2_password 
   
   
   mysql_native_password_proxy_users 
   Removed with the mysql_native_password plugin. Was customer-visible and modifiable in 8.4 
   
   
   replica_parallel_type 
   Parallel replication now uses LOGICAL_CLOCK exclusively 
   
   
   group_replication_allow_local_lower_version_join 
   Version compatibility is enforced automatically. No longer configurable 
   
   
   temptable_use_mmap 
   Feature removed 
   
   
   innodb_undo_tablespaces 
   Removed. The setting had been ignored since MySQL 8.0.14 (use CREATE/DROP UNDO TABLESPACE instead) 
   
  
 
Replication variable renames (master/slave to source/replica) 
MySQL has completed its shift from master/slave terminology to source/replica. As part of that change, MySQL 9.5.0 renamed the semisynchronous replication plugins: semisync_master became semisync_source and semisync_slave became semisync_replica. 
On Amazon RDS for MySQL, the following parameters are visible but RDS-managed, so you can’t set them yourself. Their names still change with the upgrade, which matters if your monitoring, dashboards, or automation read them through SHOW VARIABLES or SHOW STATUS: 
 
  
   
   MySQL 8.4 
   MySQL 9.7 
   
   
   rpl_semi_sync_master_enabled 
   rpl_semi_sync_source_enabled 
   
   
   rpl_semi_sync_slave_enabled 
   rpl_semi_sync_replica_enabled 
   
   
   rpl_semi_sync_master_timeout 
   rpl_semi_sync_source_timeout 
   
   
   rpl_semi_sync_master_wait_point 
   rpl_semi_sync_source_wait_point 
   
   
   rpl_semi_sync_master_wait_for_slave_count 
   rpl_semi_sync_source_wait_for_replica_count 
   
   
   rpl_semi_sync_master_wait_no_slave 
   rpl_semi_sync_source_wait_no_replica 
   
   
   rpl_semi_sync_master_target_apply_lag 
   rpl_semi_sync_source_target_apply_lag 
   
  
 
Update any monitoring or automation that reads these names before upgrading. 
Removed functions 
MySQL 9.x moves some built-in SQL functions out of the server core. Review your code for the functions described in the following section before you upgrade. 
MD5() and SHA1() SQL functions aren’t available 
In MySQL 9.x, the MD5() and SHA1() SQL functions were moved out of the server core into a separate classic_hashing component. Amazon RDS for MySQL 9.7 doesn’t offer the classic_hashing component, so the MD5() and SHA1() SQL functions aren’t available. 
Before upgrading, review your application and stored programs for use of MD5() and SHA1(). For hashing needs, use SHA2() (for example, SHA2(col, 256)), which remains available. There is no supported MD5() replacement, so plan application changes where it’s used. 
Transitional foreign key handling variables 
Amazon RDS for MySQL 9.7 sets innodb_native_foreign_keys=ON by default, so foreign key checks and cascade operations are handled by the InnoDB engine, the same behavior as MySQL 8.4. Cascaded changes are therefore not individually written to the binary log and change-data-capture (CDC) tools such as AWS Database Migration Service (AWS DMS) and Debezium see the same events they did on 8.4. 
In community MySQL, innodb_native_foreign_keys defaults to OFF and the SQL layer performs foreign key cascades, which does write each cascaded change to the binary log, increasing binlog volume and surfacing those changes to CDC. The parameter is customer-modifiable, so if you set innodb_native_foreign_keys=OFF on RDS, plan for those additional binlog entries. 
innodb_native_foreign_keys and enable_cascade_triggers are transitional variables introduced in the 9.x series (innodb_native_foreign_keys in 9.6, enable_cascade_triggers in 9.7) and already marked for eventual removal, so avoid building long-term dependencies on them. 
Amazon RDS for MySQL 9.7 enhancements 
Beyond the community engine changes described earlier, Amazon RDS for MySQL 9.7 makes several RDS-specific parameter-group and default choices. The changes most likely to affect an existing 8.4 workload are highlighted here. 
Connections require TLS by default 
require_secure_transport now defaults to ON in RDS for MySQL 9.7, changed from OFF in 8.4. Clients that connect without SSL/TLS are rejected. Before upgrading, confirm that your applications and drivers connect over TLS, or set require_secure_transport=OFF in your parameter group if you must continue to allow non-TLS connections. 
GTID defaults on RDS 
Community MySQL changed the GTID defaults to ON in 9.5.0. Amazon RDS for MySQL 9.7 deliberately doesn’t adopt those values as-is: 
 
 enforce_gtid_consistency stays OFF, unchanged from 8.4. Enable it explicitly if your workload needs GTID consistency enforcement. 
 gtid_mode moves from OFF_PERMISSIVE (8.4) to ON_PERMISSIVE, promoting GTIDs rather than enforcing them. 
 
This keeps GTID behavior close to 8.4 while moving in the community’s direction. 
Because ON_PERMISSIVE assigns GTIDs to new transactions, two things change for replication. First, a replica running gtid_mode=OFF isn’t a supported target for a 9.7 source, although it was supported for an 8.4 source at OFF_PERMISSIVE. If you replicate from Amazon RDS for MySQL to a self-managed or external replica, set that replica to at least OFF_PERMISSIVE before you upgrade. Replication into 9.7 from a source without GTIDs is unaffected. Second, GTIDs now appear in the binary log, so verify any tool that reads binlogs directly. GTID auto-positioning still requires gtid_mode=ON. Moving to ON requires setting enforce_gtid_consistency=ON first, which rejects GTID-unsafe statements such as a transaction that updates both InnoDB and MyISAM tables. 
Group Replication (active-active clusters) 
For active-active clusters that use the MySQL Group Replication plugin, group_replication_ssl_mode now defaults to REQUIRED, enforcing SSL for Group Replication communication. 
Authentication changes 
 
 authentication_policy can’t name mysql_native_password, because the plugin is removed from the engine. Set it to caching_sha2_password instead (see Removal of mysql_native_password). 
 ssl_cipher is now dynamic: it can be changed without a restart (it was static in 8.4). 
 
Password validation moves to the component 
RDS for MySQL 9.7 uses component-based password validation exclusively (the plugin form is gone): 
 
 The validate-password parameter is replaced by rds.validate_password, which controls installation of the validate_password component. 
 The underscore-style variables are renamed to dot-notation, for example validate_password_length becomes validate_password.length and validate_password_policy becomes validate_password.policy. Two new checks are available: validate_password.changed_characters_percentage and validate_password.check_user_name. 
 
Because the parameters are renamed, your existing validate_password_* values don’t carry across the upgrade. A value set in your 8.4 parameter group isn’t applied in 9.7, and password validation falls back to the component defaults, which might be weaker than your current policy. Record your current validate_password_* values before you upgrade, then set the matching validate_password.* parameters in your 9.7 parameter group. Also update any automation that references the old validate_password_* names or the validate-password plugin. 
New customer-visible parameters 
RDS for MySQL 9.7 exposes several new modifiable parameters: 
 
  
   
   Parameter 
   Purpose 
   
   
   activate_mandatory_roles 
   Activate mandatory roles in addition to an account’s default roles (new in 9.7. Defaults to ON) 
   
   
   innodb_native_foreign_keys 
   InnoDB-level foreign key checks and cascades. Defaults to ON on RDS (same as 8.4). See Transitional foreign key handling variables 
   
   
   caching_sha2_password_storage_format 
   Storage format for caching_sha2_password credentials (CRYPT5 or PBKDF2_SHA512. Default CRYPT5) 
   
   
   caching_sha2_password_proxy_users 
   Whether caching_sha2_password supports proxy users (effective only with check_proxy_users) 
   
   
   enable_cascade_triggers 
   Execute triggers on child tables during foreign key cascade operations (transitional) 
   
   
   table_open_cache_triggers 
   Maximum number of cached open tables with loaded triggers 
   
   
   connection_memory_status_limit 
   Per-connection memory threshold before the status counter increments 
   
   
   global_connection_memory_status_limit 
   Total connection-memory threshold before the status counter increments 
   
  
 
Other parameter changes 
 
 Removed variables: see the preceding Removed and deprecated variables section (replica_parallel_type, innodb_undo_tablespaces and temptable_use_mmap). 
 replica_parallel_workers minimum allowed value changes from 0 to 1: single-threaded applier mode is gone. 
 binlog_transaction_dependency_history_size allowed range on RDS expands to 1 to 100,000,000. 
 Character sets: the utf8 alias appears as utf8mb3 in the allowed values for the character_set_* parameters and the utf8 collations are renamed to utf8mb3_*. 
 
Migration checklist 
The following checklist recaps the changes covered in this post so that you can work through them as a single list when upgrading from MySQL 8.4 to 9.7. The list moves from tasks to complete before you start the upgrade, through changed defaults to review, to new capabilities worth evaluating after you’re running 9.7. 
 
 If RDS Proxy isn’t attached, drop the rdsproxyadmin user if it exists. In MySQL 9.7, rdsproxyadmin becomes a reserved user on all hosts and must be absent before you upgrade. If RDS Proxy is attached, it creates and manages this user, so leave it in place. 
 Confirm your deployment type is supported. Multi-AZ DB clusters (two readable standbys) aren’t supported at launch and upgrades from an 8.4 Multi-AZ DB cluster are blocked. 
 Migrate any accounts using mysql_native_password to caching_sha2_password (the plugin is removed as of 9.0) and update client drivers. 
 Review your application and stored programs for MD5() and SHA1(). These SQL functions aren’t available in RDS for MySQL 9.7. For hashing, use SHA2() (for example, SHA2(col, 256)), which remains available. There is no supported MD5() replacement, so plan application changes where it’s used. 
 Move password validation from the validate-password plugin to the validate_password component. RDS for MySQL 9.7 provides the component form only: the plugin is replaced by rds.validate_password and the validate_password_* variables are renamed to dot notation, for example validate_password_length becomes validate_password.length. Update automation that references the plugin or the old variable names. 
 Confirm that applications and drivers connect over SSL/TLS. require_secure_transport now defaults to ON, so non-TLS connections are rejected. Set require_secure_transport=OFF in your parameter group if you must allow them. 
 Update monitoring and automation that read the semisynchronous replication variables and status. They were renamed from master and slave to source and replica, for example rpl_semi_sync_master_timeout becomes rpl_semi_sync_source_timeout. These parameters are visible but RDS-managed, so you can’t set them yourself. 
 Remove references to replica_parallel_type and group_replication_allow_local_lower_version_join, which are removed in the 9.x series. 
 Check for schema objects named with the new reserved keywords EXTERNAL and LIBRARY (reserved as of MySQL 9.7). A reserved word must be quoted with backticks to be used as an identifier, so quote or rename any database, table, or column that uses them. When you create a 9.7 instance, don’t pass EXTERNAL or LIBRARY as the initial database name (--db-name) in the create-db-instance call, because a reserved word isn’t accepted there. 
 Review explain_format. The default changed from TRADITIONAL to TREE, so update any tooling that parses EXPLAIN output. Note that explain_json_format_version now defaults to 2. 
 Review innodb_change_buffering. The default changed from none to all. 
 Review innodb_log_writer_threads. The 8.4 default was always ON. In 9.7 it’s binary-log and CPU aware, so it can default to OFF on smaller instances. 
 Review binlog_transaction_dependency_history_size. The default increased from 25,000 to 10,000,000 (allowed range up to 100,000,000). 
 Confirm the RDS for MySQL 9.7 defaults for gtid_mode and enforce_gtid_consistency. Community MySQL changed both to ON in 9.5.0 and RDS manages them through the parameter group. 
 Consider JSON Duality Views for JSON-relational workloads. 
 Evaluate the Hypergraph Optimizer for complex, multi-join queries. It’s opt-in through optimizer_switch and disabled by default. 
 Test thoroughly in a Blue/Green deployment, on the Green environment, before promoting. 
 
Conclusion 
In this post, we reviewed the MySQL community versioning model, the new capabilities in MySQL 9.7, key security and replication changes, removed and transitional variables and a recommended upgrade approach using Amazon RDS Blue/Green deployments. We encourage you to test MySQL 9.7’s new features with your workloads. To get started, create or upgrade a database instance on the Amazon RDS console. For more information, see the Amazon RDS for MySQL User Guide. 
Sources 
 
 MySQL 9.7 Release Notes. 
 MySQL 9.5.0 Release Notes. 
 MySQL 9.7 What Is New. 
 MySQL 9.7 Variables Added, Deprecated and Removed. 
 Amazon Aurora and RDS for MySQL expand Extended Support for MySQL 5.7 through June 2029. 
 
  
 
About the authors 
 
  
  
   
   
  Sabalesh Mahajan 
  Sabalesh is a Senior Database Engineer at Amazon Web Services, with 19 years of experience in databases. He collaborates closely with the Amazon RDS product teams, contributing his expertise to the development and enhancement of innovative products. 
  
  
  
   
   
  Akshat Nehra 
  Akshat is a Software Development Engineer on the Amazon RDS for MySQL and MariaDB Engine team at Amazon Web Services, where he mostly works on the data plane. He focuses on delivering new community engine versions, contributing fixes upstream to the MySQL and MariaDB communities and security enhancements, helping make engine upgrades safe, reliable and seamless for customers. 
  
  
  
   
   
  Chelluru V N S S Vidyadhar 
  Chelluru is a Senior Database Engineer at Amazon Web Services with 14 years of experience in open source databases. 
  
</description>
    <content:encoded><![CDATA[<p>Today, <a href="https://aws.amazon.com/rds/" target="_blank" rel="noopener">Amazon Relational Database Service</a> (Amazon RDS) announced support for MySQL version 9.7, the latest Long-Term Support (LTS) major version from the MySQL community, in addition to MySQL 8.0 and 8.4. As with prior versions, you can bring your existing MySQL code, applications and tools to Amazon RDS. For customers who need more time to upgrade, Amazon RDS continues to offer earlier major versions under <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/extended-support.html" target="_blank" rel="noopener">RDS Extended Support</a>: MySQL 8.0 entered RDS Extended Support on August 1, 2026, and MySQL 5.7 remains available under RDS Extended Support through June 30, 2029.</p> 
<p>MySQL 9.7 is the second LTS Release from the community, following MySQL 8.4. It includes the features delivered across the 9.0 through 9.6 Innovation Releases and adds 9.7-specific capabilities. In this post, we explore notable features, list breaking changes and provide recommendations to ease the migration of your workloads to this version.</p> 
<p><strong>Note for MySQL 5.7 customers:</strong> RDS Extended Support for MySQL 5.7 on Amazon Aurora MySQL-Compatible Edition and Amazon RDS for MySQL has been extended through June 30, 2029 (previously February 28, 2027), with no change to Extended Support pricing. During this period, AWS plans to provide critical security patches and bug fixes. If you’re running MySQL 5.7 workloads, you can use this additional time to plan and validate upgrades to MySQL 8.4 or to the latest 9.7 without disruption (follow the upgrade path). For details, see <a href="https://aws.amazon.com/about-aws/whats-new/2026/06/rds-mysql-es-extension/" target="_blank" rel="noopener">Amazon Aurora and RDS for MySQL expand Extended Support for MySQL 5.7 through June 2029</a>.</p> 
<p>Before we turn to RDS for MySQL 9.7, we first revisit the MySQL versioning model. The community classifies every engine release as either an Innovation Release or a Long-Term Support (LTS) Release.</p> 
<ul> 
 <li><strong>Innovation Releases</strong>: Released frequently for early and rapid experimentation, giving access to the latest features and changes. Each Innovation Release is supported only until the next Innovation Release, so support cycles are short. The MySQL community classified versions 9.0 through 9.6 as Innovation Releases. Because of their short lifecycle, RDS offers these versions in the Amazon RDS Database Preview Environment.</li> 
 <li><strong>Long-Term Support (LTS) Releases</strong>: LTS Releases include stable, proven features, including functionality introduced in prior Innovation Releases. Features can be added or removed only in the first release of an LTS series (for example, 9.7.0) and there are no removals later in the series. Incremental releases within an LTS series (such as 9.7.1 and 9.7.2) contain only the fixes needed to reduce behavior-change risk and prioritize backward compatibility and security. MySQL 9.7 is an LTS series and follows MySQL 8.4 as the next LTS baseline.</li> 
</ul> 
<p>Note that MySQL 9.7 is the final release line to use the current sequential version numbers. Subsequent Innovation and LTS Releases move to calendar versioning in a <code>YY.M</code> format. For example, MySQL 26.7 is the July 2026 release. An LTS Release keeps its <code>YY.M</code> version fixed throughout that series’ support lifecycle.</p> 
<p>For more information, review <a href="https://dev.mysql.com/doc/refman/9.7/en/mysql-releases.html" target="_blank" rel="noopener">MySQL Releases: Innovation and LTS</a>.</p> 
<h2>Upgrading your Amazon RDS for MySQL 8.4 database to MySQL 9.7</h2> 
<p>Before planning your upgrade, review the MySQL community’s <a href="https://dev.mysql.com/doc/refman/9.7/en/upgrade-paths.html" target="_blank" rel="noopener">upgrade path guidelines</a> and <a href="https://dev.mysql.com/doc/refman/9.7/en/mysql-releases.html" target="_blank" rel="noopener">release model</a>. MySQL supports upgrading to the next LTS series but not skipping one, so the path is <strong>LTS-to-LTS</strong>: MySQL 8.4 to MySQL 9.7. If you’re running MySQL 8.0, upgrade to 8.4 first, then to 9.7.</p> 
<p>You have multiple options when upgrading: use <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html" target="_blank" rel="noopener">Amazon RDS Blue/Green deployments</a>, <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.MySQL.ReducedDowntime.html" target="_blank" rel="noopener">read replicas</a>, or an in-place upgrade. We recommend blue/green deployments to minimize risk and downtime. A blue/green deployment creates two environments, your current production environment (Blue) and a staging environment (Green), kept in sync with native logical replication. You can safely test your application against 9.7 on the Green environment and promote it only after you’re confident in compatibility, without impacting production.</p> 
<p>Before you upgrade, note the following requirements and limitations:</p> 
<ul> 
 <li><code>rdsproxyadmin</code> <strong>becomes a reserved user.</strong> In MySQL 9.7, <code>rdsproxyadmin</code> is protected on all hosts, so you can no longer create, drop, rename, or change privileges for it. RDS Proxy creates and manages this user automatically when you register a proxy target, so if you use RDS Proxy, leave it in place. If you created the <code>rdsproxyadmin</code> user yourself, drop or rename it before you upgrade.</li> 
 <li><strong>Blue/Green deployments are supported.</strong> RDS for MySQL supports <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RDS_Fea_Regions_DB-eng.Feature.BlueGreenDeployments.html" target="_blank" rel="noopener">Blue/Green deployments</a> on all available 5.7, 8.0 and 8.4 versions. If you’re running MySQL 5.7, you can’t move directly to 9.7. Upgrade in steps, using a blue/green deployment for each hop: 5.7 to 8.0, then 8.0 to 8.4, then 8.4 to 9.7.</li> 
 <li><strong>Replication from a MySQL 5.7 source to a 9.7 replica isn’t supported.</strong> Use a supported intermediate major version if you need to replicate between these versions.</li> 
 <li><strong>Replicas running gtid_mode=OFF aren’t supported targets.</strong> RDS for MySQL 9.7 defaults to <code>gtid_mode=ON_PERMISSIVE</code>, which assigns global transaction identifiers (GTIDs) to new transactions. Raise any self-managed or external replica to at least <code>OFF_PERMISSIVE</code> before upgrading.</li> 
 <li><strong>Multi-AZ DB clusters (Multi-AZ with two readable standbys) aren’t supported at launch.</strong> Amazon RDS for MySQL 9.7 doesn’t support Multi-AZ DB cluster deployments at GA and upgrades to 9.7 from an 8.4 Multi-AZ DB cluster with two readable standbys are blocked. Use a Single-AZ or Multi-AZ DB instance (single standby) deployment for 9.7.</li> 
</ul> 
<p>In the following sections, we discuss changes introduced with MySQL 9.7 that might be relevant for your applications.</p> 
<h2>Changed parameter defaults</h2> 
<p>Amazon RDS for MySQL offers the 9.7 LTS series at minor version 9.7.4. Because MySQL 9.7 is an LTS series, its incremental releases (9.7.1 through 9.7.4) contain only bug and security fixes. No features are added or removed and no parameter defaults change, after 9.7.0. Every default change in the following table was introduced earlier in the 9.x Innovation series (specifically MySQL 9.5.0) and is carried into the 9.7 LTS baseline that 9.7.4 ships. The 9.7.0 release itself introduced no default changes. We recommend thoroughly testing your applications with these defaults before upgrading. The following table shows the default changes, relative to MySQL 8.4, for parameters available on Amazon RDS for MySQL:</p> 
<table border="1px" width="100%" cellpadding="10px"> 
 <tbody> 
  <tr> 
   <td><strong>Parameter</strong></td> 
   <td><strong>MySQL 8.4 Default</strong></td> 
   <td><strong>MySQL 9.7 Default</strong></td> 
  </tr> 
  <tr> 
   <td><code>explain_format</code></td> 
   <td>TRADITIONAL</td> 
   <td>TREE</td> 
  </tr> 
  <tr> 
   <td><code>explain_json_format_version</code></td> 
   <td>1</td> 
   <td>2</td> 
  </tr> 
  <tr> 
   <td><code>innodb_change_buffering</code></td> 
   <td>none</td> 
   <td>all</td> 
  </tr> 
  <tr> 
   <td><code>innodb_log_writer_threads</code></td> 
   <td>ON (dedicated log writer threads always enabled)</td> 
   <td>Binary-log aware: if <code>log_bin=ON</code>, ON when CPUs &gt; 32 (otherwise OFF). If <code>log_bin=OFF</code>, ON when CPUs &gt; 4 (otherwise OFF)</td> 
  </tr> 
  <tr> 
   <td><code>binlog_transaction_dependency_history_size</code></td> 
   <td>25,000 (max 1,000,000)</td> 
   <td>10,000,000 (max 100,000,000)</td> 
  </tr> 
 </tbody> 
</table> 
<p>The change to <code>explain_format</code> is especially visible: a bare <code>EXPLAIN</code> now returns TREE-format output instead of the traditional tabular format, which can affect tooling that parses <code>EXPLAIN</code> results. If you set any of these parameters explicitly in a parameter group today, review whether the new defaults better fit your workload before overriding them.</p> 
<p>In addition to the preceding community defaults, Amazon RDS for MySQL 9.7 applies the following default values changes (relative to MySQL 8.4):</p> 
<ul> 
 <li><code>require_secure_transport</code> now defaults to ON, so client connections must use SSL/TLS.</li> 
 <li><code>gtid_mode</code> now defaults to <code>ON_PERMISSIVE</code>.</li> 
 <li><code>group_replication_ssl_mode</code> defaults to REQUIRED, enforcing SSL for Group Replication (active-active cluster) communication.</li> 
 <li><code>authentication_policy</code> allows only <code>caching_sha2_password</code> (the removed <code>mysql_native_password</code> plugin is no longer accepted).</li> 
 <li><code>innodb_change_buffer_max_size</code> now defaults to 5 (down from 25), as a percentage of the buffer pool.</li> 
 <li><code>back_log</code> now defaults to 10000 (up from 151).</li> 
</ul> 
<p>Community MySQL 9.5.0 changed <code>gtid_mode</code> and <code>enforce_gtid_consistency</code> to ON. Amazon RDS for MySQL 9.7 keeps <code>enforce_gtid_consistency=OFF</code> and uses <code>gtid_mode=ON_PERMISSIVE</code>. See the following <em>Amazon RDS for MySQL 9.7 enhancements</em> section for details, including the note on <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-active-active-clusters.html" target="_blank" rel="noopener">active-active clusters</a> (Group Replication is supported on all MySQL 9.7, 8.4 versions and on MySQL 8.0.35+).</p> 
<h2>New capabilities in MySQL 9.7</h2> 
<p>MySQL 9.7 adds several notable capabilities, highlighted in the following sections.</p> 
<h3>Hypergraph Optimizer</h3> 
<p>MySQL 9.7 makes the Hypergraph Optimizer available in the engine. It’s disabled by default and controlled through the <code>optimizer_switch</code> system variable, which you can set per session, globally, persisted, or per statement:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">-- Session scope
SET optimizer_switch = 'hypergraph_optimizer=on';

-- Per-statement hint
SELECT /*+ SET_VAR(optimizer_switch = 'hypergraph_optimizer=on') */ ... ;</code></pre> 
</div> 
<p>The Hypergraph Optimizer can improve plan quality for complex queries with many joins. We recommend evaluating it against representative queries before enabling it broadly. On Amazon RDS for MySQL, the session and per-statement forms shown earlier work unchanged. To enable it for an entire instance, set <code>optimizer_switch</code> in a DB parameter group.</p> 
<h3>JSON Duality Views</h3> 
<p>MySQL 9.7 introduces JSON Duality Views, which you can use to access relational tables as JSON documents, with support for data manipulation language (DML), not just data definition language (DDL), and auto-increment. This provides a bridge for applications that work naturally with JSON while keeping data in normalized relational tables.</p> 
<h3>Group Replication enhancements</h3> 
<p>MySQL 9.7 adds several Group Replication components that improve operational visibility and resilience:</p> 
<ul> 
 <li><strong>Replication Applier Metrics Component</strong>: extended statistics for lag, throughput, queued work and worker behavior.</li> 
 <li><strong>Group Replication Flow Control Statistics Component</strong>: visibility into cluster throttling.</li> 
 <li><strong>Group Replication Resource Manager Component</strong>: automatic eviction of unhealthy members and automatic rejoin.</li> 
 <li><strong>Group Replication Primary Election Component</strong>: up-to-date-aware primary election that prefers the most current member.</li> 
</ul> 
<h2>Security and authentication changes</h2> 
<p>MySQL 9.7 changes how accounts authenticate and how connections are secured. Review the following changes before you upgrade.</p> 
<h3>Removal of mysql_native_password</h3> 
<p>The <code>mysql_native_password</code> authentication plugin has reached the end of its deprecation path. It was deprecated in MySQL 8.0.34, disabled by default in MySQL 8.4 and <strong>removed as of MySQL 9.0</strong>, so it doesn’t exist in MySQL 9.7. Accounts that still authenticate with <code>mysql_native_password</code> can’t connect after the upgrade. The server also rejects native-password authentication requests from older client programs that lack the <code>CLIENT_PLUGIN_AUTH</code> capability.</p> 
<p>Before upgrading, migrate any such accounts to <code>caching_sha2_password</code> (the default authentication plugin since MySQL 8.0) and update client drivers accordingly. To find affected accounts on your MySQL 8.4 instance, check for users whose plugin is still <code>mysql_native_password</code>:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SELECT user, host FROM mysql.user WHERE plugin = 'mysql_native_password';</code></pre> 
</div> 
<p>On RDS for MySQL 9.7, the <code>authentication_policy</code> parameter can’t name <code>mysql_native_password</code>, because the plugin no longer exists. If you set that parameter in your current parameter group, make sure it specifies <code>caching_sha2_password</code> instead.</p> 
<h3>PBKDF2 storage format for caching_sha2_password</h3> 
<p>MySQL 9.7 adds a PBKDF2 storage format for <code>caching_sha2_password</code> credentials. On Amazon RDS for MySQL you select the format with <code>caching_sha2_password_storage_format</code>, which accepts <code>CRYPT5</code> or <code>PBKDF2_SHA512</code>. The default remains <code>CRYPT5</code>.</p> 
<p>RDS for MySQL 9.7 also makes the related <code>caching_sha2_password_proxy_users</code> parameter modifiable. It adds proxy-user support for the plugin and takes effect only when <code>check_proxy_users</code> is enabled.</p> 
<h2>Removed and deprecated variables</h2> 
<p>Several variables and plugins present in MySQL 8.4 are removed in the 9.x series. Review and update any automation, monitoring, or parameter groups that reference them before upgrading, because setting a removed variable can cause startup or upgrade issues.</p> 
<table border="1px" width="100%" cellpadding="10px"> 
 <tbody> 
  <tr> 
   <td><strong>Variable / plugin</strong></td> 
   <td><strong>Replacement or behavior</strong></td> 
  </tr> 
  <tr> 
   <td><code>mysql_native_password</code> (plugin)</td> 
   <td>Removed. Migrate accounts to <code>caching_sha2_password</code></td> 
  </tr> 
  <tr> 
   <td><code>mysql_native_password_proxy_users</code></td> 
   <td>Removed with the <code>mysql_native_password</code> plugin. Was customer-visible and modifiable in 8.4</td> 
  </tr> 
  <tr> 
   <td><code>replica_parallel_type</code></td> 
   <td>Parallel replication now uses <code>LOGICAL_CLOCK</code> exclusively</td> 
  </tr> 
  <tr> 
   <td><code>group_replication_allow_local_lower_version_join</code></td> 
   <td>Version compatibility is enforced automatically. No longer configurable</td> 
  </tr> 
  <tr> 
   <td><code>temptable_use_mmap</code></td> 
   <td>Feature removed</td> 
  </tr> 
  <tr> 
   <td><code>innodb_undo_tablespaces</code></td> 
   <td>Removed. The setting had been ignored since MySQL 8.0.14 (use <code>CREATE/DROP UNDO TABLESPACE</code> instead)</td> 
  </tr> 
 </tbody> 
</table> 
<h2>Replication variable renames (master/slave to source/replica)</h2> 
<p>MySQL has completed its shift from master/slave terminology to source/replica. As part of that change, MySQL 9.5.0 renamed the semisynchronous replication plugins: <code>semisync_master</code> became <code>semisync_source</code> and <code>semisync_slave</code> became <code>semisync_replica</code>.</p> 
<p>On Amazon RDS for MySQL, the following parameters are visible but RDS-managed, so you can’t set them yourself. Their names still change with the upgrade, which matters if your monitoring, dashboards, or automation read them through <code>SHOW VARIABLES</code> or <code>SHOW STATUS</code>:</p> 
<table border="1px" width="100%" cellpadding="10px"> 
 <tbody> 
  <tr> 
   <td><strong>MySQL 8.4</strong></td> 
   <td><strong>MySQL 9.7</strong></td> 
  </tr> 
  <tr> 
   <td><code>rpl_semi_sync_master_enabled</code></td> 
   <td><code>rpl_semi_sync_source_enabled</code></td> 
  </tr> 
  <tr> 
   <td><code>rpl_semi_sync_slave_enabled</code></td> 
   <td><code>rpl_semi_sync_replica_enabled</code></td> 
  </tr> 
  <tr> 
   <td><code>rpl_semi_sync_master_timeout</code></td> 
   <td><code>rpl_semi_sync_source_timeout</code></td> 
  </tr> 
  <tr> 
   <td><code>rpl_semi_sync_master_wait_point</code></td> 
   <td><code>rpl_semi_sync_source_wait_point</code></td> 
  </tr> 
  <tr> 
   <td><code>rpl_semi_sync_master_wait_for_slave_count</code></td> 
   <td><code>rpl_semi_sync_source_wait_for_replica_count</code></td> 
  </tr> 
  <tr> 
   <td><code>rpl_semi_sync_master_wait_no_slave</code></td> 
   <td><code>rpl_semi_sync_source_wait_no_replica</code></td> 
  </tr> 
  <tr> 
   <td><code>rpl_semi_sync_master_target_apply_lag</code></td> 
   <td><code>rpl_semi_sync_source_target_apply_lag</code></td> 
  </tr> 
 </tbody> 
</table> 
<p>Update any monitoring or automation that reads these names before upgrading.</p> 
<h2>Removed functions</h2> 
<p>MySQL 9.x moves some built-in SQL functions out of the server core. Review your code for the functions described in the following section before you upgrade.</p> 
<h3>MD5() and SHA1() SQL functions aren’t available</h3> 
<p>In MySQL 9.x, the <code>MD5()</code> and <code>SHA1()</code> SQL functions were moved out of the server core into a separate <code>classic_hashing</code> component. Amazon RDS for MySQL 9.7 doesn’t offer the <code>classic_hashing</code> component, so the <code>MD5()</code> and <code>SHA1()</code> SQL functions aren’t available.</p> 
<p>Before upgrading, review your application and stored programs for use of <code>MD5()</code> and <code>SHA1()</code>. For hashing needs, use <code>SHA2()</code> (for example, <code>SHA2(col, 256)</code>), which remains available. There is no supported <code>MD5()</code> replacement, so plan application changes where it’s used.</p> 
<h2>Transitional foreign key handling variables</h2> 
<p>Amazon RDS for MySQL 9.7 sets <code>innodb_native_foreign_keys=ON</code> by default, so foreign key checks and cascade operations are handled by the InnoDB engine, the same behavior as MySQL 8.4. Cascaded changes are therefore <strong>not</strong> individually written to the binary log and change-data-capture (CDC) tools such as AWS Database Migration Service (AWS DMS) and Debezium see the same events they did on 8.4.</p> 
<p>In community MySQL, <code>innodb_native_foreign_keys</code> defaults to OFF and the SQL layer performs foreign key cascades, which <em>does</em> write each cascaded change to the binary log, increasing binlog volume and surfacing those changes to CDC. The parameter is customer-modifiable, so if you set <code>innodb_native_foreign_keys=OFF</code> on RDS, plan for those additional binlog entries.</p> 
<p><code>innodb_native_foreign_keys</code> and <code>enable_cascade_triggers</code> are transitional variables introduced in the 9.x series (<code>innodb_native_foreign_keys</code> in 9.6, <code>enable_cascade_triggers</code> in 9.7) and already marked for eventual removal, so avoid building long-term dependencies on them.</p> 
<h2>Amazon RDS for MySQL 9.7 enhancements</h2> 
<p>Beyond the community engine changes described earlier, Amazon RDS for MySQL 9.7 makes several RDS-specific parameter-group and default choices. The changes most likely to affect an existing 8.4 workload are highlighted here.</p> 
<h3>Connections require TLS by default</h3> 
<p><code>require_secure_transport</code> now defaults to ON in RDS for MySQL 9.7, changed from OFF in 8.4. Clients that connect without SSL/TLS are rejected. Before upgrading, confirm that your applications and drivers connect over TLS, or set <code>require_secure_transport=OFF</code> in your parameter group if you must continue to allow non-TLS connections.</p> 
<h3>GTID defaults on RDS</h3> 
<p>Community MySQL changed the GTID defaults to ON in 9.5.0. Amazon RDS for MySQL 9.7 deliberately doesn’t adopt those values as-is:</p> 
<ul> 
 <li><code>enforce_gtid_consistency</code> stays OFF, unchanged from 8.4. Enable it explicitly if your workload needs GTID consistency enforcement.</li> 
 <li><code>gtid_mode</code> moves from <code>OFF_PERMISSIVE</code> (8.4) to <code>ON_PERMISSIVE</code>, promoting GTIDs rather than enforcing them.</li> 
</ul> 
<p>This keeps GTID behavior close to 8.4 while moving in the community’s direction.</p> 
<p>Because <code>ON_PERMISSIVE</code> assigns GTIDs to new transactions, two things change for replication. First, a replica running <code>gtid_mode=OFF</code> isn’t a supported target for a 9.7 source, although it was supported for an 8.4 source at <code>OFF_PERMISSIVE</code>. If you replicate from Amazon RDS for MySQL to a self-managed or external replica, set that replica to at least <code>OFF_PERMISSIVE</code> before you upgrade. Replication into 9.7 from a source without GTIDs is unaffected. Second, GTIDs now appear in the binary log, so verify any tool that reads binlogs directly. GTID auto-positioning still requires <code>gtid_mode=ON</code>. Moving to ON requires setting <code>enforce_gtid_consistency=ON</code> first, which rejects GTID-unsafe statements such as a transaction that updates both InnoDB and MyISAM tables.</p> 
<h3>Group Replication (active-active clusters)</h3> 
<p>For <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-active-active-clusters.html" target="_blank" rel="noopener">active-active clusters that use the MySQL Group Replication plugin</a>, <code>group_replication_ssl_mode</code> now defaults to REQUIRED, enforcing SSL for Group Replication communication.</p> 
<h3>Authentication changes</h3> 
<ul> 
 <li><strong>authentication_policy</strong> can’t name <code>mysql_native_password</code>, because the plugin is removed from the engine. Set it to <code>caching_sha2_password</code> instead (see <em>Removal of mysql_native_password</em>).</li> 
 <li><strong>ssl_cipher is now dynamic</strong>: it can be changed without a restart (it was static in 8.4).</li> 
</ul> 
<h3>Password validation moves to the component</h3> 
<p>RDS for MySQL 9.7 uses component-based password validation exclusively (the plugin form is gone):</p> 
<ul> 
 <li>The <code>validate-password</code> parameter is replaced by <code>rds.validate_password</code>, which controls installation of the <code>validate_password</code> component.</li> 
 <li>The underscore-style variables are renamed to dot-notation, for example <code>validate_password_length</code> becomes <code>validate_password.length</code> and <code>validate_password_policy</code> becomes <code>validate_password.policy</code>. Two new checks are available: <code>validate_password.changed_characters_percentage</code> and <code>validate_password.check_user_name</code>.</li> 
</ul> 
<p>Because the parameters are renamed, your existing <code>validate_password_*</code> values don’t carry across the upgrade. A value set in your 8.4 parameter group isn’t applied in 9.7, and password validation falls back to the component defaults, which might be weaker than your current policy. Record your current <code>validate_password_*</code> values before you upgrade, then set the matching <code>validate_password.*</code> parameters in your 9.7 parameter group. Also update any automation that references the old <code>validate_password_*</code> names or the <code>validate-password</code> plugin.</p> 
<h3>New customer-visible parameters</h3> 
<p>RDS for MySQL 9.7 exposes several new modifiable parameters:</p> 
<table border="1px" width="100%" cellpadding="10px"> 
 <tbody> 
  <tr> 
   <td><strong>Parameter</strong></td> 
   <td><strong>Purpose</strong></td> 
  </tr> 
  <tr> 
   <td><code>activate_mandatory_roles</code></td> 
   <td>Activate mandatory roles in addition to an account’s default roles (new in 9.7. Defaults to ON)</td> 
  </tr> 
  <tr> 
   <td><code>innodb_native_foreign_keys</code></td> 
   <td>InnoDB-level foreign key checks and cascades. Defaults to ON on RDS (same as 8.4). See <em>Transitional foreign key handling variables</em></td> 
  </tr> 
  <tr> 
   <td><code>caching_sha2_password_storage_format</code></td> 
   <td>Storage format for <code>caching_sha2_password</code> credentials (<code>CRYPT5</code> or <code>PBKDF2_SHA512</code>. Default <code>CRYPT5</code>)</td> 
  </tr> 
  <tr> 
   <td><code>caching_sha2_password_proxy_users</code></td> 
   <td>Whether <code>caching_sha2_password</code> supports proxy users (effective only with <code>check_proxy_users</code>)</td> 
  </tr> 
  <tr> 
   <td><code>enable_cascade_triggers</code></td> 
   <td>Execute triggers on child tables during foreign key cascade operations (transitional)</td> 
  </tr> 
  <tr> 
   <td><code>table_open_cache_triggers</code></td> 
   <td>Maximum number of cached open tables with loaded triggers</td> 
  </tr> 
  <tr> 
   <td><code>connection_memory_status_limit</code></td> 
   <td>Per-connection memory threshold before the status counter increments</td> 
  </tr> 
  <tr> 
   <td><code>global_connection_memory_status_limit</code></td> 
   <td>Total connection-memory threshold before the status counter increments</td> 
  </tr> 
 </tbody> 
</table> 
<h3>Other parameter changes</h3> 
<ul> 
 <li><strong>Removed variables:</strong> see the preceding <em>Removed and deprecated variables</em> section (<code>replica_parallel_type</code>, <code>innodb_undo_tablespaces</code> and <code>temptable_use_mmap</code>).</li> 
 <li><strong>replica_parallel_workers</strong> minimum allowed value changes from 0 to 1: single-threaded applier mode is gone.</li> 
 <li><strong>binlog_transaction_dependency_history_size</strong> allowed range on RDS expands to 1 to 100,000,000.</li> 
 <li><strong>Character sets:</strong> the <code>utf8</code> alias appears as <code>utf8mb3</code> in the allowed values for the <code>character_set_*</code> parameters and the <code>utf8</code> collations are renamed to <code>utf8mb3_*</code>.</li> 
</ul> 
<h2>Migration checklist</h2> 
<p>The following checklist recaps the changes covered in this post so that you can work through them as a single list when upgrading from MySQL 8.4 to 9.7. The list moves from tasks to complete before you start the upgrade, through changed defaults to review, to new capabilities worth evaluating after you’re running 9.7.</p> 
<ol type="1"> 
 <li>If RDS Proxy isn’t attached, drop the <code>rdsproxyadmin</code> user if it exists. In MySQL 9.7, <code>rdsproxyadmin</code> becomes a reserved user on all hosts and must be absent before you upgrade. If RDS Proxy is attached, it creates and manages this user, so leave it in place.</li> 
 <li>Confirm your deployment type is supported. Multi-AZ DB clusters (two readable standbys) aren’t supported at launch and upgrades from an 8.4 Multi-AZ DB cluster are blocked.</li> 
 <li>Migrate any accounts using <code>mysql_native_password</code> to <code>caching_sha2_password</code> (the plugin is removed as of 9.0) and update client drivers.</li> 
 <li>Review your application and stored programs for <code>MD5()</code> and <code>SHA1()</code>. These SQL functions aren’t available in RDS for MySQL 9.7. For hashing, use <code>SHA2()</code> (for example, <code>SHA2(col, 256)</code>), which remains available. There is no supported <code>MD5()</code> replacement, so plan application changes where it’s used.</li> 
 <li>Move password validation from the <code>validate-password</code> plugin to the <code>validate_password</code> component. RDS for MySQL 9.7 provides the component form only: the plugin is replaced by <code>rds.validate_password</code> and the <code>validate_password_*</code> variables are renamed to dot notation, for example <code>validate_password_length</code> becomes <code>validate_password.length</code>. Update automation that references the plugin or the old variable names.</li> 
 <li>Confirm that applications and drivers connect over SSL/TLS. <code>require_secure_transport</code> now defaults to ON, so non-TLS connections are rejected. Set <code>require_secure_transport=OFF</code> in your parameter group if you must allow them.</li> 
 <li>Update monitoring and automation that read the semisynchronous replication variables and status. They were renamed from master and slave to source and replica, for example <code>rpl_semi_sync_master_timeout</code> becomes <code>rpl_semi_sync_source_timeout</code>. These parameters are visible but RDS-managed, so you can’t set them yourself.</li> 
 <li>Remove references to <code>replica_parallel_type</code> and <code>group_replication_allow_local_lower_version_join</code>, which are removed in the 9.x series.</li> 
 <li>Check for schema objects named with the new reserved keywords <code>EXTERNAL</code> and <code>LIBRARY</code> (reserved as of MySQL 9.7). A reserved word must be quoted with backticks to be used as an identifier, so quote or rename any database, table, or column that uses them. When you create a 9.7 instance, don’t pass <code>EXTERNAL</code> or <code>LIBRARY</code> as the initial database name (<code>--db-name</code>) in the <code>create-db-instance</code> call, because a reserved word isn’t accepted there.</li> 
 <li>Review <code>explain_format</code>. The default changed from TRADITIONAL to TREE, so update any tooling that parses <code>EXPLAIN</code> output. Note that <code>explain_json_format_version</code> now defaults to 2.</li> 
 <li>Review <code>innodb_change_buffering</code>. The default changed from none to all.</li> 
 <li>Review <code>innodb_log_writer_threads</code>. The 8.4 default was always ON. In 9.7 it’s binary-log and CPU aware, so it can default to OFF on smaller instances.</li> 
 <li>Review <code>binlog_transaction_dependency_history_size</code>. The default increased from 25,000 to 10,000,000 (allowed range up to 100,000,000).</li> 
 <li>Confirm the RDS for MySQL 9.7 defaults for <code>gtid_mode</code> and <code>enforce_gtid_consistency</code>. Community MySQL changed both to ON in 9.5.0 and RDS manages them through the parameter group.</li> 
 <li>Consider JSON Duality Views for JSON-relational workloads.</li> 
 <li>Evaluate the Hypergraph Optimizer for complex, multi-join queries. It’s opt-in through <code>optimizer_switch</code> and disabled by default.</li> 
 <li>Test thoroughly in a Blue/Green deployment, on the Green environment, before promoting.</li> 
</ol> 
<h2>Conclusion</h2> 
<p>In this post, we reviewed the MySQL community versioning model, the new capabilities in MySQL 9.7, key security and replication changes, removed and transitional variables and a recommended upgrade approach using Amazon RDS Blue/Green deployments. We encourage you to test MySQL 9.7’s new features with your workloads. To get started, create or upgrade a database instance on the <a href="https://console.aws.amazon.com/rds/" target="_blank" rel="noopener">Amazon RDS console</a>. For more information, see the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html" target="_blank" rel="noopener">Amazon RDS for MySQL User Guide</a>.</p> 
<h2>Sources</h2> 
<ul> 
 <li><a href="https://dev.mysql.com/doc/relnotes/mysql/9.7/en" target="_blank" rel="noopener">MySQL 9.7 Release Notes</a>.</li> 
 <li><a href="https://dev.mysql.com/doc/relnotes/mysql/9.5/en/news-9-5-0.html" target="_blank" rel="noopener">MySQL 9.5.0 Release Notes</a>.</li> 
 <li><a href="https://dev.mysql.com/doc/refman/9.7/en/mysql-nutshell.html" target="_blank" rel="noopener">MySQL 9.7 What Is New</a>.</li> 
 <li><a href="https://dev.mysql.com/doc/refman/9.7/en/added-deprecated-removed.html" target="_blank" rel="noopener">MySQL 9.7 Variables Added, Deprecated and Removed</a>.</li> 
 <li><a href="https://aws.amazon.com/about-aws/whats-new/2026/06/rds-mysql-es-extension/" target="_blank" rel="noopener">Amazon Aurora and RDS for MySQL expand Extended Support for MySQL 5.7 through June 2029</a>.</li> 
</ul> 
<p> </p> 
<hr> 
<h2>About the authors</h2> 
<footer> 
 <div class="blog-author-box"> 
  <div class="blog-author-image">
   <img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/06/DBBLOG-5727-1.jpg" alt="Sabalesh Mahajan" width="100" height="133">
  </div> 
  <h3 class="lb-h4">Sabalesh Mahajan</h3> 
  <p>Sabalesh is a Senior Database Engineer at Amazon Web Services, with 19 years of experience in databases. He collaborates closely with the Amazon RDS product teams, contributing his expertise to the development and enhancement of innovative products.</p> 
 </div> 
 <div class="blog-author-box"> 
  <div class="blog-author-image">
   <img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/06/DBBLOG-5727-2.jpg" alt="Akshat Nehra" width="100" height="133">
  </div> 
  <h3 class="lb-h4">Akshat Nehra</h3> 
  <p>Akshat is a Software Development Engineer on the Amazon RDS for MySQL and MariaDB Engine team at Amazon Web Services, where he mostly works on the data plane. He focuses on delivering new community engine versions, contributing fixes upstream to the MySQL and MariaDB communities and security enhancements, helping make engine upgrades safe, reliable and seamless for customers.</p> 
 </div> 
 <div class="blog-author-box"> 
  <div class="blog-author-image">
   <img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/06/DBBLOG-5727-3.jpg" alt="Chelluru V N S S Vidyadhar" width="100" height="133">
  </div> 
  <h3 class="lb-h4">Chelluru V N S S Vidyadhar</h3> 
  <p>Chelluru is a Senior Database Engineer at Amazon Web Services with 14 years of experience in open source databases.</p> 
 </div> 
</footer>]]></content:encoded>
    <pubDate>Wed, 18 Nov 2026 20:52:19 +0000</pubDate>
    <dc:creator>AWS Database Blog</dc:creator>
    <category>Announcements</category>
    <category>Intermediate (200)</category>
    <category>RDS for MySQL</category>
  </item>

  <item>
    <title>Village News: MySQL News + Events (18 November 2026)</title>
    <guid isPermaLink="false">6afdec57cbc1950001276ca2</guid>
    <link>https://villagesql.com/blog/village-news-mysql-news-events-18-november-2026/</link>
    <description>Welcome back to Village News, our curated roundup of MySQL and database news. This week, MariaDB shipped its Q4 maintenance releases across four LTS series, and OurSQL Foundation opened a call for MySQL-compatible projects to share its FOSDEM 2027 booth.
If you want to get these updates, just subscribe to the blog.
Enjoy!
MySQL News
Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated)
David Stokes: &quot;Nine Times Out of Ten, It's MySQL&quot;
Maria Nesterova, OurSQL Foundation Blog
TL;DR - An interview with David Stokes, now Community Manager at DBeaver, who worked on MySQL under MySQL AB, Sun and Oracle. He covers running MySQL certification and why he calls Postgres &quot;free as in puppy.&quot;
Presentation: Behind Booking.com's AI Evolution: The Unpolished Story
Jabez Eliezer Manuel, InfoQ
TL;DR - A 20-year history of Booking.com's data platform: scaling MySQL topologies, retiring Hadoop for the cloud, and serving 400B ML predictions a day.
MySQL Community Public Meeting #6: Building Momentum Together
The Oracle MySQL Blog
TL;DR - A recap of Oracle's sixth public MySQL community meeting, held October 22, on more capabilities in Community Edition, community contributions reaching users, and clearer ways to get involved.
MySQL tables in four open source Iceberg catalogs, read by Trino and DuckDB
dbtrail Blog
TL;DR - MySQL tables were exported from the binlog into Nessie, LakeKeeper, Polaris and Gravitino. Trino and DuckDB matched MySQL in all four catalogs, but PyIceberg can't yet read the incremental exports because it lacks equality-delete support.
Public Preview: Azure Database for MySQL on Premium SSD v2
Azure Updates
TL;DR - Azure Database for MySQL now offers Premium SSD v2 storage in preview, which scales capacity and performance independently. Server rename also went GA the same day, so a server can be renamed without being recreated.
How VillageSQL does documentation
Adam Levin, VillageSQL
TL;DR - The VillageSQL docs live in a public GitHub repo. An AI agent drafts updates for pages that lag the code and tests their SQL examples against a live server, then a critic agent and a human reviewer check each change before it merges.
MySQL Development as it Happens, Part 2 – Vector Indexes
The Oracle MySQL Blog
TL;DR - The second post in Oracle's series on MySQL Server development covers VECTOR INDEX support, part of the &quot;AI &amp;amp; Cloud&quot; focus area.
Share Our Booth at FOSDEM 2027: Call for MySQL-Compatible Projects
OurSQL Foundation Blog
TL;DR - OurSQL Foundation invites open source MySQL-compatible projects to share its booth at FOSDEM 2027 in Brussels (January 30–31).
Another way to run DuckDB 2.0 inside of MySQL
WarpSQL Blog
TL;DR - WarpSQL updates &quot;coldfront,&quot; a HeatWave-style option where you choose tables to load into a DuckDB in-memory column store. It ships in the next beta binary, or you can build it from source now.
Database News
MariaDB Server 12.3, 11.8, 11.4 and 10.11 – Q4 2026 Maintenance Releases
Frédéric Descamps, MariaDB Foundation Blog
TL;DR - MariaDB 12.3.4, 11.8.10, 11.4.14 and 10.11.20 shipped November 16. They include security fixes in the CONNECT engine and in privilege checks, plus InnoDB crash-recovery, Galera, replication and optimizer fixes.
A 9-TiB OLTP Benchmark on PostgreSQL
Franck Pachot
TL;DR - Pachot reproduces the self-managed PostgreSQL baseline (~50K NOPM) behind Azure HorizonDB's &quot;8x faster&quot; claim and finds data-file reads are the first bottleneck. Three follow-up posts trace it on through WAL sync waits to Linux writeback throttling.
Built on object storage, served from cache
Carlota Soto, Databricks Blog
TL;DR - Lakebase Postgres keeps object storage off the query path. About 99.5% of page reads hit the compute cache, the rest come from pageserver NVMe, and only ~39 requests/s reach object storage.
Hunting Performance Regressions Through Years of MariaDB History
Jonathan Miller, MariaDB Foundation Blog
TL;DR - Point-query workloads ran ~50% below the 10.3 baseline (MDEV-38049). Testing builds across years of history traced the recovery to MDEV-33966's buffer-page contention fix, which raised throughput from ~187K to 402K TPS.
PostgreSQL large objects: Best practices for Amazon Aurora and RDS
Sachin Murkar, AWS Database Blog
TL;DR - Autovacuum never reclaims orphaned large objects. This guide covers monitoring them, running vacuumlo safely, lo_manage triggers and avoiding OID exhaustion.
Query Timeouts That Follow the Query, Not the User
ProxySQL Blog
TL;DR - ProxySQL 3.0.10 adds a per-hostgroup default_query_timeout, so analytics replicas and OLTP writers can have different limits without any query rules.
Talks: Please Implement A Quorum Database
Alex Miller, transactional.blog
TL;DR - In this HPTS'26 talk, Miller argues that leaderless quorum replication is now practical for production. Newer protocols such as QuePaxa, Accord and escrow-based designs meet the latency needs older approaches missed.
Upcoming Database Events
PgConf.Nepal 2026
November 18-21, 2026
Pulchowk Campus, Tribhuvan University
Kathmandu, Nepal
PGConf.PL 2026
November 24, 2026
ARCHE Dwór Uphagena
Gdańsk, Poland
PostgreSQL Conference Japan 2026
November 27, 2026
AP Tokyo Yaesu
Tokyo, Japan
AWS re:Invent 2026
November 30 – December 4, 2026
Las Vegas, NV
OurSQL Foundation Webinar: What Is the Fastest Database? Open-Source Benchmarking with HammerDB
December 3, 2026, 10:00 AM ET / 16:00 CET
Online
Open Source Summit Japan
December 7-9, 2026
Tokyo, Japan
CIDR 2027
January 24-27, 2027
Mövenpick Hotel Amsterdam City Centre
Amsterdam, The Netherlands
MySQL Belgian Days 2027
January 28-29, 2027
ICAB Incubator
Brussels, Belgium
FOSDEM PGDay 2027
January 28-29, 2027
Brussels Marriott Hotel Grand Place
Brussels, Belgium
FOSDEM 2027
January 30-31, 2027
ULB Solbosch Campus
Brussels, Belgium
CERN PGDay 2027
February 12, 2027
CERN Council Chamber
Geneva, Switzerland
ConFoo 2027
February 24-26, 2027
Hotel Bonaventure
Montréal, QC, Canada
PGConf India 2027
March 2-5, 2027
Sheraton Grand Hotel at Brigade Gateway
Bengaluru, India
FOSSASIA PGDay 2027
March 9, 2027
Bangkok, Thailand
KubeCon + CloudNativeCon Europe 2027
March 15-18, 2027
Barcelona, Spain
Nordic PGDay 2027
March 16, 2027
Courtyard Kungsholmen
Stockholm, Sweden
POSETTE: An Event for Postgres 2027
March 30-31, 2027
Online
SCaLE 24x
April 1-4, 2027
Pasadena Convention Center
Pasadena, CA
DuckCon #8
April 7, 2027
De Wester (Westergas)
Amsterdam, The Netherlands
PgDay Boston 2027
April 10, 2027
Tufts University Joyce Cummings Center
Boston, MA
PGConf.DE 2027
April 13-14, 2027
Haus der Technik
Essen, Germany
Postgres Conference 2027
April 21-23, 2027
San Jose Marriott
San Jose, CA
Percona Live Bay Area 2027
May 11-13, 2027
Computer History Museum
Mountain View, CA
PGConf.dev 2027
May 11-14, 2027
Plaza Centre-Ville
Montréal, QC, Canada</description>
    <content:encoded><![CDATA[<img src="https://storage.ghost.io/c/db/c7/dbc78ca4-dcfe-468b-b2fa-030396d2a98e/content/images/2026/11/village-news-banner-3.jpg" alt="Village News: MySQL News + Events (18 November 2026)"><p>Welcome back to Village News, our curated roundup of MySQL and database news. This week, MariaDB shipped its Q4 maintenance releases across four LTS series, and OurSQL Foundation opened a call for MySQL-compatible projects to share its FOSDEM 2027 booth.</p>
<p>If you want to get these updates, just <a href="https://villagesql.com/blog/blog#/portal/signup">subscribe</a> to the blog.</p>
<p>Enjoy!</p>
<h2>MySQL News</h2>
<p><em>Note: Aggregated MySQL news can be found at <a href="https://planet.oursqlcommunity.org/?source=villagesql">Planet for MySQL Community</a> and <a href="https://planet.mysql.com/">Planet MySQL</a> (Oracle curated)</em></p>
<p><strong><a href="https://oursqlfoundation.org/blog/david-stokes-interview/">David Stokes: "Nine Times Out of Ten, It's MySQL"</a></strong><br>
<em>Maria Nesterova, OurSQL Foundation Blog</em></p>
<p><strong>TL;DR -</strong> An interview with David Stokes, now Community Manager at DBeaver, who worked on MySQL under MySQL AB, Sun and Oracle. He covers running MySQL certification and why he calls Postgres "free as in puppy."</p>
<p><strong><a href="https://www.infoq.com/presentations/bookingcom-ml-data-platform-evolution/">Presentation: Behind Booking.com's AI Evolution: The Unpolished Story</a></strong><br>
<em>Jabez Eliezer Manuel, InfoQ</em></p>
<p><strong>TL;DR -</strong> A 20-year history of Booking.com's data platform: scaling MySQL topologies, retiring Hadoop for the cloud, and serving 400B ML predictions a day.</p>
<p><strong><a href="https://blogs.oracle.com/mysql/mysql-community-public-meeting-6">MySQL Community Public Meeting #6: Building Momentum Together</a></strong><br>
<em>The Oracle MySQL Blog</em></p>
<p><strong>TL;DR -</strong> A recap of Oracle's sixth public MySQL community meeting, held October 22, on more capabilities in Community Edition, community contributions reaching users, and clearer ways to get involved.</p>
<p><strong><a href="https://blog.dbtrail.com/mysql-tables-in-four-open-source-iceberg-catalogs-read-by-trino-and-duckdb/">MySQL tables in four open source Iceberg catalogs, read by Trino and DuckDB</a></strong><br>
<em>dbtrail Blog</em></p>
<p><strong>TL;DR -</strong> MySQL tables were exported from the binlog into Nessie, LakeKeeper, Polaris and Gravitino. Trino and DuckDB matched MySQL in all four catalogs, but PyIceberg can't yet read the incremental exports because it lacks equality-delete support.</p>
<p><strong><a href="https://azure.microsoft.com/updates?id=577515">Public Preview: Azure Database for MySQL on Premium SSD v2</a></strong><br>
<em>Azure Updates</em></p>
<p><strong>TL;DR -</strong> Azure Database for MySQL now offers Premium SSD v2 storage in preview, which scales capacity and performance independently. Server rename also went GA the same day, so a server can be renamed without being recreated.</p>
<p><strong><a href="https://villagesql.com/blog/documentation/">How VillageSQL does documentation</a></strong><br>
<em>Adam Levin, VillageSQL</em></p>
<p><strong>TL;DR -</strong> The VillageSQL docs live in a public GitHub repo. An AI agent drafts updates for pages that lag the code and tests their SQL examples against a live server, then a critic agent and a human reviewer check each change before it merges.</p>
<p><strong><a href="https://blogs.oracle.com/mysql/mysql-development-as-it-happens-part-2-vector-indexes">MySQL Development as it Happens, Part 2 – Vector Indexes</a></strong><br>
<em>The Oracle MySQL Blog</em></p>
<p><strong>TL;DR -</strong> The second post in Oracle's series on MySQL Server development covers VECTOR INDEX support, part of the "AI &amp; Cloud" focus area.</p>
<p><strong><a href="https://oursqlfoundation.org/blog/fosdem-2027-booth-call-for-projects/">Share Our Booth at FOSDEM 2027: Call for MySQL-Compatible Projects</a></strong><br>
<em>OurSQL Foundation Blog</em></p>
<p><strong>TL;DR -</strong> OurSQL Foundation invites open source MySQL-compatible projects to share its booth at FOSDEM 2027 in Brussels (January 30–31).</p>
<p><strong><a href="https://warpsqlblog.wordpress.com/2026/11/16/another-way-to-run-duckdb-2-0-inside-of-mysql/">Another way to run DuckDB 2.0 inside of MySQL</a></strong><br>
<em>WarpSQL Blog</em></p>
<p><strong>TL;DR -</strong> WarpSQL updates "coldfront," a HeatWave-style option where you choose tables to load into a DuckDB in-memory column store. It ships in the next beta binary, or you can build it from source now.</p>
<h2>Database News</h2>
<p><strong><a href="https://mariadb.org/mariadb-server-12-3-11-8-11-4-and-10-11-q4-2026-maintenance-releases/">MariaDB Server 12.3, 11.8, 11.4 and 10.11 – Q4 2026 Maintenance Releases</a></strong><br>
<em>Frédéric Descamps, MariaDB Foundation Blog</em></p>
<p><strong>TL;DR -</strong> MariaDB 12.3.4, 11.8.10, 11.4.14 and 10.11.20 shipped November 16. They include security fixes in the CONNECT engine and in privilege checks, plus InnoDB crash-recovery, Galera, replication and optimizer fixes.</p>
<p><strong><a href="https://dev.to/franckpachot/a-9-tib-oltp-benchmark-on-postgresql-4jh0">A 9-TiB OLTP Benchmark on PostgreSQL</a></strong><br>
<em>Franck Pachot</em></p>
<p><strong>TL;DR -</strong> Pachot reproduces the self-managed PostgreSQL baseline (~50K NOPM) behind Azure HorizonDB's "8x faster" claim and finds data-file reads are the first bottleneck. Three follow-up posts trace it on through WAL sync waits to Linux writeback throttling.</p>
<p><strong><a href="https://www.databricks.com/blog/built-object-storage-served-cache">Built on object storage, served from cache</a></strong><br>
<em>Carlota Soto, Databricks Blog</em></p>
<p><strong>TL;DR -</strong> Lakebase Postgres keeps object storage off the query path. About 99.5% of page reads hit the compute cache, the rest come from pageserver NVMe, and only ~39 requests/s reach object storage.</p>
<p><strong><a href="https://mariadb.org/hunting-performance-regressions-through-years-of-mariadb-history/">Hunting Performance Regressions Through Years of MariaDB History</a></strong><br>
<em>Jonathan Miller, MariaDB Foundation Blog</em></p>
<p><strong>TL;DR -</strong> Point-query workloads ran ~50% below the 10.3 baseline (MDEV-38049). Testing builds across years of history traced the recovery to MDEV-33966's buffer-page contention fix, which raised throughput from ~187K to 402K TPS.</p>
<p><strong><a href="https://aws.amazon.com/blogs/database/postgresql-large-objects-best-practices-for-amazon-aurora-and-rds/">PostgreSQL large objects: Best practices for Amazon Aurora and RDS</a></strong><br>
<em>Sachin Murkar, AWS Database Blog</em></p>
<p><strong>TL;DR -</strong> Autovacuum never reclaims orphaned large objects. This guide covers monitoring them, running vacuumlo safely, lo_manage triggers and avoiding OID exhaustion.</p>
<p><strong><a href="https://proxysql.com/blog/per-hostgroup-query-timeouts/">Query Timeouts That Follow the Query, Not the User</a></strong><br>
<em>ProxySQL Blog</em></p>
<p><strong>TL;DR -</strong> ProxySQL 3.0.10 adds a per-hostgroup default_query_timeout, so analytics replicas and OLTP writers can have different limits without any query rules.</p>
<p><strong><a href="https://transactional.blog/talk/please-implement-a-quorum-database.html">Talks: Please Implement A Quorum Database</a></strong><br>
<em>Alex Miller, transactional.blog</em></p>
<p><strong>TL;DR -</strong> In this HPTS'26 talk, Miller argues that leaderless quorum replication is now practical for production. Newer protocols such as QuePaxa, Accord and escrow-based designs meet the latency needs older approaches missed.</p>
<h2>Upcoming Database Events</h2>
<p><strong><a href="https://2026.pgconf.org.np/">PgConf.Nepal 2026</a></strong><br>
November 18-21, 2026<br>
Pulchowk Campus, Tribhuvan University<br>
Kathmandu, Nepal</p>
<p><strong><a href="https://2026.pgconf.pl/">PGConf.PL 2026</a></strong><br>
November 24, 2026<br>
ARCHE Dwór Uphagena<br>
Gdańsk, Poland</p>
<p><strong><a href="https://www.postgresql.jp/jpug-pgcon2026">PostgreSQL Conference Japan 2026</a></strong><br>
November 27, 2026<br>
AP Tokyo Yaesu<br>
Tokyo, Japan</p>
<p><strong><a href="https://aws.amazon.com/events/reinvent">AWS re:Invent 2026</a></strong><br>
November 30 – December 4, 2026<br>
Las Vegas, NV</p>
<p><strong><a href="https://oursqlfoundation.org/community/webinars/fastest-database-hammerdb/">OurSQL Foundation Webinar: What Is the Fastest Database? Open-Source Benchmarking with HammerDB</a></strong><br>
December 3, 2026, 10:00 AM ET / 16:00 CET<br>
Online</p>
<p><strong><a href="https://events.linuxfoundation.org/open-source-summit-japan/">Open Source Summit Japan</a></strong><br>
December 7-9, 2026<br>
Tokyo, Japan</p>
<p><strong><a href="https://www.cidrdb.org/cidr2027/">CIDR 2027</a></strong><br>
January 24-27, 2027<br>
Mövenpick Hotel Amsterdam City Centre<br>
Amsterdam, The Netherlands</p>
<p><strong><a href="https://sessionize.com/mysql-belgian-days-2027/">MySQL Belgian Days 2027</a></strong><br>
January 28-29, 2027<br>
ICAB Incubator<br>
Brussels, Belgium</p>
<p><strong><a href="https://2027.fosdempgday.org/">FOSDEM PGDay 2027</a></strong><br>
January 28-29, 2027<br>
Brussels Marriott Hotel Grand Place<br>
Brussels, Belgium</p>
<p><strong><a href="https://fosdem.org/2027/">FOSDEM 2027</a></strong><br>
January 30-31, 2027<br>
ULB Solbosch Campus<br>
Brussels, Belgium</p>
<p><strong><a href="https://indico.cern.ch/event/1720940/">CERN PGDay 2027</a></strong><br>
February 12, 2027<br>
CERN Council Chamber<br>
Geneva, Switzerland</p>
<p><strong><a href="https://confoo.ca/en/2027">ConFoo 2027</a></strong><br>
February 24-26, 2027<br>
Hotel Bonaventure<br>
Montréal, QC, Canada</p>
<p><strong><a href="https://pgconf.in/">PGConf India 2027</a></strong><br>
March 2-5, 2027<br>
Sheraton Grand Hotel at Brigade Gateway<br>
Bengaluru, India</p>
<p><strong><a href="https://www.postgresql.org/about/event/fossasia-pgday-2027-returns-to-thailand-on-march-9-cfp-open-2611/">FOSSASIA PGDay 2027</a></strong><br>
March 9, 2027<br>
Bangkok, Thailand</p>
<p><strong><a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/">KubeCon + CloudNativeCon Europe 2027</a></strong><br>
March 15-18, 2027<br>
Barcelona, Spain</p>
<p><strong><a href="https://2027.nordicpgday.org/">Nordic PGDay 2027</a></strong><br>
March 16, 2027<br>
Courtyard Kungsholmen<br>
Stockholm, Sweden</p>
<p><strong><a href="https://posetteconf.com/2027/">POSETTE: An Event for Postgres 2027</a></strong><br>
March 30-31, 2027<br>
Online</p>
<p><strong><a href="https://www.socallinuxexpo.org/scale/24x">SCaLE 24x</a></strong><br>
April 1-4, 2027<br>
Pasadena Convention Center<br>
Pasadena, CA</p>
<p><strong><a href="https://duckdb.org/events/2027/04/07/duckcon8/">DuckCon #8</a></strong><br>
April 7, 2027<br>
De Wester (Westergas)<br>
Amsterdam, The Netherlands</p>
<p><strong><a href="https://2027.pgdayboston.org/">PgDay Boston 2027</a></strong><br>
April 10, 2027<br>
Tufts University Joyce Cummings Center<br>
Boston, MA</p>
<p><strong><a href="https://2027.pgconf.de/">PGConf.DE 2027</a></strong><br>
April 13-14, 2027<br>
Haus der Technik<br>
Essen, Germany</p>
<p><strong><a href="https://postgresconf.org/conferences/2027">Postgres Conference 2027</a></strong><br>
April 21-23, 2027<br>
San Jose Marriott<br>
San Jose, CA</p>
<p><strong><a href="https://perconalive.com/">Percona Live Bay Area 2027</a></strong><br>
May 11-13, 2027<br>
Computer History Museum<br>
Mountain View, CA</p>
<p><strong><a href="https://2027.pgconf.dev/">PGConf.dev 2027</a></strong><br>
May 11-14, 2027<br>
Plaza Centre-Ville<br>
Montréal, QC, Canada</p>]]></content:encoded>
    <pubDate>Wed, 18 Nov 2026 18:51:10 +0000</pubDate>
    <dc:creator>VillageSQL</dc:creator>
  </item>

  <item>
    <title>David Stokes: “Nine Times Out of Ten, It’s MySQL”</title>
    <guid isPermaLink="false">https://oursqlfoundation.org/blog/david-stokes-interview/</guid>
    <link>https://oursqlfoundation.org/blog/david-stokes-interview/</link>
    <description>David Stokes has been part of MySQL’s story under every one of its owners. He joined MySQL AB as a PHP developer, ran its certification program, and stayed through the Sun and Oracle acquisitions, spending about a decade on Oracle’s MySQL Community team alongside Frédéric Descamps. After Oracle and a spell at Percona, he now works at DBeaver, a database tool that connects to more than 170 databases, which gives him an unusually wide view of where MySQL stands. We talked about what each owner did to MySQL, why he walked away from Oracle, why Postgres is “free as in puppy”, and what the MySQL world must do to win the next generation of developers.</description>
    <content:encoded><![CDATA[<p><em>David Stokes has been part of MySQL’s story under every one of its owners. He joined MySQL AB as a PHP developer, ran its certification program, and stayed through the Sun and Oracle acquisitions, spending about a decade on Oracle’s MySQL Community team alongside Frédéric Descamps. After Oracle and a spell at Percona, he now works at DBeaver, a database tool that connects to more than 170 databases, which gives him an unusually wide view of where MySQL stands. We talked about what each owner did to MySQL, why he walked away from Oracle, why Postgres is “free as in puppy”, and what the MySQL world must do to win the next generation of developers.</em></p>]]></content:encoded>
    <pubDate>Wed, 18 Nov 2026 00:00:00 +0000</pubDate>
    <dc:creator>OurSQL Foundation</dc:creator>
  </item>

  <item>
    <title>MySQL Shorts -  Episode 107 - MySQL Transactions: Maintain Data Consistency</title>
    <guid isPermaLink="false">planet-manual-fc7e43b6adc4cd4491d67d12bab79b6c</guid>
    <link>https://www.youtube.com/watch?v=kuAAXCtNavI&amp;list=PLWx5a9Tn2EvG4C90YFJ9eU61IpALeE0SN&amp;index=1</link>
    <description>MySQL transactions help maintain data consistency by grouping related SQL updates so they can be committed together or rolled back when an operation fails. See how START TRANSACTION and ROLLBACK work with inventory updates and database constraints to reverse uncommitted changes, and explore the MySQL transaction documentation: https://dev.mysql.com/doc/refman/26.10/en/commit.html</description>
    <pubDate>Tue, 17 Nov 2026 19:41:48 +0000</pubDate>
    <dc:creator>MySQL Community Team</dc:creator>
  </item>

  <item>
    <title>MySQL Community Public Meeting #6: Building Momentum Together</title>
    <guid isPermaLink="false">2f119744261f39bed2e683b691c13862</guid>
    <link>https://blogs.oracle.com/mysql/mysql-community-public-meeting-6</link>
    <description>On October 22, 2026, we held our sixth public meeting with the MySQL community. The theme, “Continued Development Momentum,” captured the progress well: more capabilities in Community Edition, community contributions reaching users, and clearer paths for getting involved. Our priorities remain delivering innovation into Community Edition, growing the ecosystem, and making MySQL development more transparent […]</description>
    <pubDate>Tue, 17 Nov 2026 15:35:25 +0000</pubDate>
    <dc:creator>Oracle MySQL Group</dc:creator>
    <category>MySQL</category>
    <category>MySQL Community</category>
    <category>mysql</category>
    <category>MySQL 26.7</category>
    <category>MySQL 9.7 LTS</category>
  </item>

  <item>
    <title>How VillageSQL does documentation</title>
    <guid isPermaLink="false">6afb4004a5b39900015d2609</guid>
    <link>https://villagesql.com/blog/documentation/</link>
    <description>We've heard positive feedback about VillageSQL's documentation, so we wanted to share a bit more on how we build it and how we hope to evolve it. VillageSQL Server tracks upstream MySQL and adds an extension framework, so the documentation has two moving targets. We pull in upstream changes with each MySQL release, and VillageSQL's own changes land daily in the dev branch and ship with each release. We believe that it's important for a complex project to provide accurate documentation that is updated as frequently as code ships. This is easier said than done. We use AI extensively to generate our docs, but we believe it's critical to have a human as a review gate before any change is shipped. AI makes mistakes, so we make it prove every change to itself and to humans by testing against a running server.Frequent readers of the VillageSQL docs will also notice a recent redesign. The goal with that was to make the site easier to navigate. As with everything we do, we'd love your feedback.The docs are a public repositoryThe documentation lives in villagesql-docs on GitHub as a public repo under the same rules as the server repo. Anyone can submit a change as a pull request. If you want to know more about VillageSQL's approach to open source, read our post &quot;How VillageSQL does open source&quot;.The docs have English, Japanese, Korean, Chinese (Simplified), and Brazilian Portuguese versions. The translations are the product of AI. The current set was translated by Claude Fable 5.1 and then reviewed by Gemini 4, which compares each translated page with the English source page. A finding from the Gemini reviewer is verified by a third agent, Claude Fable 5.1, before it's applied, because the reviewer is as capable of being wrong as the translator. We have experimented with locally running models as both the translator and reviewer but found that hosted models hit the right price/performance bar for our current needs. We are grateful to the Brazilian MySQL community for feedback on the Portuguese translations.Why we like MintlifyThe docs site is hosted on Mintlify. We like Mintlify because their platform is flexible, intuitive for humans and agents to use, and their team is responsive and inquisitive. We picked Mintlify when we launched the documentation site and chose to stay on it through the recent redesign.The docs are just markdown files, so they are simple for humans and agents to work with. Mintlify automatically publishes a machine-readable index of the docs for AI agents. For a complex piece of software like a database, making the docs accessible to both humans and agents matters.We are starting to experiment with Mintlify's advanced features like Autopilot, which can read changes in the server repo and open pull requests against our docs repo, as a complement to the homegrown workflow below.An AI agent with a human as the gateEvery weekday, an agent looks for docs pages that have fallen behind the VillageSQL Server code and VillageSQL-authored extensions. It then drafts suggested changes. The server pages have a copy for each MySQL version we track (8.4 and 9.7). The agent covers those pages, the server README, and the README of every VillageSQL-authored extension.When we started down this path, the workflow ended at this draft stage. We quickly discovered that even AIs make mistakes (sarcasm). So, we introduced a workflow step that has mattered a lot: the agent pulls every SQL statement and command out of its own drafts and runs each one on a live VillageSQL Server build that is current up to the commit of the change being tested. If the output on the page doesn't match the output from the server, one of them is wrong, and the agent works out which before finalizing the docs change.Live verification catches what the agent initially got wrong. It doesn't catch what the agent didn't think to check, so a critic agent with none of the authoring agent's context checks the draft against its sources and flags any claim that has no source. The authoring agent gets two rounds to fix what the critic raises. Whatever still stands after that goes to a human to make a judgment call. During drafting, the work is split across smaller helper agents: each one owns one page or one pull request, and a coordinating agent reads their findings. Through trial and error, we have found that giving agents discrete tasks keeps each one's context small, and in our experience that means fewer invented claims when dealing with a complex codebase.Next comes the human review. The agent summarizes the workflow run: what changed, why, and what pull requests it recommends. A person says yes or no. On yes, the agent commits and opens draft pull requests, or adds commits to pull requests that are still in review. The agent groups the docs changes into one pull request per engineer whose server change triggered them, so the engineer who made the change can review it. As soon as that human review is complete and the pull request merges, the live docs site is rebuilt with the update.This workflow has evolved significantly over time. It started life as a simple workflow running on a laptop and is now an Anthropic Managed Agent, running on a Mac mini that interacts with its human reviewers via Slack. Some revisions are about accuracy, such as the critic pass and reading each pull request diff three times independently before trusting it. We can't seem to get better than an 80% accuracy score (tested by planting defects) in a single session, which is why we run so many passes over a single change.How the agent learnsThe agent starts every run drawing on a set of instructions about VillageSQL that has been built up through the many iterations of the workflow. There are three kinds of files. A rules file says how things are done: what a docs change looks like, where a new feature's documentation goes, and how error text is captured. A lessons file holds an accumulated list of assumptions and mistakes past agents have made. A history file holds the story behind each lesson, so a rule that looks arbitrary can be understood. When a person corrects the agent, the correction goes into those files before the session ends so the next run learns from it.Guardrails on every pageInstructions tell the agent what to do. Checks help keep it in line when it doesn't follow them. Like instructions, most of the checks we use come from trial and error.Every docs page about extensions, the SDK, or the server declares its audience in its front matter: someone writing an extension, someone contributing to the server, or someone using the database. The audience field forces the question &quot;who is this for?&quot; at the moment of writing. This helps prevent the AI from conflating topics on a page. For example, the raw ABI is what the SDK connects to, and the SDK is what extension authors connect to, so a page for extension authors should not drift into the raw ABI. That distinction was confusing to agents initially.Other checks read the code blocks on a page and compare them against a list of banned phrases and past mistakes. Documented table columns and system variables are checked against a running server. A code block that names a variable the server doesn't have, or uses a MySQL concept that isn't relevant to VillageSQL-specific documentation, gets flagged by name. Checks also look for AI-slop language we want to keep out of the docs, such as transition language (&quot;the server now calls&quot;, &quot;previously the extension&quot;), because a reference page describes the current state, and a reader who never saw the old page does not need the correction.The strictest rule is about error text. Any error message, error number, or warning on a page must be pasted verbatim from a running server. An agent never types one from memory or infers one from the source code. Some interesting lessons we have taught the agentBelow are some examples of lessons we have had to &quot;teach&quot; the agent.A stale server produces confident, wrong findings. Before anything is verified, we compare when the server was built with when the code last changed. A result from a stale build is reproducible and often false, which is the worst kind of wrong.AI adds. On its own, it rarely removes. When an agent finds a wrong sentence, its instinct is to add a note beside it rather than fix it. We make it change the sentence, and every review pass on our docs has asked for removals.A fix in one copy is a bug in the others. We maintain docs for multiple versions in multiple languages. A fix that lands in just one copy and misses the rest leaves the docs disagreeing with themselves. A rule is that a fix goes to every copy.The workflow is a means to an endAn open process that continuously improves the docs is the goal, so users have the latest as soon as code ships. Our current workflow is just a means to an end. AI tooling is improving fast enough that the best way to achieve our goals a year from now probably doesn't exist yet. We'll keep the rules, the lessons, and the checks, because those are ours, and we'll swap the tooling under them whenever something better comes along. If you'd like to contribute to the VillageSQL documentation, you can do so in the public repo.</description>
    <content:encoded><![CDATA[<img src="https://storage.ghost.io/c/db/c7/dbc78ca4-dcfe-468b-b2fa-030396d2a98e/content/images/2026/11/IMG_8197-clean.jpg" alt="How VillageSQL does documentation"><p>We've heard positive feedback about VillageSQL's <a href="http://villagesql.com/docs"><u>documentation</u></a>, so we wanted to share a bit more on how we build it and how we hope to evolve it. VillageSQL Server tracks upstream MySQL and adds an extension framework, so the documentation has two moving targets. We pull in upstream changes with each MySQL release, and VillageSQL's own changes land daily in the dev branch and ship with each release. We believe that it's important for a complex project to provide accurate documentation that is updated as frequently as code ships. This is easier said than done. We use AI extensively to generate our docs, but we believe it's critical to have a human as a review gate before any change is shipped. AI makes mistakes, so we make it prove every change to itself and to humans by testing against a running server.</p><p>Frequent readers of the VillageSQL docs will also notice a recent redesign. The goal with that was to make the site easier to navigate. As with everything we do, we'd love your <a href="https://discord.gg/KSr6whd3Fr"><u>feedback</u></a>.</p><h2>The docs are a public repository</h2><p>The documentation lives in <a href="https://github.com/villagesql/villagesql-docs"><u>villagesql-docs</u></a> on GitHub as a public repo under the same rules as the server repo. Anyone can submit a change as a pull request. If you want to know more about VillageSQL's approach to open source, read our post <a href="https://villagesql.com/blog/open-source/"><u>"How VillageSQL does open source"</u></a>.</p><p>The docs have English, Japanese, Korean, Chinese (Simplified), and Brazilian Portuguese versions. The translations are the product of AI. The current set was translated by Claude Fable 5.1 and then reviewed by Gemini 4, which compares each translated page with the English source page. A finding from the Gemini reviewer is verified by a third agent, Claude Fable 5.1, before it's applied, because the reviewer is as capable of being wrong as the translator. We have experimented with locally running models as both the translator and reviewer but found that hosted models hit the right price/performance bar for our current needs. We are grateful to the Brazilian MySQL community for feedback on the Portuguese translations.</p><h2>Why we like Mintlify</h2><p>The docs site is hosted on <a href="https://www.mintlify.com/" rel="noreferrer">Mintlify</a>. We like Mintlify because their platform is flexible, intuitive for humans and agents to use, and their team is responsive and inquisitive. We picked Mintlify when we launched the documentation site and chose to stay on it through the recent redesign.</p><p>The docs are just markdown files, so they are simple for humans and agents to work with. Mintlify automatically publishes a machine-readable index of the docs for AI agents. For a complex piece of software like a database, making the docs accessible to both humans and agents matters.</p><p>We are starting to experiment with Mintlify's advanced features like Autopilot, which can read changes in the server repo and open pull requests against our docs repo, as a complement to the homegrown workflow below.</p><h2>An AI agent with a human as the gate</h2><p>Every weekday, an agent looks for docs pages that have fallen behind the VillageSQL Server code and VillageSQL-authored extensions. It then drafts suggested changes. The server pages have a copy for each MySQL version we track (8.4 and 9.7). The agent covers those pages, the server README, and the README of every VillageSQL-authored extension.</p><p>When we started down this path, the workflow ended at this draft stage. We quickly discovered that even AIs make mistakes (sarcasm). So, we introduced a workflow step that has mattered a lot: the agent pulls every SQL statement and command out of its own drafts and runs each one on a live VillageSQL Server build that is current up to the commit of the change being tested. If the output on the page doesn't match the output from the server, one of them is wrong, and the agent works out which before finalizing the docs change.</p><p>Live verification catches what the agent initially got wrong. It doesn't catch what the agent didn't think to check, so a critic agent with none of the authoring agent's context checks the draft against its sources and flags any claim that has no source. The authoring agent gets two rounds to fix what the critic raises. Whatever still stands after that goes to a human to make a judgment call. During drafting, the work is split across smaller helper agents: each one owns one page or one pull request, and a coordinating agent reads their findings. Through trial and error, we have found that giving agents discrete tasks keeps each one's context small, and in our experience that means fewer invented claims when dealing with a complex codebase.</p><p>Next comes the human review. The agent summarizes the workflow run: what changed, why, and what pull requests it recommends. A person says yes or no. On yes, the agent commits and opens draft pull requests, or adds commits to pull requests that are still in review. The agent groups the docs changes into one pull request per engineer whose server change triggered them, so the engineer who made the change can review it. As soon as that human review is complete and the pull request merges, the live docs site is rebuilt with the update.</p><p>This workflow has evolved significantly over time. It started life as a simple workflow running on a laptop and is now an Anthropic Managed Agent, running on a Mac mini that interacts with its human reviewers via Slack. Some revisions are about accuracy, such as the critic pass and reading each pull request diff three times independently before trusting it. We can't seem to get better than an 80% accuracy score (tested by planting defects) in a single session, which is why we run so many passes over a single change.</p><h2>How the agent learns</h2><p>The agent starts every run drawing on a set of instructions about VillageSQL that has been built up through the many iterations of the workflow. There are three kinds of files. A rules file says how things are done: what a docs change looks like, where a new feature's documentation goes, and how error text is captured. A lessons file holds an accumulated list of assumptions and mistakes past agents have made. A history file holds the story behind each lesson, so a rule that looks arbitrary can be understood. When a person corrects the agent, the correction goes into those files before the session ends so the next run learns from it.</p><h2>Guardrails on every page</h2><p>Instructions tell the agent what to do. Checks help keep it in line when it doesn't follow them. Like instructions, most of the checks we use come from trial and error.</p><p>Every docs page about extensions, the SDK, or the server declares its audience in its front matter: someone writing an extension, someone contributing to the server, or someone using the database. The audience field forces the question "who is this for?" at the moment of writing. This helps prevent the AI from conflating topics on a page. For example, the raw ABI is what the SDK connects to, and the SDK is what extension authors connect to, so a page for extension authors should not drift into the raw ABI. That distinction was confusing to agents initially.</p><p>Other checks read the code blocks on a page and compare them against a list of banned phrases and past mistakes. Documented table columns and system variables are checked against a running server. A code block that names a variable the server doesn't have, or uses a MySQL concept that isn't relevant to VillageSQL-specific documentation, gets flagged by name. Checks also look for AI-slop language we want to keep out of the docs, such as transition language ("the server now calls", "previously the extension"), because a reference page describes the current state, and a reader who never saw the old page does not need the correction.</p><p>The strictest rule is about error text. Any error message, error number, or warning on a page must be pasted verbatim from a running server. An agent never types one from memory or infers one from the source code. </p><h2>Some interesting lessons we have taught the agent</h2><p>Below are some examples of lessons we have had to "teach" the agent.</p><ul><li><strong>A stale server produces confident, wrong findings.</strong> Before anything is verified, we compare when the server was built with when the code last changed. A result from a stale build is reproducible and often false, which is the worst kind of wrong.</li><li><strong>AI adds. On its own, it rarely removes.</strong> When an agent finds a wrong sentence, its instinct is to add a note beside it rather than fix it. We make it change the sentence, and every review pass on our docs has asked for removals.</li><li><strong>A fix in one copy is a bug in the others.</strong> We maintain docs for multiple versions in multiple languages. A fix that lands in just one copy and misses the rest leaves the docs disagreeing with themselves. A rule is that a fix goes to every copy.</li></ul><h2>The workflow is a means to an end</h2><p>An open process that continuously improves the docs is the goal, so users have the latest as soon as code ships. Our current workflow is just a means to an end. AI tooling is improving fast enough that the best way to achieve our goals a year from now probably doesn't exist yet. We'll keep the rules, the lessons, and the checks, because those are ours, and we'll swap the tooling under them whenever something better comes along. If you'd like to contribute to the VillageSQL documentation, you can do so in the <a href="https://github.com/villagesql/villagesql-docs" rel="noreferrer">public repo.</a></p>]]></content:encoded>
    <pubDate>Tue, 17 Nov 2026 14:00:00 +0000</pubDate>
    <dc:creator>VillageSQL</dc:creator>
    <category>MySQL</category>
    <category>VillageSQL</category>
    <category>AI</category>
    <category>Open Source</category>
  </item>

  <item>
    <title>MySQL Development as it Happens, Part 2 – Vector Indexes</title>
    <guid isPermaLink="false">a9f92fd58cf09c75033546e4bf67d4f9</guid>
    <link>https://blogs.oracle.com/mysql/mysql-development-as-it-happens-part-2-vector-indexes</link>
    <description>In my previous blog post on what’s going on in MySQL Server development, I gave an overview of how the MySQL development process has been changing this year.   Today, I want to dive deeper into one of the four strategic focus areas: AI &amp;amp; Cloud, with a closer look at VECTOR INDEX support.  Project News  Before […]</description>
    <content:encoded><![CDATA[In my previous blog post on what’s going on in MySQL Server development, I gave an overview of how the MySQL development process has been changing this year.   Today, I want to dive deeper into one of the four strategic focus areas: AI &amp; Cloud, with a closer look at VECTOR INDEX support.  Project News  Before […]]]></content:encoded>
    <pubDate>Mon, 16 Nov 2026 15:32:25 +0000</pubDate>
    <dc:creator>Oracle MySQL Group</dc:creator>
    <category>MySQL Community</category>
  </item>

  <item>
    <title>Share Our Booth at FOSDEM 2027: Call for MySQL-Compatible Projects</title>
    <guid isPermaLink="false">https://oursqlfoundation.org/blog/fosdem-2027-booth-call-for-projects/</guid>
    <link>https://oursqlfoundation.org/blog/fosdem-2027-booth-call-for-projects/</link>
    <description>FOSDEM is coming, and this year we want to bring some friends with us.
OurSQL Foundation is opening a call for open source MySQL-compatible projects to join us at FOSDEM 2027 in Brussels (January 30–31). If you build an open source project, alone or with a team, and you want more people to know about it, this is for you.

What you get
A place at our booth. We will share our stand with you. You can run a demo, show what you built, or just talk with your users and future users. FOSDEM brings several thousand developers from Europe and all over the world, so it’s a good crowd to meet.</description>
    <content:encoded><![CDATA[<p>FOSDEM is coming, and this year we want to bring some friends with us.</p>
<p>OurSQL Foundation is opening a call for open source MySQL-compatible projects to join us at FOSDEM 2027 in Brussels (January 30–31). If you build an open source project, alone or with a team, and you want more people to know about it, this is for you.</p>

<h2>What you get</h2>
<p><strong>A place at our booth.</strong> We will share our stand with you. You can run a demo, show what you built, or just talk with your users and future users. FOSDEM brings several thousand developers from Europe and all over the world, so it’s a good crowd to meet.</p>]]></content:encoded>
    <pubDate>Mon, 16 Nov 2026 00:00:00 +0000</pubDate>
    <dc:creator>OurSQL Foundation</dc:creator>
  </item>

  <item>
    <title>From Big Iron to AMD EPYC: Preparing TAF to Push MariaDB at Scale</title>
    <guid isPermaLink="false">https://mariadb.org/?p=43161</guid>
    <link>https://mariadb.org/from-big-iron-to-amd-epyc-preparing-taf-to-push-mariadb-at-scale/</link>
    <description>Some of the first large database servers I worked with were DEC VAX/VMS systems running Sybase. They were physically huge machines. Towers of quad 90.91 MHz processors, with more towers beside them holding mirrored RAID5 storage. At the time, this was big iron, taking food stamps in Texas to debit cards. A lot has changed […]
From Big Iron to AMD EPYC: Preparing TAF to Push MariaDB at Scale appeared first on MariaDB.org</description>
    <content:encoded><![CDATA[<p>Some of the first large database servers I worked with were DEC VAX/VMS systems running Sybase. They were physically huge machines. Towers of quad 90.91 MHz processors, with more towers beside them holding mirrored RAID5 storage. At the time, this was big iron, taking food stamps in Texas to debit cards. A lot has changed […]</p>
<p><a href="https://mariadb.org/from-big-iron-to-amd-epyc-preparing-taf-to-push-mariadb-at-scale/">From Big Iron to AMD EPYC: Preparing TAF to Push MariaDB at Scale</a> appeared first on <a href="https://mariadb.org/">MariaDB.org</a></p>]]></content:encoded>
    <pubDate>Sun, 15 Nov 2026 22:23:16 +0000</pubDate>
    <dc:creator>Jonathan Miller</dc:creator>
    <category>Generic</category>
    <category>MariaDB</category>
    <category>MariaDB Foundation</category>
    <category>mysql</category>
    <category>Performance</category>
  </item>

  <item>
    <title>Fix Amazon Aurora MySQL error 63955: Virtual index ID overflow</title>
    <guid isPermaLink="false">ff4ece1b2f9fa0664a6f75182835943d7d5f8700</guid>
    <link>https://aws.amazon.com/blogs/database/fix-amazon-aurora-mysql-error-63955-virtual-index-id-overflow/</link>
    <description>Your Amazon Aurora MySQL cluster looks healthy. Reads and writes succeed and every metric is green. Then a routine CREATE INDEX fails with error 63955, and from then on no index that includes a virtual column can be created on the cluster until you rebuild its databases. 
 
 Error Code: 63955. 
Index cannot be created because virtual index ID limit (4294967295) was reached. 
 
 
 What it means: the cluster has used up the internal IDs Aurora MySQL uses when it evaluates indexes that contain a virtual column. This error does not affect existing data or ordinary indexes, but new indexes on VIRTUAL generated columns, functional indexes, and multi-valued indexes fail. 
 What helps short term: converting the generated column in the failing index from VIRTUAL to STORED, so that the index no longer contains a virtual column. This does not reset the IDs, and every new index that contains a virtual column still fails. 
 What fixes it: a logical dump and restore into a new cluster, with binary log replication to keep the cutover short. 
 How to get warning next time: run Aurora MySQL 3.10.0 or later, and alarm on the VIRTUAL_INDEX_ID_OVERFLOW_WARNING message in the error log. 
 
Aurora MySQL assigns each index an internal identifier, the virtual index ID, to track index metadata for undo log management. When you create indexes through DDL operations, these operations consume the identifiers over the life of the cluster, and Aurora MySQL does not provide a way to reset them in place. Large, long-lived clusters with heavy data definition language (DDL) activity can reach the limit. Typical examples are multi-database clusters, schema-per-tenant software as a service (SaaS) platforms, and applications that continuously create, drop, and rebuild indexes. 
The impact is narrow but serious. Existing tables continue serving reads and writes, this error does not cause data loss or corruption, and you can still create indexes on ordinary columns. However, CREATE INDEX, ALTER TABLE ... ADD INDEX, and CREATE TABLE fail when the new index contains a virtual column. Table-rebuilding ALTER TABLE statements on tables that already have such an index fail too. Until you rebuild the database, schema migrations and releases that add these indexes are blocked, and so is onboarding new tenants in a schema-per-tenant design whose tables use them. 
In this post, we explain the limit and how to recognize error 63955. We then walk through the documented resolution, a logical dump and restore, including the binary log setup, user and grant migration, validation, cutover, and rollback limits. Finally, we set up the monitoring that turns the next occurrence into a scheduled project rather than an incident. 
Which engines and versions this applies to 
Error 63955 is specific to Amazon Aurora MySQL-Compatible Edition, referred to as Aurora MySQL throughout this post. The code does not exist in upstream MySQL, and the condition does not affect Amazon RDS for MySQL. The Virtual index ID overflow errors documentation says “RDS might throw error code 63955.” That wording appears because you manage Aurora clusters through the Amazon Relational Database Service (Amazon RDS) API and console. Any Aurora MySQL cluster with enough accumulated DDL can reach the limit. What differs by version is whether you get advance warning. 
Starting with Aurora MySQL version 3.10.0, the engine writes a warning to the MySQL error log as the limit approaches. Earlier versions give no advance warning, so the first sign is a failed CREATE INDEX. Aurora MySQL 3.10 is also a long-term support (LTS) release. For both reasons, this post restores into 3.10 or later rather than into the source’s version. 
To check your version, run the following statement, or look at the Engine version field on the cluster’s Configuration tab in the Amazon RDS console. 
 
 SELECT AURORA_VERSION(); 
 
For the authoritative description of this limit, refer to Virtual index ID overflow errors and the Aurora MySQL 3.10.0 release notes. 
The virtual index ID limit 
The documentation states that Aurora MySQL limits virtual index ID values to prevent an issue caused by the undo format in MySQL, and that a cluster with an index above the limit might not be available. When an index approaches the limit, or when you attempt to create one above it, Aurora MySQL returns warning or error code 63955 to the session that ran the statement. 
Three points prevent the most common misreadings. 
 
 Aurora MySQL checks the limit only for indexes that contain a virtual column. Every index that the cluster creates uses up an ID, but Aurora MySQL compares the ID with the limit only when the new index includes a virtual column. Indexes on ordinary columns keep working, even on a cluster past the limit. The next section lists the index types that count. 
 The number in the error message is a fixed ceiling, not a reading. The client error always prints 4294967295. The error-log warning is different: it includes the ID of the index being created, so subtracting that ID from 4294967295 shows how much headroom is left. In alerting, match the error-log tags shown in Setting up the error-log alarm, not the number. 
 The warning is the only place the current ID is reported. No RDS API, Amazon CloudWatch metric, or Amazon CloudWatch Database Insights view shows it, so the error-log warning is your early signal. 
 
Failing the DDL is the protective behavior. Aurora refuses the statement rather than create an index above the limit, so reads and writes on existing tables continue. Because Aurora MySQL does not provide a way to reset the identifier in place, the documented resolution is a rebuild rather than a configuration change. 
Which indexes count against the limit 
An index counts if it contains a virtual column, whether you define the column or MySQL creates one automatically. 
 
 A secondary index on a VIRTUAL generated column, for example c INT AS (a+b) VIRTUAL with KEY (c). A composite index counts if any one of its columns is virtual. 
 A functional index (MySQL 8.0.13 and later), such as KEY ((a+b)) or KEY ((LOWER(name))). MySQL builds these on a hidden virtual generated column, so they count even if you never declared a generated column. 
 A multi-valued index on a JSON array, such as KEY ((CAST(j-&amp;gt;'$.tags' AS UNSIGNED ARRAY))). These also use a hidden virtual column. 
 
Indexes on ordinary columns and on STORED generated columns still use up IDs, but Aurora MySQL does not check them against the limit. 
Recognizing error 63955 
Any statement that creates an index containing a virtual column fails. A functional index on a small table is enough to demonstrate it. 
 
 CREATE INDEX idx_email_lower ON app_db.customer_profile ((LOWER(email)));
Error Code: 63955. Index cannot be created because virtual index ID limit (4294967295) was reached. 
 
The two signals surface in different places, and that decides what you can monitor. 
 
 Error 63955 is returned to the client session that ran the statement. You see it in your SQL client, your application or schema-migration tool logs, or your deployment pipeline. 
 The error log records a warning tagged VIRTUAL_INDEX_ID_OVERFLOW_WARNING (MY-014078) on Aurora MySQL 3.10.0 and higher, as the limit approaches. The log line does not contain 63955, but it does include the ID of the index being created. The session that ran the DDL also receives warning 63955, but the statement succeeds, and most applications and schema-migration tools do not surface SQL warnings. A cluster can therefore write the warning for weeks without anyone noticing. 
 
In the error log, the warning and the error look like the following. The ID values are illustrative. 
 
 [Warning] [MY-014078] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_WARNING] Index `idx_email_lower` in table `app_db/customer_profile` has id 4000000123 and is approaching virtual index id limit of 4294967295.
[ERROR] [MY-014079] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_ERROR] Index `idx_email_lower` in table `app_db/customer_profile` cannot be created because index id 4294967296 is above virtual index id limit of 4294967295. 
 
To confirm the diagnosis without touching a production table, create a functional index like the example above on a small, existing, non-critical table, and then drop it. An index on an ordinary column is not a valid test, because it succeeds even on a cluster past the limit. Use an existing table rather than creating one, because CREATE TABLE with an index that contains a virtual column can fail the same way. For retrieving the error log, refer to Monitoring Amazon Aurora log files. 
Converting to STORED: A short-term workaround 
Because the limit applies only to indexes that contain a virtual column, converting the generated column in a failing index from VIRTUAL to STORED and rebuilding the index takes that index out of the check. In one case, the customer could create new indexes again after this change. It is a workaround, not a fix: the conversion does not reset the IDs, and any new index that contains a virtual column, including a functional or multi-valued index, still fails. Plan the logical rebuild anyway. 
Weigh these trade-offs before you use it. 
 
 The conversion rebuilds the table. Changing a column from VIRTUAL to STORED copies the table, which takes time on large tables, and the stored values add storage. The rebuild fails with error 63955 if the table has another index that still contains a virtual column. 
 STORED is only valid on generated columns. Applying it to an ordinary column returns Error Code: 1064, a syntax error. 
 
Solution overview 
The recommended resolution in the AWS documentation is to recreate your database with a logical dump and restore. A logical rebuild recreates the schema and reloads the data, rather than copying the existing physical structures. Snapshot restore, Aurora cloning, point-in-time recovery, and Blue/Green deployments all produce a physical copy of the volume, and none of them is a documented way to clear this condition. 
The procedure has these stages: 
 
 Turn on binary logging on the source cluster. 
 Clone the source and record the binary log position of the clone. 
 Dump the clone with MyDumper and restore into a new Aurora MySQL 3.10 cluster with MyLoader. 
 Migrate users and grants. 
 Validate the new cluster. 
 Replicate from the source to the new cluster, starting at the recorded position. 
 Cut over by stopping traffic, letting replication catch up, and switching endpoints. 
 
The documentation does not mandate a new cluster as the target. Reloading in place on the same cluster is possible, but it takes the affected databases offline for the whole reload and removes the rollback path. This post uses a new cluster. 
Prerequisites 
 
 AWS Identity and Access Management (IAM) permissions to create and modify Aurora DB clusters, instances, and cluster parameter groups (rds:CreateDBCluster, rds:CreateDBInstance, rds:RestoreDBClusterToPointInTime for the clone, rds:ModifyDBCluster, rds:ModifyDBClusterParameterGroup, rds:RebootDBInstance, rds:DescribeEvents), plus logs:PutMetricFilter, logs:PutRetentionPolicy, and cloudwatch:PutMetricAlarm for the alarm. 
 A custom DB cluster parameter group on the source cluster. You cannot change binlog_format in the default group. 
 An Amazon Elastic Compute Cloud (Amazon EC2) host in the same Region and virtual private cloud (VPC) as the cluster, with MyDumper and MyLoader installed. This post was tested with MyDumper and MyLoader v1.0.11-1 against Aurora MySQL 3.10.3. Pin the version you test with. 
 Amazon Elastic Block Store (Amazon EBS) for the dump. Dump files are commonly about a third of the size that information_schema reports, but size the volume with headroom, because the ratio depends on your data. 
 A maintenance window for one reboot of the source writer, described in the next section. 
 
Step 1: Turn on binary logging on the source 
Replication from the source to the new cluster needs binary logging on the source. On Aurora MySQL, binlog_format defaults to OFF, and changing it from OFF requires a reboot of the cluster. On many clusters, this reboot is the first planned downtime of the migration, so schedule it like one. If binlog_format is already set on your source, skip to the retention step. 
 
 aws rds modify-db-cluster-parameter-group \
--db-cluster-parameter-group-name my-cluster-params \
--parameters &quot;ParameterName=binlog_format,ParameterValue=MIXED,ApplyMethod=pending-reboot&quot;
aws rds reboot-db-instance --db-instance-identifier my-cluster-writer 
 
Then connect to the source writer and retain binary logs for longer than the whole dump, restore, validation, and replication window. Aurora MySQL accepts up to 2,160 hours (90 days). 
 
 CALL mysql.rds_set_configuration('binlog retention hours', 168);
CALL mysql.rds_show_configuration; 
 
If the retention period ends before replication starts, the source deletes binary logs the new cluster still needs, and you must start the migration again. For the procedure, refer to Setting up binary log replication for Aurora MySQL. 
Step 2: Clone the source and record the binary log position 
Dump from an Aurora clone rather than from the source. No production workload competes with the dump, no writer-side DDL can terminate its connections, and nothing writes to the clone, so every MyDumper thread sees the same data. Create the clone after binary logging is on. For the procedure, refer to Cloning a volume for an Amazon Aurora DB cluster. 
Replication must start at the source’s binary log position at the moment the clone was created. Do not use a position that MyDumper records from the clone, which describes the clone’s own binary log and would make replication skip or repeat writes. Aurora records the right position as an event on the clone’s writer instance. 
 
 aws rds describe-events \
--source-type db-instance \
--source-identifier my-cluster-clone-writer \
--duration 1440 \
--query &quot;Events[?contains(Message, 'Binlog position')].[Date,Message]&quot; \
--output table 
 
The output contains a line like the following. Save the file name and position. 
 
 Binlog position from crash recovery is mysql-bin-changelog.000003 4278 
 
The same line also appears in the clone writer’s MySQL error log. 
Step 3: Dump the clone and restore into a new cluster 
Create the new cluster on the current Aurora MySQL 3.10 LTS release or later. Binary log replication supports a target that is the same version as the source or newer, and the rebuild is the cheapest moment to get the version that writes the advance warning. 
Before the dump, set max_execution_time to 0 in the clone’s parameter group. MyDumper SELECT statements on large tables can exceed a non-zero value, and the dump fails with Query execution was interrupted, maximum statement execution time exceeded. The parameter is dynamic. If the dump drops with Lost connection to MySQL server during query, check whether net_read_timeout, net_write_timeout, or wait_timeout have been lowered in your parameter groups, and raise them for the migration window. 
Store the credentials in a defaults file rather than on the command line, so that the password does not appear in process listings. 
 
 cat &amp;gt; ~/.mydumper.cnf &amp;lt;&amp;lt;'EOF'
[mydumper]
user=admin
password=&amp;lt;password&amp;gt;
[myloader]
user=admin
password=&amp;lt;password&amp;gt;
EOF
chmod 600 ~/.mydumper.cnf 
 
Dump all user databases from the clone’s cluster endpoint, excluding the system schemas. 
 
 mydumper \
--defaults-file=$HOME/.mydumper.cnf \
--host=&amp;lt;clone-cluster-endpoint&amp;gt; \
--outputdir=/mnt/full-cluster-dump \
--threads=32 \
--rows=500000 \
--compress \
--events \
--routines \
--triggers \
--sync-thread-lock-mode=SAFE_NO_LOCK \
--regex '^(?!(mysql|sys|information_schema|performance_schema)(\.|$))' \
--verbose=3 
 
MyDumper detects Aurora and refuses to run in its default lock mode, so you must set --sync-thread-lock-mode. Without it, the dump stops with this message: 
 
 We support LOCK_ALL and SAFE_NO_LOCK modes for RDS/Aurora. Select one of them to configure --sync-thread-lock-mode 
 
SAFE_NO_LOCK takes no global lock, and it fails the dump if the worker threads do not start at the same point in time, so a dump that completes is consistent. The --regex filter is required, not just tidy. MyDumper applies the pattern to database.table for tables and to the bare schema name for schema-level objects, such as each schema’s CREATE DATABASE statement. The (\.|$) part therefore excludes a system schema name only when a dot or the end of the name follows it. Without it, the filter also skips customer schemas whose names start with the same words, such as sys_audit or mysql_app, and the dump leaves them out without a warning. Without the filter at all, MyDumper reaches the non-transactional table mysql.general_log_backup and stops the consistent dump. A successful run logs lines like these: 
 
 Using --trx-tables options, binlog coordinates will not be accurate if you are writing to non transactional tables.
Executing in SAFE_NO_LOCK mode. This backup will fail if all threads are not in the same point in time, which ensures consistency
Transactions started, unlocking tables
Finished dump at: 2026-11-10 15:07:21 
 
The log also reports @@tokudb_version not found - ERROR 1193. MyDumper checks for the TokuDB engine, which Aurora does not have, so you can ignore this line. The warning about --trx-tables and binlog coordinates does not affect this procedure, because replication starts from the position in the clone’s event in Step 2, not from the position MyDumper records. 
Then restore into the new cluster. 
 
 myloader \
--defaults-file=$HOME/.mydumper.cnf \
--host=&amp;lt;new-cluster-writer-endpoint&amp;gt; \
--directory=/mnt/full-cluster-dump \
--threads=32 \
--verbose=3 
 
The new cluster is empty, so MyLoader needs no option to drop or overwrite existing tables. At the end of the restore, MyLoader compares checksums of each table’s structure and indexes, the views, and the events against values recorded in the dump, and logs a line such as Structure checksum confirmed for shop.orders for each one, followed by Restore completed. These checks cover structure, not row data, so still run the validation in Step 5. 
In one migration, a database that information_schema reported at about 400 GB produced a dump of about 115 GB. The reported size includes secondary index storage, which the dump carries only as DDL, plus InnoDB page overhead and fragmentation. Treat the ratio as an illustration, not a planning figure. 
Step 4: Migrate users and grants 
This is the step most likely to break a cutover, because it surfaces only when the application points at the new cluster and nothing can authenticate. MyDumper and MyLoader do not migrate accounts, which live in the mysql system schema. 
On Aurora MySQL 3, which is compatible with MySQL 8.0, GRANT no longer creates a user, and SHOW GRANTS does not include the password. Replaying grants alone therefore fails against accounts that do not exist on the new cluster. Generate SHOW CREATE USER statements first, then SHOW GRANTS, and replay them in that order. 
Run the following on the clone, not on the source. The clone matches the binary log position you recorded in Step 2. Any account created or changed on the source after the clone was taken reaches the new cluster through replication in Step 6, instead of appearing in both the script and the binary log. The query skips the RDS-reserved accounts, the AWS-managed roles, and the primary (admin) user, which already exists on the new cluster. 
 
 SET SESSION print_identified_with_as_hex = ON;
SELECT CONCAT('SHOW CREATE USER ', QUOTE(user), '@', QUOTE(host), ';')
FROM mysql.user
WHERE user NOT IN ('rdsadmin', 'rdsrepladmin', 'rdsproxyadmin', 'admin',
'mysql.sys', 'mysql.session', 'mysql.infoschema',
'rds_superuser_role')
AND user NOT LIKE 'AWS\_%'
UNION ALL
SELECT CONCAT('SHOW GRANTS FOR ', QUOTE(user), '@', QUOTE(host), ';')
FROM mysql.user
WHERE user NOT IN ('rdsadmin', 'rdsrepladmin', 'rdsproxyadmin', 'admin',
'mysql.sys', 'mysql.session', 'mysql.infoschema',
'rds_superuser_role')
AND user NOT LIKE 'AWS\_%'; 
 
Replace admin with your primary user name. print_identified_with_as_hex makes the password hashes print as hexadecimal, because caching_sha2_password hashes can contain characters that do not survive a copy and paste. 
Run the generated statements, and save their output as a script with every CREATE USER statement before any GRANT. User-defined roles are accounts in mysql.user, so they come through the same way, as locked accounts with no password. The CREATE USER output for a user also carries that user’s DEFAULT ROLE, and the role it names must already exist when the statement runs. Move the CREATE USER statements for roles (the ones that end in ACCOUNT LOCK) to the top of the script. Replay the script on the new cluster after the data has loaded, because table-level and routine-level grants fail against objects that do not exist yet. Because the hashes transfer unchanged, application passwords stay the same. 
On Aurora MySQL version 2, which is compatible with MySQL 5.7, the same two-statement approach works, without the print_identified_with_as_hex setting. 
For a role app_read and a user app_ro that has the role as its default, the replay script looks like this (hashes shortened): 
 
 CREATE USER `app_read`@`%` IDENTIFIED WITH 'mysql_native_password' REQUIRE NONE PASSWORD EXPIRE ACCOUNT LOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT;
CREATE USER `app_ro`@`%` IDENTIFIED WITH 'mysql_native_password' AS '&amp;lt;hash&amp;gt;' DEFAULT ROLE `app_read`@`%` REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT;
GRANT USAGE ON *.* TO `app_read`@`%`;
GRANT SELECT ON `shop`.* TO `app_read`@`%`;
GRANT USAGE ON *.* TO `app_ro`@`%`;
GRANT `app_read`@`%` TO `app_ro`@`%`; 
 
After the replay, log in to the new cluster as one or two application accounts and run a query they depend on. That confirms both the password hash and the grants before you cut over. 
Step 5: Validate the new cluster 
Validate more than row counts. COUNT(*) misses lost routines, triggers, events, views, and secondary indexes. Run the following on the clone and on the new cluster, and compare the output. 
 
 SELECT s.schema_name,
(SELECT COUNT(*) FROM information_schema.tables t
WHERE t.table_schema = s.schema_name AND t.table_type = 'BASE TABLE') AS tables_,
(SELECT COUNT(*) FROM information_schema.views v
WHERE v.table_schema = s.schema_name) AS views_,
(SELECT COUNT(DISTINCT st.table_name, st.index_name) FROM information_schema.statistics st
WHERE st.table_schema = s.schema_name) AS indexes_,
(SELECT COUNT(*) FROM information_schema.routines r
WHERE r.routine_schema = s.schema_name) AS routines_,
(SELECT COUNT(*) FROM information_schema.triggers tr
WHERE tr.trigger_schema = s.schema_name) AS triggers_,
(SELECT COUNT(*) FROM information_schema.events e
WHERE e.event_schema = s.schema_name) AS events_
FROM information_schema.schemata s
WHERE s.schema_name NOT IN ('mysql', 'sys', 'information_schema', 'performance_schema')
ORDER BY s.schema_name; 
 
Then compare exact SELECT COUNT(*) results on your largest tables. For critical tables, CHECKSUM TABLE on both sides gives a stronger check, but it reads every row, so run it on the clone rather than on the source. 
Finally, run the statement that failed on the old cluster. It should now succeed. 
 
 CREATE INDEX idx_email_lower ON app_db.customer_profile ((LOWER(email))); 
 
Step 6: Replicate and cut over 
Start replication on the new cluster from the source, using the binary log file name and position you recorded in Step 2. For the procedures, refer to Setting up binary log replication for Aurora MySQL, which covers mysql.rds_set_external_source and mysql.rds_start_replication. 
If replication stops on an account statement, such as error 1396 (Operation CREATE USER failed), the replay script and the binary log contain the same account change. Check that you generated the script on the clone, as described in Step 4. If the account already matches the source, skip the duplicate statement with mysql.rds_skip_repl_error and replication continues. 
When replication has caught up, cut over: 
 
 Stop application traffic to the source cluster. 
 Confirm that replication lag on the new cluster is zero, then stop replication. 
 Switch endpoints, using one of the options in the next section. 
 Resume traffic. 
 
The dump and restore happen before this window, so the application is offline only for the switchover. Test the sequence in a non-production environment before you schedule it. 
Keeping your application endpoints unchanged 
 
 Rename the cluster identifiers. Rename the old cluster, for example from my-cluster to my-cluster-old, then rename the new cluster to the original name. The cluster endpoints follow the new identifier. Instance endpoints and custom endpoints do not, so rename those separately if your applications use them. Clients also keep the old DNS response until its time to live (TTL) expires, so allow for that in the cutover window. Refer to Modifying an Amazon Aurora DB cluster. 
 Use an Amazon Route 53 CNAME. If applications connect through your own DNS name, update the CNAME at cutover. The same TTL consideration applies. 
 Use Amazon RDS Proxy. Retarget the proxy to the new cluster without changing connection strings. Refer to Using Amazon RDS Proxy. 
 
Considerations and limitations 
 
 The binary log reboot. Turning on binary logging on the source requires a reboot, as described in Step 1. 
 Rollback only lasts until the first schema change. For a rollback path, configure reverse replication from the new cluster to the old one before cutover. This requires binary logging on the new cluster as well, which you can turn on before it takes traffic. The old cluster is still at the identifier limit, so any index DDL that runs on the new cluster after cutover replicates back, fails with 63955, and stops reverse replication. Freeze index DDL until you have decided not to roll back, or accept that rollback is valid only until the first schema change. 
 Dump the clone, not the live cluster. Replication starts from the clone’s binary log position in Step 2, so the dump must come from the same clone. A dump of the live cluster would not line up with that position. 
 In-place reload. Reloading on the same cluster avoids a second cluster but takes the databases offline for the whole reload and has no rollback. 
 
Cost considerations 
This migration runs paid resources in parallel. Plan for the following. 
 
 Two Aurora clusters. The source and the new cluster both bill for instance hours and storage during migration and validation. 
 An Aurora clone. The clone bills for its DB instances. Its storage cost starts small and grows as the clone and source diverge. 
 An Amazon EC2 dump host and the Amazon EBS volume that holds the dump files. 
 Binary log retention on the source, which adds storage for the retained logs. 
 
Keep the validation window as short as your testing allows, and clean up as soon as validation passes. 
Clean up 
Most of the clean-up is ordinary deletion. Terminate the Amazon EC2 dump host, delete the Amazon EBS volume that held the dump (it is not deleted with the instance unless its DeleteOnTermination attribute is set), and delete the clone’s instances and then the clone cluster. For the procedure, refer to Deleting Aurora DB clusters and DB instances. 
Three steps are specific to this migration. 
Stop replication. On the new cluster, and on the old one if you configured reverse replication, run the following. The second procedure has a different name on Aurora MySQL version 2. Refer to Configuring, starting, and stopping binary log replication. 
 
 CALL mysql.rds_stop_replication;
CALL mysql.rds_reset_external_source; 
 
Clear binary log retention on any cluster you keep that no longer replicates, so that it stops retaining log files. The value 0 is not accepted. 
 
 CALL mysql.rds_set_configuration('binlog retention hours', NULL); 
 
Delete the old cluster last, with a final snapshot. Do this only after the new cluster has served production traffic normally and you no longer need the rollback path. The old cluster holds the only copy of your pre-migration data. Turn off deletion protection, delete its instances, then delete the cluster with a final snapshot. 
 
 aws rds modify-db-cluster --db-cluster-identifier my-cluster-old \
--no-deletion-protection --apply-immediately
aws rds delete-db-cluster --db-cluster-identifier my-cluster-old \
--final-db-snapshot-identifier my-cluster-old-final \
--no-delete-automated-backups 
 
Prevention 
 
 Alarm on the virtual index ID overflow messages in the error log. This is your only advance signal. The next section walks through the setup. 
 Reduce unnecessary DDL churn. Avoid repeatedly creating and dropping indexes, and treat table-rebuilding ALTER TABLE as drawing down a finite budget. 
 Limit indexes that contain virtual columns on clusters with heavy DDL. Aurora MySQL does not check indexes on ordinary columns or STORED generated columns against the limit. 
 Plan a periodic logical rebuild for databases with very heavy DDL activity, so that the work happens on your schedule. 
 Stay on Aurora MySQL 3.10.0 or later. Earlier versions give no advance warning. 
 
Setting up the error-log alarm 
Turn on error-log export to Amazon CloudWatch Logs for the cluster. 
 
 aws rds modify-db-cluster \
--db-cluster-identifier my-cluster \
--cloudwatch-logs-export-configuration '{&quot;EnableLogTypes&quot;:[&quot;error&quot;]}' 
 
Aurora creates the log group /aws/rds/cluster/my-cluster/error with no expiry, so set a retention period. 
 
 aws logs put-retention-policy \
--log-group-name /aws/rds/cluster/my-cluster/error \
--retention-in-days 90 
 
Create a metric filter that matches the warning. The pattern matches the two tags that Aurora MySQL writes to the error log: VIRTUAL_INDEX_ID_OVERFLOW_WARNING as the limit approaches, and VIRTUAL_INDEX_ID_OVERFLOW_ERROR when an index is refused. Match the tags rather than the message text. Filter patterns are case sensitive, and the error-log message spells id in lowercase, unlike the client error. The bare number 63955 is also a poor match, because it can appear in log sequence numbers, thread IDs and byte counts. 
 
 aws logs put-metric-filter \
--log-group-name /aws/rds/cluster/my-cluster/error \
--filter-name aurora-virtual-index-id-warning \
--filter-pattern '?&quot;VIRTUAL_INDEX_ID_OVERFLOW_WARNING&quot; ?&quot;VIRTUAL_INDEX_ID_OVERFLOW_ERROR&quot;' \
--metric-transformations \
metricName=VirtualIndexIdWarning,metricNamespace=Custom/AuroraMySQL,metricValue=1,defaultValue=0 
 
Test the pattern before relying on it. A healthy cluster has no warning line to test with, so the following command uses a sample line in the format that Aurora MySQL writes. The output should show one match. 
 
 aws logs test-metric-filter \
--filter-pattern '?&quot;VIRTUAL_INDEX_ID_OVERFLOW_WARNING&quot; ?&quot;VIRTUAL_INDEX_ID_OVERFLOW_ERROR&quot;' \
--log-event-messages '[Warning] [MY-014078] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_WARNING] Index `idx_b` in table `app_db/orders` has id 4000000123 and is approaching virtual index id limit of 4294967295.' 
 
Then create the alarm. Aurora writes the error log to disk every five minutes and rotates it hourly, so a one-hour period is appropriate. 
 
 aws cloudwatch put-metric-alarm \
--alarm-name aurora-virtual-index-id-warning-my-cluster \
--alarm-description &quot;Aurora MySQL virtual index ID overflow warning or error&quot; \
--namespace Custom/AuroraMySQL \
--metric-name VirtualIndexIdWarning \
--statistic Sum \
--period 3600 \
--evaluation-periods 1 \
--threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:us-east-1:111122223333:my-dba-topic 
 
--treat-missing-data notBreaching matters because a healthy cluster can go hours without writing to the error log. Metrics extracted from log events are billed as custom metrics. 
Covering every cluster 
A metric filter applies to one log group, and each cluster has its own. The following loop sets up export, retention, the filter, and the alarm for every Aurora MySQL cluster in the Region. Log export needs no parameter group change, so the loop also works on clusters that use the default parameter group. 
 
 FILTER='?&quot;VIRTUAL_INDEX_ID_OVERFLOW_WARNING&quot; ?&quot;VIRTUAL_INDEX_ID_OVERFLOW_ERROR&quot;'
TOPIC=arn:aws:sns:us-east-1:111122223333:my-dba-topic
for c in $(aws rds describe-db-clusters \
--query &quot;DBClusters[?Engine=='aurora-mysql'].DBClusterIdentifier&quot; --output text); do
aws rds modify-db-cluster --db-cluster-identifier &quot;$c&quot; \
--cloudwatch-logs-export-configuration '{&quot;EnableLogTypes&quot;:[&quot;error&quot;]}' &amp;gt;/dev/null
lg=&quot;/aws/rds/cluster/$c/error&quot;
until aws logs describe-log-groups --log-group-name-prefix &quot;$lg&quot; \
--query 'logGroups[0].logGroupName' --output text | grep -q &quot;$lg&quot;; do sleep 30; done
aws logs put-retention-policy --log-group-name &quot;$lg&quot; --retention-in-days 90
aws logs put-metric-filter --log-group-name &quot;$lg&quot; \
--filter-name aurora-virtual-index-id-warning --filter-pattern &quot;$FILTER&quot; \
--metric-transformations &quot;metricName=VirtualIndexIdWarning-$c,metricNamespace=Custom/AuroraMySQL,metricValue=1,defaultValue=0&quot;
aws cloudwatch put-metric-alarm --alarm-name &quot;aurora-virtual-index-id-warning-$c&quot; \
--namespace Custom/AuroraMySQL --metric-name &quot;VirtualIndexIdWarning-$c&quot; \
--statistic Sum --period 3600 --evaluation-periods 1 --threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreaching --alarm-actions &quot;$TOPIC&quot;
done 
 
The metric name includes the cluster identifier, so each alarm tracks one cluster. For an infrastructure-as-code version, the metric filter and alarm map to the AWS::Logs::MetricFilter and AWS::CloudWatch::Alarm AWS CloudFormation types. 
For reference, refer to Publishing Amazon Aurora MySQL logs to Amazon CloudWatch Logs and Creating metrics from log events using filters. 
Conclusion 
Error 63955 means the cluster has used up the virtual index IDs Aurora MySQL uses when it evaluates indexes that contain a virtual column, including functional and multi-valued indexes. Converting a generated column to STORED is a short-term workaround for one index, no physical copy clears the condition, and the documented resolution is a logical dump and restore into a new cluster. 
Plan four things carefully: the reboot to turn on binary logging, the binary log position of the clone, users and grants as their own step, and the rollback limit after the first schema change. Restore into Aurora MySQL 3.10 or later, and alarm on the VIRTUAL_INDEX_ID_OVERFLOW_WARNING message in the error log so the next cluster gives you notice. 
If you are unsure which restore path fits your cluster and engine version, raise a case with AWS Support before you commit to a migration window. 
For more detail, refer to Virtual index ID overflow errors, the Aurora MySQL 3.10.0 release notes, and Migrate very large databases to Amazon Aurora MySQL using MyDumper and MyLoader. 
  
 
About the author 
 
  
  
   
   
  Kartik Saini 
  Kartik is a Cloud Support Engineer on the Amazon RDS and Amazon Aurora team at AWS. He specializes in troubleshooting MySQL-compatible database engines, and works with customers to diagnose and resolve complex Aurora MySQL issues. 
  
</description>
    <content:encoded><![CDATA[<p>Your Amazon Aurora MySQL cluster looks healthy. Reads and writes succeed and every metric is green. Then a routine <code>CREATE INDEX</code> fails with error 63955, and from then on no index that includes a virtual column can be created on the cluster until you rebuild its databases.</p> 
<div class="hide-language"> 
 <pre><code class="language-text">Error Code: 63955. 
Index cannot be created because virtual index ID limit (4294967295) was reached.</code></pre> 
</div> 
<ul> 
 <li><strong>What it means:</strong> the cluster has used up the internal IDs Aurora MySQL uses when it evaluates indexes that contain a virtual column. This error does not affect existing data or ordinary indexes, but new indexes on <code>VIRTUAL</code> generated columns, functional indexes, and multi-valued indexes fail.</li> 
 <li><strong>What helps short term:</strong> converting the generated column in the failing index from <code>VIRTUAL</code> to <code>STORED</code>, so that the index no longer contains a virtual column. This does not reset the IDs, and every new index that contains a virtual column still fails.</li> 
 <li><strong>What fixes it:</strong> a logical dump and restore into a new cluster, with binary log replication to keep the cutover short.</li> 
 <li><strong>How to get warning next time:</strong> run Aurora MySQL 3.10.0 or later, and alarm on the <code>VIRTUAL_INDEX_ID_OVERFLOW_WARNING</code> message in the error log.</li> 
</ul> 
<p>Aurora MySQL assigns each index an internal identifier, the virtual index ID, to track index metadata for undo log management. When you create indexes through DDL operations, these operations consume the identifiers over the life of the cluster, and Aurora MySQL does not provide a way to reset them in place. Large, long-lived clusters with heavy data definition language (DDL) activity can reach the limit. Typical examples are multi-database clusters, schema-per-tenant software as a service (SaaS) platforms, and applications that continuously create, drop, and rebuild indexes.</p> 
<p>The impact is narrow but serious. Existing tables continue serving reads and writes, this error does not cause data loss or corruption, and you can still create indexes on ordinary columns. However, <code>CREATE INDEX</code>, <code>ALTER TABLE ... ADD INDEX</code>, and <code>CREATE TABLE</code> fail when the new index contains a virtual column. Table-rebuilding <code>ALTER TABLE</code> statements on tables that already have such an index fail too. Until you rebuild the database, schema migrations and releases that add these indexes are blocked, and so is onboarding new tenants in a schema-per-tenant design whose tables use them.</p> 
<p>In this post, we explain the limit and how to recognize error 63955. We then walk through the documented resolution, a logical dump and restore, including the binary log setup, user and grant migration, validation, cutover, and rollback limits. Finally, we set up the monitoring that turns the next occurrence into a scheduled project rather than an incident.</p> 
<h2>Which engines and versions this applies to</h2> 
<p>Error 63955 is specific to Amazon Aurora MySQL-Compatible Edition, referred to as Aurora MySQL throughout this post. The code does not exist in upstream MySQL, and the condition does not affect Amazon RDS for MySQL. The <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.BestPractices.Performance.html#AuroraMySQL.BestPractices.Performance.VirtualIndexIDOverflow" target="_blank" rel="noopener">Virtual index ID overflow errors</a> documentation says “RDS might throw error code 63955.” That wording appears because you manage Aurora clusters through the Amazon Relational Database Service (Amazon RDS) API and console. Any Aurora MySQL cluster with enough accumulated DDL can reach the limit. What differs by version is whether you get advance warning.</p> 
<p>Starting with Aurora MySQL version 3.10.0, the engine writes a warning to the MySQL error log as the limit approaches. Earlier versions give no advance warning, so the first sign is a failed <code>CREATE INDEX</code>. Aurora MySQL 3.10 is also a long-term support (LTS) release. For both reasons, this post restores into 3.10 or later rather than into the source’s version.</p> 
<p>To check your version, run the following statement, or look at the <strong>Engine version</strong> field on the cluster’s <strong>Configuration</strong> tab in the Amazon RDS console.</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SELECT AURORA_VERSION();</code></pre> 
</div> 
<p>For the authoritative description of this limit, refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.BestPractices.Performance.html#AuroraMySQL.BestPractices.Performance.VirtualIndexIDOverflow" target="_blank" rel="noopener">Virtual index ID overflow errors</a> and the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraMySQLReleaseNotes/AuroraMySQL.Updates.3100.html" target="_blank" rel="noopener">Aurora MySQL 3.10.0 release notes</a>.</p> 
<h2>The virtual index ID limit</h2> 
<p>The documentation states that Aurora MySQL limits virtual index ID values to prevent an issue caused by the undo format in MySQL, and that a cluster with an index above the limit might not be available. When an index approaches the limit, or when you attempt to create one above it, Aurora MySQL returns warning or error code 63955 to the session that ran the statement.</p> 
<p>Three points prevent the most common misreadings.</p> 
<ul> 
 <li><strong>Aurora MySQL checks the limit only for indexes that contain a virtual column.</strong> Every index that the cluster creates uses up an ID, but Aurora MySQL compares the ID with the limit only when the new index includes a virtual column. Indexes on ordinary columns keep working, even on a cluster past the limit. The next section lists the index types that count.</li> 
 <li><strong>The number in the error message is a fixed ceiling, not a reading.</strong> The client error always prints 4294967295. The error-log warning is different: it includes the ID of the index being created, so subtracting that ID from 4294967295 shows how much headroom is left. In alerting, match the error-log tags shown in Setting up the error-log alarm, not the number.</li> 
 <li><strong>The warning is the only place the current ID is reported.</strong> No RDS API, Amazon CloudWatch metric, or Amazon CloudWatch Database Insights view shows it, so the error-log warning is your early signal.</li> 
</ul> 
<p>Failing the DDL is the protective behavior. Aurora refuses the statement rather than create an index above the limit, so reads and writes on existing tables continue. Because Aurora MySQL does not provide a way to reset the identifier in place, the documented resolution is a rebuild rather than a configuration change.</p> 
<h3>Which indexes count against the limit</h3> 
<p>An index counts if it contains a virtual column, whether you define the column or MySQL creates one automatically.</p> 
<ul> 
 <li><strong>A secondary index on a</strong> <code>VIRTUAL</code> <strong>generated column,</strong> for example <code>c INT AS (a+b) VIRTUAL</code> with <code>KEY (c)</code>. A composite index counts if any one of its columns is virtual.</li> 
 <li><strong>A functional index</strong> (MySQL 8.0.13 and later), such as <code>KEY ((a+b))</code> or <code>KEY ((LOWER(name)))</code>. MySQL builds these on a hidden virtual generated column, so they count even if you never declared a generated column.</li> 
 <li><strong>A multi-valued index on a JSON array,</strong> such as <code>KEY ((CAST(j-&gt;'$.tags' AS UNSIGNED ARRAY)))</code>. These also use a hidden virtual column.</li> 
</ul> 
<p>Indexes on ordinary columns and on <code>STORED</code> generated columns still use up IDs, but Aurora MySQL does not check them against the limit.</p> 
<h2>Recognizing error 63955</h2> 
<p>Any statement that creates an index containing a virtual column fails. A functional index on a small table is enough to demonstrate it.</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">CREATE INDEX idx_email_lower ON app_db.customer_profile ((LOWER(email)));
Error Code: 63955. Index cannot be created because virtual index ID limit (4294967295) was reached.</code></pre> 
</div> 
<p>The two signals surface in different places, and that decides what you can monitor.</p> 
<ul> 
 <li><strong>Error 63955 is returned to the client session that ran the statement.</strong> You see it in your SQL client, your application or schema-migration tool logs, or your deployment pipeline.</li> 
 <li><strong>The error log records a warning tagged</strong> <code>VIRTUAL_INDEX_ID_OVERFLOW_WARNING</code> <strong>(MY-014078)</strong> on Aurora MySQL 3.10.0 and higher, as the limit approaches. The log line does not contain 63955, but it does include the ID of the index being created. The session that ran the DDL also receives warning 63955, but the statement succeeds, and most applications and schema-migration tools do not surface SQL warnings. A cluster can therefore write the warning for weeks without anyone noticing.</li> 
</ul> 
<p>In the error log, the warning and the error look like the following. The ID values are illustrative.</p> 
<div class="hide-language"> 
 <pre><code class="language-text">[Warning] [MY-014078] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_WARNING] Index `idx_email_lower` in table `app_db/customer_profile` has id 4000000123 and is approaching virtual index id limit of 4294967295.
[ERROR] [MY-014079] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_ERROR] Index `idx_email_lower` in table `app_db/customer_profile` cannot be created because index id 4294967296 is above virtual index id limit of 4294967295.</code></pre> 
</div> 
<p>To confirm the diagnosis without touching a production table, create a functional index like the example above on a small, existing, non-critical table, and then drop it. An index on an ordinary column is not a valid test, because it succeeds even on a cluster past the limit. Use an existing table rather than creating one, because <code>CREATE TABLE</code> with an index that contains a virtual column can fail the same way. For retrieving the error log, refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_LogAccess.html" target="_blank" rel="noopener">Monitoring Amazon Aurora log files</a>.</p> 
<h2>Converting to STORED: A short-term workaround</h2> 
<p>Because the limit applies only to indexes that contain a virtual column, converting the generated column in a failing index from <code>VIRTUAL</code> to <code>STORED</code> and rebuilding the index takes that index out of the check. In one case, the customer could create new indexes again after this change. It is a workaround, not a fix: the conversion does not reset the IDs, and any new index that contains a virtual column, including a functional or multi-valued index, still fails. Plan the logical rebuild anyway.</p> 
<p>Weigh these trade-offs before you use it.</p> 
<ul> 
 <li><strong>The conversion rebuilds the table.</strong> Changing a column from <code>VIRTUAL</code> to <code>STORED</code> copies the table, which takes time on large tables, and the stored values add storage. The rebuild fails with error 63955 if the table has another index that still contains a virtual column.</li> 
 <li><strong><code>STORED</code> is only valid on generated columns.</strong> Applying it to an ordinary column returns <code>Error Code: 1064</code>, a syntax error.</li> 
</ul> 
<h2>Solution overview</h2> 
<p>The recommended resolution in the AWS documentation is to recreate your database with a logical dump and restore. A logical rebuild recreates the schema and reloads the data, rather than copying the existing physical structures. Snapshot restore, Aurora cloning, point-in-time recovery, and Blue/Green deployments all produce a physical copy of the volume, and none of them is a documented way to clear this condition.</p> 
<p>The procedure has these stages:</p> 
<ol type="1"> 
 <li>Turn on binary logging on the source cluster.</li> 
 <li>Clone the source and record the binary log position of the clone.</li> 
 <li>Dump the clone with <code>MyDumper</code> and restore into a new Aurora MySQL 3.10 cluster with <code>MyLoader</code>.</li> 
 <li>Migrate users and grants.</li> 
 <li>Validate the new cluster.</li> 
 <li>Replicate from the source to the new cluster, starting at the recorded position.</li> 
 <li>Cut over by stopping traffic, letting replication catch up, and switching endpoints.</li> 
</ol> 
<p>The documentation does not mandate a new cluster as the target. Reloading in place on the same cluster is possible, but it takes the affected databases offline for the whole reload and removes the rollback path. This post uses a new cluster.</p> 
<h2>Prerequisites</h2> 
<ul> 
 <li><strong>AWS Identity and Access Management (IAM) permissions</strong> to create and modify Aurora DB clusters, instances, and cluster parameter groups (<code>rds:CreateDBCluster</code>, <code>rds:CreateDBInstance</code>, <code>rds:RestoreDBClusterToPointInTime</code> for the clone, <code>rds:ModifyDBCluster</code>, <code>rds:ModifyDBClusterParameterGroup</code>, <code>rds:RebootDBInstance</code>, <code>rds:DescribeEvents</code>), plus <code>logs:PutMetricFilter</code>, <code>logs:PutRetentionPolicy</code>, and <code>cloudwatch:PutMetricAlarm</code> for the alarm.</li> 
 <li><strong>A custom DB cluster parameter group</strong> on the source cluster. You cannot change <code>binlog_format</code> in the default group.</li> 
 <li><strong>An Amazon Elastic Compute Cloud (Amazon EC2) host</strong> in the same Region and virtual private cloud (VPC) as the cluster, with <code>MyDumper</code> and <code>MyLoader</code> installed. This post was tested with <code>MyDumper</code> and <code>MyLoader</code> v1.0.11-1 against Aurora MySQL 3.10.3. Pin the version you test with.</li> 
 <li><strong>Amazon Elastic Block Store (Amazon EBS)</strong> for the dump. Dump files are commonly about a third of the size that <code>information_schema</code> reports, but size the volume with headroom, because the ratio depends on your data.</li> 
 <li><strong>A maintenance window for one reboot of the source writer</strong>, described in the next section.</li> 
</ul> 
<h2>Step 1: Turn on binary logging on the source</h2> 
<p>Replication from the source to the new cluster needs binary logging on the source. On Aurora MySQL, <code>binlog_format</code> defaults to <code>OFF</code>, and changing it from <code>OFF</code> requires a reboot of the cluster. <strong>On many clusters, this reboot is the first planned downtime of the migration</strong>, so schedule it like one. If <code>binlog_format</code> is already set on your source, skip to the retention step.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws rds modify-db-cluster-parameter-group \
--db-cluster-parameter-group-name my-cluster-params \
--parameters "ParameterName=binlog_format,ParameterValue=MIXED,ApplyMethod=pending-reboot"
aws rds reboot-db-instance --db-instance-identifier my-cluster-writer</code></pre> 
</div> 
<p>Then connect to the source writer and retain binary logs for longer than the whole dump, restore, validation, and replication window. Aurora MySQL accepts up to 2,160 hours (90 days).</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">CALL mysql.rds_set_configuration('binlog retention hours', 168);
CALL mysql.rds_show_configuration;</code></pre> 
</div> 
<p>If the retention period ends before replication starts, the source deletes binary logs the new cluster still needs, and you must start the migration again. For the procedure, refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Replication.MySQL.SettingUp.html" target="_blank" rel="noopener">Setting up binary log replication for Aurora MySQL</a>.</p> 
<h2>Step 2: Clone the source and record the binary log position</h2> 
<p>Dump from an Aurora clone rather than from the source. No production workload competes with the dump, no writer-side DDL can terminate its connections, and nothing writes to the clone, so every <code>MyDumper</code> thread sees the same data. Create the clone after binary logging is on. For the procedure, refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Clone.html" target="_blank" rel="noopener">Cloning a volume for an Amazon Aurora DB cluster</a>.</p> 
<p>Replication must start at the source’s binary log position at the moment the clone was created. Do not use a position that <code>MyDumper</code> records from the clone, which describes the clone’s own binary log and would make replication skip or repeat writes. Aurora records the right position as an event on the clone’s writer instance.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws rds describe-events \
--source-type db-instance \
--source-identifier my-cluster-clone-writer \
--duration 1440 \
--query "Events[?contains(Message, 'Binlog position')].[Date,Message]" \
--output table</code></pre> 
</div> 
<p>The output contains a line like the following. Save the file name and position.</p> 
<div class="hide-language"> 
 <pre><code class="language-text">Binlog position from crash recovery is mysql-bin-changelog.000003 4278</code></pre> 
</div> 
<p>The same line also appears in the clone writer’s MySQL error log.</p> 
<h2>Step 3: Dump the clone and restore into a new cluster</h2> 
<p>Create the new cluster on the current Aurora MySQL 3.10 LTS release or later. Binary log replication supports a target that is the same version as the source or newer, and the rebuild is the cheapest moment to get the version that writes the advance warning.</p> 
<p>Before the dump, set <code>max_execution_time</code> to <code>0</code> in the clone’s parameter group. <code>MyDumper</code> <code>SELECT</code> statements on large tables can exceed a non-zero value, and the dump fails with <code>Query execution was interrupted, maximum statement execution time exceeded</code>. The parameter is dynamic. If the dump drops with <code>Lost connection to MySQL server during query</code>, check whether <code>net_read_timeout</code>, <code>net_write_timeout</code>, or <code>wait_timeout</code> have been lowered in your parameter groups, and raise them for the migration window.</p> 
<p>Store the credentials in a defaults file rather than on the command line, so that the password does not appear in process listings.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">cat &gt; ~/.mydumper.cnf &lt;&lt;'EOF'
[mydumper]
user=admin
password=&lt;password&gt;
[myloader]
user=admin
password=&lt;password&gt;
EOF
chmod 600 ~/.mydumper.cnf</code></pre> 
</div> 
<p>Dump all user databases from the clone’s cluster endpoint, excluding the system schemas.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">mydumper \
--defaults-file=$HOME/.mydumper.cnf \
--host=&lt;clone-cluster-endpoint&gt; \
--outputdir=/mnt/full-cluster-dump \
--threads=32 \
--rows=500000 \
--compress \
--events \
--routines \
--triggers \
--sync-thread-lock-mode=SAFE_NO_LOCK \
--regex '^(?!(mysql|sys|information_schema|performance_schema)(\.|$))' \
--verbose=3</code></pre> 
</div> 
<p><code>MyDumper</code> detects Aurora and refuses to run in its default lock mode, so you must set <code>--sync-thread-lock-mode</code>. Without it, the dump stops with this message:</p> 
<div class="hide-language"> 
 <pre><code class="language-text">We support LOCK_ALL and SAFE_NO_LOCK modes for RDS/Aurora. Select one of them to configure --sync-thread-lock-mode</code></pre> 
</div> 
<p><code>SAFE_NO_LOCK</code> takes no global lock, and it fails the dump if the worker threads do not start at the same point in time, so a dump that completes is consistent. The <code>--regex</code> filter is required, not just tidy. <code>MyDumper</code> applies the pattern to <code>database.table</code> for tables and to the bare schema name for schema-level objects, such as each schema’s <code>CREATE DATABASE</code> statement. The <code>(\.|$)</code> part therefore excludes a system schema name only when a dot or the end of the name follows it. Without it, the filter also skips customer schemas whose names start with the same words, such as <code>sys_audit</code> or <code>mysql_app</code>, and the dump leaves them out without a warning. Without the filter at all, <code>MyDumper</code> reaches the non-transactional table <code>mysql.general_log_backup</code> and stops the consistent dump. A successful run logs lines like these:</p> 
<div class="hide-language"> 
 <pre><code class="language-text">Using --trx-tables options, binlog coordinates will not be accurate if you are writing to non transactional tables.
Executing in SAFE_NO_LOCK mode. This backup will fail if all threads are not in the same point in time, which ensures consistency
Transactions started, unlocking tables
Finished dump at: 2026-11-10 15:07:21</code></pre> 
</div> 
<p>The log also reports <code>@@tokudb_version not found - ERROR 1193</code>. <code>MyDumper</code> checks for the TokuDB engine, which Aurora does not have, so you can ignore this line. The warning about <code>--trx-tables</code> and binlog coordinates does not affect this procedure, because replication starts from the position in the clone’s event in Step 2, not from the position <code>MyDumper</code> records.</p> 
<p>Then restore into the new cluster.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">myloader \
--defaults-file=$HOME/.mydumper.cnf \
--host=&lt;new-cluster-writer-endpoint&gt; \
--directory=/mnt/full-cluster-dump \
--threads=32 \
--verbose=3</code></pre> 
</div> 
<p>The new cluster is empty, so <code>MyLoader</code> needs no option to drop or overwrite existing tables. At the end of the restore, <code>MyLoader</code> compares checksums of each table’s structure and indexes, the views, and the events against values recorded in the dump, and logs a line such as <code>Structure checksum confirmed for shop.orders</code> for each one, followed by <code>Restore completed</code>. These checks cover structure, not row data, so still run the validation in Step 5.</p> 
<p>In one migration, a database that <code>information_schema</code> reported at about 400 GB produced a dump of about 115 GB. The reported size includes secondary index storage, which the dump carries only as DDL, plus InnoDB page overhead and fragmentation. Treat the ratio as an illustration, not a planning figure.</p> 
<h2>Step 4: Migrate users and grants</h2> 
<p>This is the step most likely to break a cutover, because it surfaces only when the application points at the new cluster and nothing can authenticate. <code>MyDumper</code> and <code>MyLoader</code> do not migrate accounts, which live in the <code>mysql</code> system schema.</p> 
<p>On Aurora MySQL 3, which is compatible with MySQL 8.0, <code>GRANT</code> no longer creates a user, and <code>SHOW GRANTS</code> does not include the password. Replaying grants alone therefore fails against accounts that do not exist on the new cluster. Generate <code>SHOW CREATE USER</code> statements first, then <code>SHOW GRANTS</code>, and replay them in that order.</p> 
<p>Run the following on the clone, not on the source. The clone matches the binary log position you recorded in Step 2. Any account created or changed on the source after the clone was taken reaches the new cluster through replication in Step 6, instead of appearing in both the script and the binary log. The query skips the RDS-reserved accounts, the AWS-managed roles, and the primary (<code>admin</code>) user, which already exists on the new cluster.</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SET SESSION print_identified_with_as_hex = ON;
SELECT CONCAT('SHOW CREATE USER ', QUOTE(user), '@', QUOTE(host), ';')
FROM mysql.user
WHERE user NOT IN ('rdsadmin', 'rdsrepladmin', 'rdsproxyadmin', 'admin',
'mysql.sys', 'mysql.session', 'mysql.infoschema',
'rds_superuser_role')
AND user NOT LIKE 'AWS\_%'
UNION ALL
SELECT CONCAT('SHOW GRANTS FOR ', QUOTE(user), '@', QUOTE(host), ';')
FROM mysql.user
WHERE user NOT IN ('rdsadmin', 'rdsrepladmin', 'rdsproxyadmin', 'admin',
'mysql.sys', 'mysql.session', 'mysql.infoschema',
'rds_superuser_role')
AND user NOT LIKE 'AWS\_%';</code></pre> 
</div> 
<p>Replace <code>admin</code> with your primary user name. <code>print_identified_with_as_hex</code> makes the password hashes print as hexadecimal, because <code>caching_sha2_password</code> hashes can contain characters that do not survive a copy and paste.</p> 
<p>Run the generated statements, and save their output as a script with every <code>CREATE USER</code> statement before any <code>GRANT</code>. User-defined roles are accounts in <code>mysql.user</code>, so they come through the same way, as locked accounts with no password. The <code>CREATE USER</code> output for a user also carries that user’s <code>DEFAULT ROLE</code>, and the role it names must already exist when the statement runs. Move the <code>CREATE USER</code> statements for roles (the ones that end in <code>ACCOUNT LOCK</code>) to the top of the script. Replay the script on the new cluster after the data has loaded, because table-level and routine-level grants fail against objects that do not exist yet. Because the hashes transfer unchanged, application passwords stay the same.</p> 
<p>On Aurora MySQL version 2, which is compatible with MySQL 5.7, the same two-statement approach works, without the <code>print_identified_with_as_hex</code> setting.</p> 
<p>For a role <code>app_read</code> and a user <code>app_ro</code> that has the role as its default, the replay script looks like this (hashes shortened):</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">CREATE USER `app_read`@`%` IDENTIFIED WITH 'mysql_native_password' REQUIRE NONE PASSWORD EXPIRE ACCOUNT LOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT;
CREATE USER `app_ro`@`%` IDENTIFIED WITH 'mysql_native_password' AS '&lt;hash&gt;' DEFAULT ROLE `app_read`@`%` REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT;
GRANT USAGE ON *.* TO `app_read`@`%`;
GRANT SELECT ON `shop`.* TO `app_read`@`%`;
GRANT USAGE ON *.* TO `app_ro`@`%`;
GRANT `app_read`@`%` TO `app_ro`@`%`;</code></pre> 
</div> 
<p>After the replay, log in to the new cluster as one or two application accounts and run a query they depend on. That confirms both the password hash and the grants before you cut over.</p> 
<h2>Step 5: Validate the new cluster</h2> 
<p>Validate more than row counts. <code>COUNT(*)</code> misses lost routines, triggers, events, views, and secondary indexes. Run the following on the clone and on the new cluster, and compare the output.</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SELECT s.schema_name,
(SELECT COUNT(*) FROM information_schema.tables t
WHERE t.table_schema = s.schema_name AND t.table_type = 'BASE TABLE') AS tables_,
(SELECT COUNT(*) FROM information_schema.views v
WHERE v.table_schema = s.schema_name) AS views_,
(SELECT COUNT(DISTINCT st.table_name, st.index_name) FROM information_schema.statistics st
WHERE st.table_schema = s.schema_name) AS indexes_,
(SELECT COUNT(*) FROM information_schema.routines r
WHERE r.routine_schema = s.schema_name) AS routines_,
(SELECT COUNT(*) FROM information_schema.triggers tr
WHERE tr.trigger_schema = s.schema_name) AS triggers_,
(SELECT COUNT(*) FROM information_schema.events e
WHERE e.event_schema = s.schema_name) AS events_
FROM information_schema.schemata s
WHERE s.schema_name NOT IN ('mysql', 'sys', 'information_schema', 'performance_schema')
ORDER BY s.schema_name;</code></pre> 
</div> 
<p>Then compare exact <code>SELECT COUNT(*)</code> results on your largest tables. For critical tables, <code>CHECKSUM TABLE</code> on both sides gives a stronger check, but it reads every row, so run it on the clone rather than on the source.</p> 
<p>Finally, run the statement that failed on the old cluster. It should now succeed.</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">CREATE INDEX idx_email_lower ON app_db.customer_profile ((LOWER(email)));</code></pre> 
</div> 
<h2>Step 6: Replicate and cut over</h2> 
<p>Start replication on the new cluster from the source, using the binary log file name and position you recorded in Step 2. For the procedures, refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Replication.MySQL.SettingUp.html" target="_blank" rel="noopener">Setting up binary log replication for Aurora MySQL</a>, which covers <code>mysql.rds_set_external_source</code> and <code>mysql.rds_start_replication</code>.</p> 
<p>If replication stops on an account statement, such as error 1396 (Operation <code>CREATE USER</code> failed), the replay script and the binary log contain the same account change. Check that you generated the script on the clone, as described in Step 4. If the account already matches the source, skip the duplicate statement with <code>mysql.rds_skip_repl_error</code> and replication continues.</p> 
<p>When replication has caught up, cut over:</p> 
<ol type="1"> 
 <li>Stop application traffic to the source cluster.</li> 
 <li>Confirm that replication lag on the new cluster is zero, then stop replication.</li> 
 <li>Switch endpoints, using one of the options in the next section.</li> 
 <li>Resume traffic.</li> 
</ol> 
<p>The dump and restore happen before this window, so the application is offline only for the switchover. Test the sequence in a non-production environment before you schedule it.</p> 
<h3>Keeping your application endpoints unchanged</h3> 
<ul> 
 <li><strong>Rename the cluster identifiers.</strong> Rename the old cluster, for example from <code>my-cluster</code> to <code>my-cluster-old</code>, then rename the new cluster to the original name. The cluster endpoints follow the new identifier. Instance endpoints and custom endpoints do not, so rename those separately if your applications use them. Clients also keep the old DNS response until its time to live (TTL) expires, so allow for that in the cutover window. Refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Modifying.html" target="_blank" rel="noopener">Modifying an Amazon Aurora DB cluster</a>.</li> 
 <li><strong>Use an Amazon Route 53 CNAME.</strong> If applications connect through your own DNS name, update the CNAME at cutover. The same TTL consideration applies.</li> 
 <li><strong>Use Amazon RDS Proxy.</strong> Retarget the proxy to the new cluster without changing connection strings. Refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy.html" target="_blank" rel="noopener">Using Amazon RDS Proxy</a>.</li> 
</ul> 
<h2>Considerations and limitations</h2> 
<ul> 
 <li><strong>The binary log reboot.</strong> Turning on binary logging on the source requires a reboot, as described in Step 1.</li> 
 <li><strong>Rollback only lasts until the first schema change.</strong> For a rollback path, configure reverse replication from the new cluster to the old one before cutover. This requires binary logging on the new cluster as well, which you can turn on before it takes traffic. The old cluster is still at the identifier limit, so any index DDL that runs on the new cluster after cutover replicates back, fails with 63955, and stops reverse replication. Freeze index DDL until you have decided not to roll back, or accept that rollback is valid only until the first schema change.</li> 
 <li><strong>Dump the clone, not the live cluster.</strong> Replication starts from the clone’s binary log position in Step 2, so the dump must come from the same clone. A dump of the live cluster would not line up with that position.</li> 
 <li><strong>In-place reload.</strong> Reloading on the same cluster avoids a second cluster but takes the databases offline for the whole reload and has no rollback.</li> 
</ul> 
<h2>Cost considerations</h2> 
<p>This migration runs paid resources in parallel. Plan for the following.</p> 
<ul> 
 <li><strong>Two Aurora clusters.</strong> The source and the new cluster both bill for instance hours and storage during migration and validation.</li> 
 <li><strong>An Aurora clone.</strong> The clone bills for its DB instances. Its storage cost starts small and grows as the clone and source diverge.</li> 
 <li><strong>An Amazon EC2 dump host</strong> and the Amazon EBS volume that holds the dump files.</li> 
 <li><strong>Binary log retention</strong> on the source, which adds storage for the retained logs.</li> 
</ul> 
<p>Keep the validation window as short as your testing allows, and clean up as soon as validation passes.</p> 
<h2>Clean up</h2> 
<p>Most of the clean-up is ordinary deletion. Terminate the Amazon EC2 dump host, delete the Amazon EBS volume that held the dump (it is not deleted with the instance unless its <code>DeleteOnTermination</code> attribute is set), and delete the clone’s instances and then the clone cluster. For the procedure, refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_DeleteCluster.html" target="_blank" rel="noopener">Deleting Aurora DB clusters and DB instances</a>.</p> 
<p>Three steps are specific to this migration.</p> 
<p><strong>Stop replication.</strong> On the new cluster, and on the old one if you configured reverse replication, run the following. The second procedure has a different name on Aurora MySQL version 2. Refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/mysql-stored-proc-replicating.html" target="_blank" rel="noopener">Configuring, starting, and stopping binary log replication</a>.</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">CALL mysql.rds_stop_replication;
CALL mysql.rds_reset_external_source;</code></pre> 
</div> 
<p><strong>Clear binary log retention</strong> on any cluster you keep that no longer replicates, so that it stops retaining log files. The value <code>0</code> is not accepted.</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">CALL mysql.rds_set_configuration('binlog retention hours', NULL);</code></pre> 
</div> 
<p><strong>Delete the old cluster last, with a final snapshot.</strong> Do this only after the new cluster has served production traffic normally and you no longer need the rollback path. The old cluster holds the only copy of your pre-migration data. Turn off deletion protection, delete its instances, then delete the cluster with a final snapshot.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws rds modify-db-cluster --db-cluster-identifier my-cluster-old \
--no-deletion-protection --apply-immediately
aws rds delete-db-cluster --db-cluster-identifier my-cluster-old \
--final-db-snapshot-identifier my-cluster-old-final \
--no-delete-automated-backups</code></pre> 
</div> 
<h2>Prevention</h2> 
<ul> 
 <li><strong>Alarm on the virtual index ID overflow messages in the error log.</strong> This is your only advance signal. The next section walks through the setup.</li> 
 <li><strong>Reduce unnecessary DDL churn.</strong> Avoid repeatedly creating and dropping indexes, and treat table-rebuilding <code>ALTER TABLE</code> as drawing down a finite budget.</li> 
 <li><strong>Limit indexes that contain virtual columns</strong> on clusters with heavy DDL. Aurora MySQL does not check indexes on ordinary columns or <code>STORED</code> generated columns against the limit.</li> 
 <li><strong>Plan a periodic logical rebuild</strong> for databases with very heavy DDL activity, so that the work happens on your schedule.</li> 
 <li><strong>Stay on Aurora MySQL 3.10.0 or later.</strong> Earlier versions give no advance warning.</li> 
</ul> 
<h3>Setting up the error-log alarm</h3> 
<p>Turn on error-log export to Amazon CloudWatch Logs for the cluster.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws rds modify-db-cluster \
--db-cluster-identifier my-cluster \
--cloudwatch-logs-export-configuration '{"EnableLogTypes":["error"]}'</code></pre> 
</div> 
<p>Aurora creates the log group <code>/aws/rds/cluster/my-cluster/error</code> with no expiry, so set a retention period.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws logs put-retention-policy \
--log-group-name /aws/rds/cluster/my-cluster/error \
--retention-in-days 90</code></pre> 
</div> 
<p>Create a metric filter that matches the warning. The pattern matches the two tags that Aurora MySQL writes to the error log: <code>VIRTUAL_INDEX_ID_OVERFLOW_WARNING</code> as the limit approaches, and <code>VIRTUAL_INDEX_ID_OVERFLOW_ERROR</code> when an index is refused. Match the tags rather than the message text. Filter patterns are case sensitive, and the error-log message spells id in lowercase, unlike the client error. The bare number 63955 is also a poor match, because it can appear in log sequence numbers, thread IDs and byte counts.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws logs put-metric-filter \
--log-group-name /aws/rds/cluster/my-cluster/error \
--filter-name aurora-virtual-index-id-warning \
--filter-pattern '?"VIRTUAL_INDEX_ID_OVERFLOW_WARNING" ?"VIRTUAL_INDEX_ID_OVERFLOW_ERROR"' \
--metric-transformations \
metricName=VirtualIndexIdWarning,metricNamespace=Custom/AuroraMySQL,metricValue=1,defaultValue=0</code></pre> 
</div> 
<p>Test the pattern before relying on it. A healthy cluster has no warning line to test with, so the following command uses a sample line in the format that Aurora MySQL writes. The output should show one match.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws logs test-metric-filter \
--filter-pattern '?"VIRTUAL_INDEX_ID_OVERFLOW_WARNING" ?"VIRTUAL_INDEX_ID_OVERFLOW_ERROR"' \
--log-event-messages '[Warning] [MY-014078] [InnoDB] [VIRTUAL_INDEX_ID_OVERFLOW_WARNING] Index `idx_b` in table `app_db/orders` has id 4000000123 and is approaching virtual index id limit of 4294967295.'</code></pre> 
</div> 
<p>Then create the alarm. Aurora writes the error log to disk every five minutes and rotates it hourly, so a one-hour period is appropriate.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws cloudwatch put-metric-alarm \
--alarm-name aurora-virtual-index-id-warning-my-cluster \
--alarm-description "Aurora MySQL virtual index ID overflow warning or error" \
--namespace Custom/AuroraMySQL \
--metric-name VirtualIndexIdWarning \
--statistic Sum \
--period 3600 \
--evaluation-periods 1 \
--threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:us-east-1:111122223333:my-dba-topic</code></pre> 
</div> 
<p><code>--treat-missing-data notBreaching</code> matters because a healthy cluster can go hours without writing to the error log. Metrics extracted from log events are billed as custom metrics.</p> 
<h3>Covering every cluster</h3> 
<p>A metric filter applies to one log group, and each cluster has its own. The following loop sets up export, retention, the filter, and the alarm for every Aurora MySQL cluster in the Region. Log export needs no parameter group change, so the loop also works on clusters that use the default parameter group.</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">FILTER='?"VIRTUAL_INDEX_ID_OVERFLOW_WARNING" ?"VIRTUAL_INDEX_ID_OVERFLOW_ERROR"'
TOPIC=arn:aws:sns:us-east-1:111122223333:my-dba-topic
for c in $(aws rds describe-db-clusters \
--query "DBClusters[?Engine=='aurora-mysql'].DBClusterIdentifier" --output text); do
aws rds modify-db-cluster --db-cluster-identifier "$c" \
--cloudwatch-logs-export-configuration '{"EnableLogTypes":["error"]}' &gt;/dev/null
lg="/aws/rds/cluster/$c/error"
until aws logs describe-log-groups --log-group-name-prefix "$lg" \
--query 'logGroups[0].logGroupName' --output text | grep -q "$lg"; do sleep 30; done
aws logs put-retention-policy --log-group-name "$lg" --retention-in-days 90
aws logs put-metric-filter --log-group-name "$lg" \
--filter-name aurora-virtual-index-id-warning --filter-pattern "$FILTER" \
--metric-transformations "metricName=VirtualIndexIdWarning-$c,metricNamespace=Custom/AuroraMySQL,metricValue=1,defaultValue=0"
aws cloudwatch put-metric-alarm --alarm-name "aurora-virtual-index-id-warning-$c" \
--namespace Custom/AuroraMySQL --metric-name "VirtualIndexIdWarning-$c" \
--statistic Sum --period 3600 --evaluation-periods 1 --threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--treat-missing-data notBreaching --alarm-actions "$TOPIC"
done</code></pre> 
</div> 
<p>The metric name includes the cluster identifier, so each alarm tracks one cluster. For an infrastructure-as-code version, the metric filter and alarm map to the <code>AWS::Logs::MetricFilter</code> and <code>AWS::CloudWatch::Alarm</code> AWS CloudFormation types.</p> 
<p>For reference, refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.CloudWatch.html" target="_blank" rel="noopener">Publishing Amazon Aurora MySQL logs to Amazon CloudWatch Logs</a> and <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html" target="_blank" rel="noopener">Creating metrics from log events using filters</a>.</p> 
<h2>Conclusion</h2> 
<p>Error 63955 means the cluster has used up the virtual index IDs Aurora MySQL uses when it evaluates indexes that contain a virtual column, including functional and multi-valued indexes. Converting a generated column to <code>STORED</code> is a short-term workaround for one index, no physical copy clears the condition, and the documented resolution is a logical dump and restore into a new cluster.</p> 
<p>Plan four things carefully: the reboot to turn on binary logging, the binary log position of the clone, users and grants as their own step, and the rollback limit after the first schema change. Restore into Aurora MySQL 3.10 or later, and alarm on the <code>VIRTUAL_INDEX_ID_OVERFLOW_WARNING</code> message in the error log so the next cluster gives you notice.</p> 
<p>If you are unsure which restore path fits your cluster and engine version, raise a case with AWS Support before you commit to a migration window.</p> 
<p>For more detail, refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.BestPractices.Performance.html#AuroraMySQL.BestPractices.Performance.VirtualIndexIDOverflow" target="_blank" rel="noopener">Virtual index ID overflow errors</a>, the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraMySQLReleaseNotes/AuroraMySQL.Updates.3100.html" target="_blank" rel="noopener">Aurora MySQL 3.10.0 release notes</a>, and <a href="https://aws.amazon.com/blogs/database/migrate-very-large-databases-to-amazon-aurora-mysql-using-mydumper-and-myloader/" target="_blank" rel="noopener">Migrate very large databases to Amazon Aurora MySQL using MyDumper and MyLoader</a>.</p> 
<p> </p> 
<hr> 
<h2>About the author</h2> 
<footer> 
 <div class="blog-author-box"> 
  <div class="blog-author-image">
   <img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/12/DBBLOG-5864-1.jpg" alt="Kartik Saini" width="100" height="133">
  </div> 
  <h3 class="lb-h4">Kartik Saini</h3> 
  <p>Kartik is a Cloud Support Engineer on the Amazon RDS and Amazon Aurora team at AWS. He specializes in troubleshooting MySQL-compatible database engines, and works with customers to diagnose and resolve complex Aurora MySQL issues.</p> 
 </div> 
</footer>]]></content:encoded>
    <pubDate>Fri, 13 Nov 2026 21:10:28 +0000</pubDate>
    <dc:creator>AWS Database Blog</dc:creator>
    <category>Advanced (300)</category>
    <category>Amazon Aurora</category>
    <category>Technical How-to</category>
  </item>

  <item>
    <title>Village News: MySQL News + Events (13 November 2026)</title>
    <guid isPermaLink="false">6af73df49329730001d5e851</guid>
    <link>https://villagesql.com/blog/village-news-mysql-news-events-13-november-2026/</link>
    <description>Welcome back to Village News, our curated roundup of MySQL and database news. This week, PostgreSQL shipped updates fixing 52 security vulnerabilities and retired PostgreSQL 14, and the FOSDEM M-Ecosystem devroom opened its call for participation.
If you want to get these updates, just subscribe to the blog.
Enjoy!
MySQL News
Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated)
Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights
Marco Tusa, Percona Blog
TL;DR - pc.weight lets you run an even-numbered Galera/PXC cluster, but working out quorum and views gets hard fast. Tusa's advice: keep to three nodes unless you truly need more.
Taming MySQL 8.4 Background Histogram Lock Wait Timeouts on Amazon RDS
Umesh Shastry, MySQL :: under_the_hood
TL;DR - On RDS for MySQL 8.4, background histogram updates collide with writes to mysql.rds_heartbeat2 and flood the log with MY-015116 warnings. The fix: drop that table's histograms, not change global settings.
Help prevent out-of-memory restarts on Aurora MySQL 8.4
AWS Database Blog
TL;DR - AWS reproduces a memory-pressure event to show how Aurora MySQL 8.4's automatic memory management avoids a restart, what to alarm on (AuroraMemoryHealthState), and when to take manual control.
How Oracle changed the future of MySQL: Dominic Preuss on Inside the Silicon Mind
VillageSQL
TL;DR - In an interview with Firas Sozan, VillageSQL's CEO covers MySQL's path from MySQL AB to Oracle, why investment went to PostgreSQL, and Oracle opening MySQL development to contributors such as AWS and Google.
DuckDB × MySQL Day: Eight Talks, One Question
OurSQL Foundation Blog
TL;DR - A walk through the agenda for the free online DuckDB × MySQL Day on November 17: eight talks on whether an embedded columnar engine can finally make analytics next to MySQL practical.
FOSDEM 2027 – The M-Ecosystem Devroom: Call for Participation
FOSDEM MySQL &amp;amp; Friends Devroom
TL;DR - The successor to the MariaDB, MySQL &amp;amp; Friends devroom runs Sunday, January 31, 2027 in Brussels. Submissions close November 29; pick &quot;The M-ecosystem&quot; track.
Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL
Vadim Tkachenko, Percona Blog
TL;DR - In DBTrail's published benchmark, 12 of 14 reports ran faster on its DuckDB copy than on a Percona Server replica, and the slowest fell from 225 s to 7.94 s. The two lightest reports were slower on the copy.
Amazon RDS for MySQL 26.10 is now available in Amazon RDS Database Preview Environment
AWS What's New
TL;DR - MySQL 26.10 can now be tested on RDS. It reuses read views to remove a global mutex on busy READ COMMITTED workloads and logs warnings when history-list length crosses innodb_max_purge_lag.
Diagnosing MySQL at Scale (Part 1): Critical System Blocks, Metadata Locks, and Cluster Convoys
Umesh Shastry, MySQL :: under_the_hood
TL;DR - A four-part series on reading Performance Schema waits. Part 1 covers MDL, data-dictionary latches and Group Replication convoys; Part 2 covers storage, buffer pool latches and memory spills.
Releasing iClaustron API, a new Rust NDB API with a C and a Rust API towards RonDB 26.10.0
Mikael Ronström
TL;DR - iClaustron 1.0.0 is an MIT-licensed Rust/C NDB API for RonDB that reaches 9M key lookups/s on a laptop. Separately, Ronström charted his commits: 2,320 in 2026, up from 220 in 2025.
Vector search in MySQL: from a plugin to the top of the Contributor Summit agenda
Alkin Tezuysal, AskDba
TL;DR - A history of vector search in MySQL, from the MyVector plugin through adoption by MySQL forks to native work from hyperscalers. Oracle, Google, Percona and Intel opened the Contributor Summit with vector proposals.
Database News
PostgreSQL 18.7, 17.12, 16.16, 15.20, 14.25, and 19 RC 2 Released!
PostgreSQL Global Development Group
TL;DR - The update fixes 52 security vulnerabilities, several of them arbitrary-code-execution bugs rated CVSS 8.8, and over 140 other bugs. It is the final PostgreSQL 14 release, and some fixes need post-update steps.
TLS Connect/Disconnect Performance Boost With WolfSSL
Georgi Kodinov, MariaDB Foundation Blog
TL;DR - A two-line build fix lets MariaDB's bundled WolfSSL use its __uint128_t big-number code on non-x86 64-bit platforms. TLS connect/disconnect became more than 3x faster (4.5 s to 1.3 s per 100).
From an Isolated Project to the ADBC Driver Foundry: MariaDB Joins the CDN!
Frédéric Descamps, lefred's blog
TL;DR - lefred's native ADBC driver for MariaDB, which connects it to DuckDB over Apache Arrow, is now built and distributed by Columnar's ADBC Driver Foundry, with prebuilt binaries on its CDN.
Introducing Percona Database Service (PDS): An Experiment in Simplifying Database Management
Andrey Glazkov, Percona Blog
TL;DR - Percona released an early version of PDS, a self-hosted platform for provisioning and managing databases, starting with MySQL and PostgreSQL on Kubernetes. Percona is asking for feedback.
Benchmarking MariaDB Raft Cluster vs. MariaDB Galera Cluster
Axel Schwenke, MariaDB Corporation Blog
TL;DR - With one node at 80 ms RTT, Raft Cluster in Enterprise Server 12.3 held about 2,150 TPS against Galera's 125. On a low-latency three-node cluster the two were close. See also the Raft Cluster overview.
Ten ways a glibc change can break your PostgreSQL cluster
Pablo Svampa, Percona Blog
TL;DR - glibc upgrades can change locale collation order or remove locales, which silently breaks text indexes and comparisons. The post lists ten ways this hits a Postgres cluster, including RHEL 8→9→10 locale renames.
Giving the Optimizer a Hint: What’s New in MariaDB Server 13.0, compatibility and beyond
Oleg Smirnov, MariaDB Foundation Blog
TL;DR - MariaDB 13.0 adds MySQL-style /*+ ... */ optimizer hints for join order, indexes, join buffers, derived-table merging and more. It also adds MariaDB-specific extensions for views and query blocks.
Next-Level Benchmarking Overfitting
Thomas Neumann, Database Architects
TL;DR - Neumann tests the GenDB-V paper's claimed 22–61x TPC-H speedups over Umbra and DuckDB against memory-bandwidth limits. He argues they are infeasible for real query processing and come from AI-driven benchmark overfitting.
Upcoming Database Events
pgDay México 2026
November 13, 2026
Puerta Polanco
Mexico City, Mexico
DuckDB × MySQL Day
November 17, 2026
Online
PGConf.Asia 2026
November 17-18, 2026
Hong Kong
OurSQL Foundation Webinar: What Happened to the MySQL Query Cache?
November 18, 2026, 10:00 AM ET / 16:00 CET
Online
PgConf.Nepal 2026
November 18-21, 2026
Pulchowk Campus, Tribhuvan University
Kathmandu, Nepal
PGConf.PL 2026
November 24, 2026
ARCHE Dwór Uphagena
Gdańsk, Poland
PostgreSQL Conference Japan 2026
November 27, 2026
AP Tokyo Yaesu
Tokyo, Japan
AWS re:Invent 2026
November 30 – December 4, 2026
Las Vegas, NV
OurSQL Foundation Webinar: What Is the Fastest Database? Open-Source Benchmarking with HammerDB
December 3, 2026, 10:00 AM ET / 16:00 CET
Online
Open Source Summit Japan
December 7-9, 2026
Tokyo, Japan
CIDR 2027
January 24-27, 2027
Mövenpick Hotel Amsterdam City Centre
Amsterdam, The Netherlands
MySQL Belgian Days 2027
January 28-29, 2027
ICAB Incubator
Brussels, Belgium
FOSDEM PGDay 2027
January 28-29, 2027
Brussels Marriott Hotel Grand Place
Brussels, Belgium
FOSDEM 2027
January 30-31, 2027
ULB Solbosch Campus
Brussels, Belgium
CERN PGDay 2027
February 12, 2027
CERN Council Chamber
Geneva, Switzerland
ConFoo 2027
February 24-26, 2027
Hotel Bonaventure
Montréal, QC, Canada
PGConf India 2027
March 2-5, 2027
Sheraton Grand Hotel at Brigade Gateway
Bengaluru, India
FOSSASIA PGDay 2027
March 9, 2027
Bangkok, Thailand
KubeCon + CloudNativeCon Europe 2027
March 15-18, 2027
Barcelona, Spain
Nordic PGDay 2027
March 16, 2027
Courtyard Kungsholmen
Stockholm, Sweden
SCaLE 24x
April 1-4, 2027
Pasadena Convention Center
Pasadena, CA
DuckCon #8
April 7, 2027
De Wester (Westergas)
Amsterdam, The Netherlands
PgDay Boston 2027
April 10, 2027
Tufts University Joyce Cummings Center
Boston, MA
Postgres Conference 2027
April 21-23, 2027
San Jose Marriott
San Jose, CA
Percona Live Bay Area 2027
May 11-13, 2027
Computer History Museum
Mountain View, CA
PGConf.dev 2027
May 11-14, 2027
Plaza Centre-Ville
Montréal, QC, Canada</description>
    <content:encoded><![CDATA[<img src="https://storage.ghost.io/c/db/c7/dbc78ca4-dcfe-468b-b2fa-030396d2a98e/content/images/2026/11/village-news-banner-1.jpg" alt="Village News: MySQL News + Events (13 November 2026)"><p>Welcome back to Village News, our curated roundup of MySQL and database news. This week, PostgreSQL shipped updates fixing 52 security vulnerabilities and retired PostgreSQL 14, and the FOSDEM M-Ecosystem devroom opened its call for participation.</p>
<p>If you want to get these updates, just <a href="https://villagesql.com/blog/blog#/portal/signup">subscribe</a> to the blog.</p>
<p>Enjoy!</p>
<h2>MySQL News</h2>
<p><em>Note: Aggregated MySQL news can be found at <a href="https://planet.oursqlcommunity.org/?source=villagesql">Planet for MySQL Community</a> and <a href="https://planet.mysql.com/">Planet MySQL</a> (Oracle curated)</em></p>
<p><strong><a href="https://www.percona.com/blog/just-because-you-can-doesnt-mean-you-should-re-evaluating-galera-node-weights/">Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights</a></strong><br>
<em>Marco Tusa, Percona Blog</em></p>
<p><strong>TL;DR -</strong> pc.weight lets you run an even-numbered Galera/PXC cluster, but working out quorum and views gets hard fast. Tusa's advice: keep to three nodes unless you truly need more.</p>
<p><strong><a href="https://ushastry.blogspot.com/2026/10/taming-mysql-84-background-histogram.html">Taming MySQL 8.4 Background Histogram Lock Wait Timeouts on Amazon RDS</a></strong><br>
<em>Umesh Shastry, MySQL :: under_the_hood</em></p>
<p><strong>TL;DR -</strong> On RDS for MySQL 8.4, background histogram updates collide with writes to mysql.rds_heartbeat2 and flood the log with MY-015116 warnings. The fix: drop that table's histograms, not change global settings.</p>
<p><strong><a href="https://aws.amazon.com/blogs/database/help-prevent-out-of-memory-restarts-on-aurora-mysql-8-4/">Help prevent out-of-memory restarts on Aurora MySQL 8.4</a></strong><br>
<em>AWS Database Blog</em></p>
<p><strong>TL;DR -</strong> AWS reproduces a memory-pressure event to show how Aurora MySQL 8.4's automatic memory management avoids a restart, what to alarm on (AuroraMemoryHealthState), and when to take manual control.</p>
<p><strong><a href="https://villagesql.com/blog/silicon-mind/">How Oracle changed the future of MySQL: Dominic Preuss on Inside the Silicon Mind</a></strong><br>
<em>VillageSQL</em></p>
<p><strong>TL;DR -</strong> In an interview with Firas Sozan, VillageSQL's CEO covers MySQL's path from MySQL AB to Oracle, why investment went to PostgreSQL, and Oracle opening MySQL development to contributors such as AWS and Google.</p>
<p><strong><a href="https://oursqlfoundation.org/blog/duckdb-mysql-day-eight-talks/">DuckDB × MySQL Day: Eight Talks, One Question</a></strong><br>
<em>OurSQL Foundation Blog</em></p>
<p><strong>TL;DR -</strong> A walk through the agenda for the free online DuckDB × MySQL Day on November 17: eight talks on whether an embedded columnar engine can finally make analytics next to MySQL practical.</p>
<p><strong><a href="https://www.mysqlandfriends.eu/post/fosdem-2027-the-m-ecosystem-devroom-call-for-participation/">FOSDEM 2027 – The M-Ecosystem Devroom: Call for Participation</a></strong><br>
<em>FOSDEM MySQL &amp; Friends Devroom</em></p>
<p><strong>TL;DR -</strong> The successor to the MariaDB, MySQL &amp; Friends devroom runs Sunday, January 31, 2027 in Brussels. Submissions close November 29; pick "The M-ecosystem" track.</p>
<p><strong><a href="https://www.percona.com/blog/heavy-mysql-reports-in-seconds-open-source-htap-with-dbtrail-and-percona-server-for-mysql/">Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL</a></strong><br>
<em>Vadim Tkachenko, Percona Blog</em></p>
<p><strong>TL;DR -</strong> In DBTrail's published benchmark, 12 of 14 reports ran faster on its DuckDB copy than on a Percona Server replica, and the slowest fell from 225 s to 7.94 s. The two lightest reports were slower on the copy.</p>
<p><strong><a href="https://aws.amazon.com/about-aws/whats-new/2026/11/rds-mysql-26-10-preview-environment/">Amazon RDS for MySQL 26.10 is now available in Amazon RDS Database Preview Environment</a></strong><br>
<em>AWS What's New</em></p>
<p><strong>TL;DR -</strong> MySQL 26.10 can now be tested on RDS. It reuses read views to remove a global mutex on busy READ COMMITTED workloads and logs warnings when history-list length crosses innodb_max_purge_lag.</p>
<p><strong><a href="https://ushastry.blogspot.com/2026/11/diagnosing-mysql-at-scale-part-1.html">Diagnosing MySQL at Scale (Part 1): Critical System Blocks, Metadata Locks, and Cluster Convoys</a></strong><br>
<em>Umesh Shastry, MySQL :: under_the_hood</em></p>
<p><strong>TL;DR -</strong> A four-part series on reading Performance Schema waits. Part 1 covers MDL, data-dictionary latches and Group Replication convoys; <a href="https://ushastry.blogspot.com/2026/11/diagnosing-mysql-at-scale-part-2.html">Part 2</a> covers storage, buffer pool latches and memory spills.</p>
<p><strong><a href="https://mikaelronstrom.blogspot.com/2026/11/releasing-iclaustron-api-new-rust-ndb.html">Releasing iClaustron API, a new Rust NDB API with a C and a Rust API towards RonDB 26.10.0</a></strong><br>
<em>Mikael Ronström</em></p>
<p><strong>TL;DR -</strong> iClaustron 1.0.0 is an MIT-licensed Rust/C NDB API for RonDB that reaches 9M key lookups/s on a laptop. Separately, Ronström <a href="https://mikaelronstrom.blogspot.com/2026/11/git-commits-on-mysql-tree-impact-by-ai.html">charted his commits</a>: 2,320 in 2026, up from 220 in 2025.</p>
<p><strong><a href="https://askdba.net/vector-search-in-mysql-from-a-plugin-to-the-top-of-the-contributor-summit-agenda/">Vector search in MySQL: from a plugin to the top of the Contributor Summit agenda</a></strong><br>
<em>Alkin Tezuysal, AskDba</em></p>
<p><strong>TL;DR -</strong> A history of vector search in MySQL, from the MyVector plugin through adoption by MySQL forks to native work from hyperscalers. Oracle, Google, Percona and Intel opened the Contributor Summit with vector proposals.</p>
<h2>Database News</h2>
<p><strong><a href="https://www.postgresql.org/about/news/postgresql-187-1712-1616-1520-1425-and-19-rc-2-released-3442/">PostgreSQL 18.7, 17.12, 16.16, 15.20, 14.25, and 19 RC 2 Released!</a></strong><br>
<em>PostgreSQL Global Development Group</em></p>
<p><strong>TL;DR -</strong> The update fixes 52 security vulnerabilities, several of them arbitrary-code-execution bugs rated CVSS 8.8, and over 140 other bugs. It is the final PostgreSQL 14 release, and some fixes need post-update steps.</p>
<p><strong><a href="https://mariadb.org/tls-connect-disconnect-performance-boost-with-wolfssl/">TLS Connect/Disconnect Performance Boost With WolfSSL</a></strong><br>
<em>Georgi Kodinov, MariaDB Foundation Blog</em></p>
<p><strong>TL;DR -</strong> A two-line build fix lets MariaDB's bundled WolfSSL use its __uint128_t big-number code on non-x86 64-bit platforms. TLS connect/disconnect became more than 3x faster (4.5 s to 1.3 s per 100).</p>
<p><strong><a href="https://lefred.be/content/from-an-isolated-project-to-the-adbc-driver-foundry-mariadb-joins-the-cdn/">From an Isolated Project to the ADBC Driver Foundry: MariaDB Joins the CDN!</a></strong><br>
<em>Frédéric Descamps, lefred's blog</em></p>
<p><strong>TL;DR -</strong> lefred's native ADBC driver for MariaDB, which connects it to DuckDB over Apache Arrow, is now built and distributed by Columnar's ADBC Driver Foundry, with prebuilt binaries on its CDN.</p>
<p><strong><a href="https://www.percona.com/blog/percona-database-service-pds/">Introducing Percona Database Service (PDS): An Experiment in Simplifying Database Management</a></strong><br>
<em>Andrey Glazkov, Percona Blog</em></p>
<p><strong>TL;DR -</strong> Percona released an early version of PDS, a self-hosted platform for provisioning and managing databases, starting with MySQL and PostgreSQL on Kubernetes. Percona is asking for feedback.</p>
<p><strong><a href="https://mariadb.com/resources/blog/benchmarking-mariadb-raft-cluster-vs-mariadb-galera-cluster/">Benchmarking MariaDB Raft Cluster vs. MariaDB Galera Cluster</a></strong><br>
<em>Axel Schwenke, MariaDB Corporation Blog</em></p>
<p><strong>TL;DR -</strong> With one node at 80 ms RTT, Raft Cluster in Enterprise Server 12.3 held about 2,150 TPS against Galera's 125. On a low-latency three-node cluster the two were close. See also the <a href="https://mariadb.com/resources/blog/mariadb-raft-cluster-consistency-without-waiting-on-your-slowest-node/">Raft Cluster overview</a>.</p>
<p><strong><a href="https://www.percona.com/blog/ten-ways-a-glibc-change-can-break-your-postgresql-cluster/">Ten ways a glibc change can break your PostgreSQL cluster</a></strong><br>
<em>Pablo Svampa, Percona Blog</em></p>
<p><strong>TL;DR -</strong> glibc upgrades can change locale collation order or remove locales, which silently breaks text indexes and comparisons. The post lists ten ways this hits a Postgres cluster, including RHEL 8→9→10 locale renames.</p>
<p><strong><a href="https://mariadb.org/giving-the-optimizer-a-hint-whats-new-in-mariadb-server-13-0-compatibility-and-beyond/">Giving the Optimizer a Hint: What’s New in MariaDB Server 13.0, compatibility and beyond</a></strong><br>
<em>Oleg Smirnov, MariaDB Foundation Blog</em></p>
<p><strong>TL;DR -</strong> MariaDB 13.0 adds MySQL-style /*+ ... */ optimizer hints for join order, indexes, join buffers, derived-table merging and more. It also adds MariaDB-specific extensions for views and query blocks.</p>
<p><strong><a href="https://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html">Next-Level Benchmarking Overfitting</a></strong><br>
<em>Thomas Neumann, Database Architects</em></p>
<p><strong>TL;DR -</strong> Neumann tests the GenDB-V paper's claimed 22–61x TPC-H speedups over Umbra and DuckDB against memory-bandwidth limits. He argues they are infeasible for real query processing and come from AI-driven benchmark overfitting.</p>
<h2>Upcoming Database Events</h2>
<p><strong><a href="https://2026.pgday.com.mx/">pgDay México 2026</a></strong><br>
November 13, 2026<br>
Puerta Polanco<br>
Mexico City, Mexico</p>
<p><strong><a href="https://oursqlfoundation.org/events/duckdb-mysql-day/">DuckDB × MySQL Day</a></strong><br>
November 17, 2026<br>
Online</p>
<p><strong><a href="https://www.pgconf.asia/EN/2026/">PGConf.Asia 2026</a></strong><br>
November 17-18, 2026<br>
Hong Kong</p>
<p><strong><a href="https://oursqlfoundation.org/community/webinars/mysql-query-cache/">OurSQL Foundation Webinar: What Happened to the MySQL Query Cache?</a></strong><br>
November 18, 2026, 10:00 AM ET / 16:00 CET<br>
Online</p>
<p><strong><a href="https://2026.pgconf.org.np/">PgConf.Nepal 2026</a></strong><br>
November 18-21, 2026<br>
Pulchowk Campus, Tribhuvan University<br>
Kathmandu, Nepal</p>
<p><strong><a href="https://2026.pgconf.pl/">PGConf.PL 2026</a></strong><br>
November 24, 2026<br>
ARCHE Dwór Uphagena<br>
Gdańsk, Poland</p>
<p><strong><a href="https://www.postgresql.jp/jpug-pgcon2026">PostgreSQL Conference Japan 2026</a></strong><br>
November 27, 2026<br>
AP Tokyo Yaesu<br>
Tokyo, Japan</p>
<p><strong><a href="https://aws.amazon.com/events/reinvent">AWS re:Invent 2026</a></strong><br>
November 30 – December 4, 2026<br>
Las Vegas, NV</p>
<p><strong><a href="https://oursqlfoundation.org/community/webinars/fastest-database-hammerdb/">OurSQL Foundation Webinar: What Is the Fastest Database? Open-Source Benchmarking with HammerDB</a></strong><br>
December 3, 2026, 10:00 AM ET / 16:00 CET<br>
Online</p>
<p><strong><a href="https://events.linuxfoundation.org/open-source-summit-japan/">Open Source Summit Japan</a></strong><br>
December 7-9, 2026<br>
Tokyo, Japan</p>
<p><strong><a href="https://www.cidrdb.org/cidr2027/">CIDR 2027</a></strong><br>
January 24-27, 2027<br>
Mövenpick Hotel Amsterdam City Centre<br>
Amsterdam, The Netherlands</p>
<p><strong><a href="https://sessionize.com/mysql-belgian-days-2027/">MySQL Belgian Days 2027</a></strong><br>
January 28-29, 2027<br>
ICAB Incubator<br>
Brussels, Belgium</p>
<p><strong><a href="https://2027.fosdempgday.org/">FOSDEM PGDay 2027</a></strong><br>
January 28-29, 2027<br>
Brussels Marriott Hotel Grand Place<br>
Brussels, Belgium</p>
<p><strong><a href="https://fosdem.org/2027/">FOSDEM 2027</a></strong><br>
January 30-31, 2027<br>
ULB Solbosch Campus<br>
Brussels, Belgium</p>
<p><strong><a href="https://indico.cern.ch/event/1720940/">CERN PGDay 2027</a></strong><br>
February 12, 2027<br>
CERN Council Chamber<br>
Geneva, Switzerland</p>
<p><strong><a href="https://confoo.ca/en/2027">ConFoo 2027</a></strong><br>
February 24-26, 2027<br>
Hotel Bonaventure<br>
Montréal, QC, Canada</p>
<p><strong><a href="https://pgconf.in/">PGConf India 2027</a></strong><br>
March 2-5, 2027<br>
Sheraton Grand Hotel at Brigade Gateway<br>
Bengaluru, India</p>
<p><strong><a href="https://www.postgresql.org/about/event/fossasia-pgday-2027-returns-to-thailand-on-march-9-cfp-open-2611/">FOSSASIA PGDay 2027</a></strong><br>
March 9, 2027<br>
Bangkok, Thailand</p>
<p><strong><a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/">KubeCon + CloudNativeCon Europe 2027</a></strong><br>
March 15-18, 2027<br>
Barcelona, Spain</p>
<p><strong><a href="https://2027.nordicpgday.org/">Nordic PGDay 2027</a></strong><br>
March 16, 2027<br>
Courtyard Kungsholmen<br>
Stockholm, Sweden</p>
<p><strong><a href="https://www.socallinuxexpo.org/scale/24x">SCaLE 24x</a></strong><br>
April 1-4, 2027<br>
Pasadena Convention Center<br>
Pasadena, CA</p>
<p><strong><a href="https://duckdb.org/events/2027/04/07/duckcon8/">DuckCon #8</a></strong><br>
April 7, 2027<br>
De Wester (Westergas)<br>
Amsterdam, The Netherlands</p>
<p><strong><a href="https://2027.pgdayboston.org/">PgDay Boston 2027</a></strong><br>
April 10, 2027<br>
Tufts University Joyce Cummings Center<br>
Boston, MA</p>
<p><strong><a href="https://postgresconf.org/conferences/2027">Postgres Conference 2027</a></strong><br>
April 21-23, 2027<br>
San Jose Marriott<br>
San Jose, CA</p>
<p><strong><a href="https://perconalive.com/">Percona Live Bay Area 2027</a></strong><br>
May 11-13, 2027<br>
Computer History Museum<br>
Mountain View, CA</p>
<p><strong><a href="https://2027.pgconf.dev/">PGConf.dev 2027</a></strong><br>
May 11-14, 2027<br>
Plaza Centre-Ville<br>
Montréal, QC, Canada</p>]]></content:encoded>
    <pubDate>Fri, 13 Nov 2026 16:54:23 +0000</pubDate>
    <dc:creator>VillageSQL</dc:creator>
  </item>

  <item>
    <title>Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights</title>
    <guid isPermaLink="false">https://www.percona.com/?p=56188</guid>
    <link>https://www.percona.com/blog/just-because-you-can-doesnt-mean-you-should-re-evaluating-galera-node-weights/</link>
    <description>We all know that when using a cluster based on Galera (Codership) such as Percona XtraDB Cluster (PXC), it is recommended to deploy an odd number of nodes. 
The generic answer to this recommendation is that otherwise the risk of having an inconsistent cluster (like split-brain) is higher.  (see Get started with Percona XtraDB Cluster – Percona XtraDB Cluster ) 
As a reminder, when split-brain occurs, a cluster will stop accepting queries, which is what a safe solution should do while waiting for the DBA to review and decide which of the two nodes is the leader. I do not even want to describe the infamous wsrep_provider_options=”pc.ignore_sb = yes”, which has been the cause of many, many nights spent trying to reconcile the corrupted datasets. If you have it, remove it.
However, in Galera, we also have the setting wsrep_provider_options=”pc.weight=N”. With it, we can define a specific weight in the quorum calculation for each node. 
The question naturally arises: if we can assign a different weight to each node, why can’t we have an even number of nodes and assign them different weights? 
The answer to the question is simple, yes, we can. 
What is not simple is to identify who should have a different weight and how to distribute it.
To understand the problem, we need to review how the quorum is calculated (I have touched on the topic long ago here TusaCentral – Geographic replication and quorum Calculation in MySQL/Galera) and what a View is.
In Percona XtraDB Cluster (PXC) and Galera, a View (or cluster view) is the agreed-upon membership of the cluster at any specific moment in time. It represents the exact list of nodes that are currently connected, synchronized, and actively communicating with each other.
Whenever a node joins the cluster, leaves gracefully, or drops out due to a network failure, the remaining connected nodes negotiate a new View and assign it a unique View ID.
Its relationship to the Primary Component (PC) is based on quorum:

The Quorum Check: Every time a new View is formed, the nodes inside it calculate whether they hold the majority of the cluster’s total votes.
The Primary Component: If the new View holds the majority, it is designated as the Primary Component. Only the Primary Component is allowed to accept and process transactions.
Non-Primary Views: If a network split occurs, isolated nodes will form their own separate View. Because this smaller View lacks a majority, it becomes a non-primary component and immediately halts database operations to prevent split-brain data corruption.

Essentially, a View defines who is currently connected to each other, while the Primary Component is the specific View that has enough members to safely keep the database running.2026-11-10T13:45:01.498927Z 0 [Note] [MY-000000] [Galera] Node 49d8df08-bad6 state primary
2026-11-10T13:45:01.500967Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,49d8df08-bad6,93)
memb {
	49d8df08-bad6,2
	5ce1a9aa-8e06,2
	8458c8f0-9b8f,2
	}
joined {}
left {}
partitioned {	85a6e937-b8c4,2}
)The formula for calculating the quorum is:
  Σ p·w  &amp;lt;  2·(Σ m·w + Σ l·w)
Where:
p = members of the last PRIMARY view 
l = members that left gracefully (LEFT)
m = members of the component being evaluated
w = pc.weight (0–255).
In short, the total weight of the previous Primary view must be less than twice the weight of the members of the new component plus the weight of the members that left gracefully.  To be clear, a crashed / unreachable node (DOWN or in another partition) still counts in Σ p·w, so it pushes the threshold up. 
A graceful leave (LEFT), instead, is credited to the new component. Also, any node that was not in the last PC is dropped from calculation, so a node that has just joined or rejoined contributes nothing to this check.
Look for the function:bool gcomm::pc::Proto::have_quorum(const View&amp;amp; view, const View&amp;amp; pc_view) const
{
    // Compare only against members and left which were part of the pc_view.
    gcomm::NodeList memb_intersection(
        node_list_intersection(view.members(), pc_view.members()));
    gcomm::NodeList left_intersection(
        node_list_intersection(view.left(), pc_view.members()));
    if (have_weights(...))
        return (weighted_sum(memb_intersection, instances_) * 2
                + weighted_sum(left_intersection, instances_) &amp;gt;
                weighted_sum(pc_view.members(), instances_));
    else
        return (memb_intersection.size()*2 + left_intersection.size() &amp;gt;
                pc_view.members().size());
}(repo https://github.com/percona/galera/tree/a6b54d887d58bae43ae3f46f5696195329405b7e) 
Now let’s assume that we have a cluster of 4 nodes, there are many possible combinations where things will work and where instead they will hit us badly.

The most common is the split-brain scenario:
 
To prevent this, we may decide to use one node with different weight:

Here we go, problem solved… really?
No, because changing weights may cause other issues, for example if we calculate the weights wrongly with respect to the number of nodes:

In this case one single node could bring down the whole cluster. 
Of course, if we perform standard operations, or crashes happen in cascade and the cluster has time to rebuild the Primary Component, all will work well:


The problem is that:

Cluster dimensions are dynamic and change in relation to the active Primary Component
In environments managed by operators (e.g., Kubernetes), scaling the number of nodes up or down is even more dynamic and difficult to predict.
In real life we may have an incredible variety of scenarios, I have built just a few (see PDF)  

The real question is not if we can do it but whether it is worth it: does using an odd number of nodes help or not?
The answer is partly. 
Odd nodes get rid of the exact-tie case in the simple situation. They don’t completely stop the cluster losing PRIMARY state.
What matters is the parity of the total weight, not the node count. 
A tie happens when present = partitioned, which means the counted weight (Σw(PC) − Left) splits exactly in half. 
That is only possible when the counted weight is even. So:

5 nodes of w=1, any two-way split: 3|2 or 4|1. One side always strictly wins.
6 nodes with one node at w=2 (total 7) behave exactly like an odd cluster. Adding the Galera Arbitrator (garbd) as a 7th voter does the same thing.
3 nodes weighted 2,1,1 (total 4) is an odd node count with an even total, so it can tie again.

Where odd nodes still lose PRIMARY:

The PC shrinks after each failure, and its parity changes. 5 nodes lose 1 and stay PRIMARY, but the new PC has 4 nodes. A 2|2 split after that ties. A rolling failure still ends at 2 nodes and losing one of those ties.
Graceful leaves change the parity. If 1 of 5 shuts down cleanly, the counted weight is 4, so a 2|2 split ties.
Splits into three or more groups. A 2|2|1 split of 5 has no tie, but no group is bigger than the rest combined (2 &amp;lt; 3), so every group goes NON-PRIMARY.
Losing a majority is still losing. 5 nodes survive 2 failures but not 3, so odd doesn’t add fault tolerance. 5 and 6 nodes both tolerate only 2 failures. The 6th node adds cost and a tie risk, and no extra resilience.
Two data centers. With 3+2 nodes, the 3-node site survives a link cut, but if that site goes down, the 2-node site stops. To survive the loss of either site we need a third location, even if it only runs garbd.

In practice: aim for an odd total weight in the PC and use a third site or an arbitrator for multi-DC setups. Even then, a cluster that has already shrunk can become even again, so the last-two-nodes tie can’t be designed away. 
Only pc.ignore_sb or a manual bootstrap gets past it. But you do not want to go there.
Given this, and given that what counts is NOT the number of nodes but the balance in the weight calculation, is using weights the right way to compensate?

Well, this may be my personal opinion, but in my experience, weights make the cluster harder to manage, and the stability they add is narrow.
A weight-2 node acts like two nodes that always fail together. Four nodes weighted 2,1,1,1 behave like a 5-node cluster where two of the nodes share one box. 
We trade a split-brain tie for a single point of fate:

Gain: a clean 2|2 split now has a fixed winner, the side holding the heavy node.
Loss: losing the heavy node plus one other is fatal. Over-weighting makes it worse: with w=3, losing that node alone causes a tie.

The set of survivable failures shifts rather than grows. We still tolerate only one arbitrary failure plus some specific splits.
What weights add to day-to-day operations:

Nodes are no longer equal. Everyone must know which node is “special”. Runbooks, monitoring and on-call decisions depend on it.
Maintenance order matters. Shut the heavy node down cleanly so its weight counts for neither side. 
If it crashes during maintenance on another node, the cluster can stop. Patching or rebooting it may first need its weight moved to another node, a step that’s easy to forget.
Weight is a configuration that can drift. It has to be consistent across config files, rebuilds and node replacements. A rebuilt node that comes back with the default w=1 silently changes the quorum maths.
The tie-breaker site is fixed. With two data centers, the weighted side always wins. That’s no different from simply putting 3 nodes in one DC and 2 in the other, and in both cases losing that DC stops the cluster.
Shrinking still breaks it. After failures the PC changes, so a setup tuned for 4 nodes may tie at 3 or 2.

Simple to most complex for the same goal:

An odd number of equal nodes: symmetric, nothing special to manage.
An even number of nodes plus the Galera Arbitrator (garbd): the data nodes stay equal, and the arbitrator holds the tie-breaking vote. If it runs at a third site, the cluster survives losing either main site, which weights alone can’t give you.
An even number of nodes with weights: use this only when you can’t add a node or an arbitrator and you deliberately want one side to always win a split.

Weights don’t make the cluster generally more stable, and they add operational complexity. They make sense as a deliberate tiebreaker, not as a fix for even node counts.
Conclusions
I stand by the evergreen KISS motto: Keep It Simple and Straightforward.
Having a PXC cluster with more than 3 nodes makes sense in very few cases, and only to scale reads or if you plan to kill 2 nodes at the same time on a cluster of 5.
Given the significant complications the increase of nodes or the use of weight brings, I do not really see why someone in a healthy state of mind should look for trouble.
Just because it can be done, it doesn’t mean “do it”, especially if the gain is minimal and the complexity increases. 
Do yourself a favor: keep your cluster to the minimum, to what is really needed and do not try to stretch it. You will save yourself from spending miserable nights doing cluster recovery or worse data reconciliation.
Great MySQL to everyone.
The post Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights appeared first on Percona.</description>
    <content:encoded><![CDATA[<p><span>We all know that when using a cluster based on Galera (Codership) such as Percona XtraDB Cluster (PXC), it is recommended to deploy an odd number of nodes. <a href="https://www.percona.com/wp-content/uploads/2026/11/goat_small.jpg"><img loading="lazy" decoding="async" class="alignright wp-image-56186 " src="https://www.percona.com/wp-content/uploads/2026/11/goat_small.jpg" alt="" width="401" height="269" srcset="https://www.percona.com/wp-content/uploads/2026/11/goat_small.jpg 600w, https://www.percona.com/wp-content/uploads/2026/11/goat_small-300x201.jpg 300w" sizes="auto, (max-width: 401px) 100vw, 401px"></a></span></p>
<p><span>The generic answer to this recommendation is that otherwise the risk of having an inconsistent cluster (like split-brain) is higher.  (see </span><a href="https://docs.percona.com/percona-xtradb-cluster/8.4/get-started-cluster.html"><span>Get started with Percona XtraDB Cluster – Percona XtraDB Cluster</span></a><span> ) </span></p>
<p><span>As a reminder, when split-brain occurs, a cluster will stop accepting queries, which is what a safe solution should do while waiting for the DBA to review and decide which of the two nodes is the leader. I do not even want to describe the infamous wsrep_provider_options=”pc.ignore_sb = yes”, which has been the cause of many, many nights spent trying to reconcile the corrupted datasets. If you have it, remove it.</span></p>
<p><span>However, in Galera, we also have the setting wsrep_provider_options=”pc.weight=N”. With it, we can define a specific weight in the quorum calculation for each node. </span><span><br>
</span><span>The question naturally arises: </span><i><span>if we can assign a different weight to each node, why can’t we have an even number of nodes and assign them different weights?</span></i><span> </span></p>
<p><span>The answer to the question is simple, yes, we can. </span></p>
<p><span>What is not simple is to identify who should have a different weight and how to distribute it.</span></p>
<p><span>To understand the problem, we need to review how the quorum is calculated (I have touched on the topic long ago here </span><a href="https://www.tusacentral.net/index.php/mysql-blogs/170-geographic-replication-and-quorum-calculation-in-mysqlgalera"><span>TusaCentral – Geographic replication and quorum Calculation in MySQL/Galera</span></a><span>) and what a View is.</span></p>
<p><span>In Percona XtraDB Cluster (PXC) and Galera, a </span><b>View</b><span> (or cluster view) is the agreed-upon membership of the cluster at any specific moment in time. It represents the exact list of nodes that are currently connected, synchronized, and actively communicating with each other.</span></p>
<p><span>Whenever a node joins the cluster, leaves gracefully, or </span><b>drops out due to a network failure</b><span>, the remaining connected nodes negotiate a new View and assign it a unique View ID.</span></p>
<p><span>Its relationship to the </span><b>Primary Component (PC)</b><span> is based on quorum:</span></p>
<ul>
<li aria-level="1"><b>The Quorum Check:</b><span> Every time a new View is formed, the nodes inside it calculate whether they hold the majority of the cluster’s total votes.</span></li>
<li aria-level="1"><b>The Primary Component:</b><span> If the new View holds the majority, it is designated as the </span><b>Primary Component</b><span>. Only the Primary Component is allowed to accept and process transactions.</span></li>
<li aria-level="1"><b>Non-Primary Views:</b><span> If a network split occurs, isolated nodes will form their own separate View. Because this smaller View lacks a majority, it becomes a non-primary component and immediately halts database operations to prevent split-brain data corruption.</span></li>
</ul>
<p><span>Essentially, a View defines </span><i><span>who</span></i><span> is currently connected to each other, while the Primary Component is the specific View that has enough members to safely keep the database running.</span></p><pre class="urvanov-syntax-highlighter-plain-tag">2026-11-10T13:45:01.498927Z 0 [Note] [MY-000000] [Galera] Node 49d8df08-bad6 state primary
2026-11-10T13:45:01.500967Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,49d8df08-bad6,93)
memb {
	49d8df08-bad6,2
	5ce1a9aa-8e06,2
	8458c8f0-9b8f,2
	}
joined {}
left {}
partitioned {	85a6e937-b8c4,2}
)</pre><p><span>The formula for calculating the quorum is:</span></p>
<p><span>  Σ p·w  &lt;  2·(Σ m·w + Σ l·w)</span></p>
<p><span>Where:</span><span><br>
</span><span>p = members of the last PRIMARY view </span><span><br>
</span><span>l = members that left gracefully (LEFT)</span><span><br>
</span><span>m = members of the component being evaluated</span><span><br>
</span><span>w = pc.weight (0–255).</span></p>
<p><span>In short, the total weight of the previous Primary view must be less than twice the weight of the members of the new component plus the weight of the members that left gracefully.  To be clear, a crashed / unreachable node (DOWN or in another partition) still counts in Σ p·w, so it pushes the threshold up. </span><span><br>
</span><span>A graceful leave (LEFT), instead, is credited to the new component. Also, any node that was not in the last PC is dropped from calculation, so a node that has just joined or rejoined contributes nothing to this check.</span></p>
<p><span>Look for the function:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">bool gcomm::pc::Proto::have_quorum(const View&amp; view, const View&amp; pc_view) const
{
    // Compare only against members and left which were part of the pc_view.
    gcomm::NodeList memb_intersection(
        node_list_intersection(view.members(), pc_view.members()));
    gcomm::NodeList left_intersection(
        node_list_intersection(view.left(), pc_view.members()));
    if (have_weights(...))
        return (weighted_sum(memb_intersection, instances_) * 2
                + weighted_sum(left_intersection, instances_) &gt;
                weighted_sum(pc_view.members(), instances_));
    else
        return (memb_intersection.size()*2 + left_intersection.size() &gt;
                pc_view.members().size());
}</pre><p><span>(repo </span><a href="https://github.com/percona/galera/tree/a6b54d887d58bae43ae3f46f5696195329405b7e"><span>https://github.com/percona/galera/tree/a6b54d887d58bae43ae3f46f5696195329405b7e</span></a><span>) </span></p>
<p><span>Now let’s assume that we have a cluster of 4 nodes, there are many possible combinations where things will work and where instead they will hit us badly.</span></p>
<p><a href="https://www.percona.com/wp-content/uploads/2026/11/legend.jpg"><img loading="lazy" decoding="async" class="alignleft size-full wp-image-56187" src="https://www.percona.com/wp-content/uploads/2026/11/legend.jpg" alt="" width="1352" height="76" srcset="https://www.percona.com/wp-content/uploads/2026/11/legend.jpg 1352w, https://www.percona.com/wp-content/uploads/2026/11/legend-300x17.jpg 300w, https://www.percona.com/wp-content/uploads/2026/11/legend-1024x58.jpg 1024w, https://www.percona.com/wp-content/uploads/2026/11/legend-768x43.jpg 768w" sizes="auto, (max-width: 1352px) 100vw, 1352px"></a></p>
<p><span>The most common is the split-brain scenario:</span></p>
<p><span> <a href="https://www.percona.com/wp-content/uploads/2026/11/1_split_brain_same_weigh.jpg"><img loading="lazy" decoding="async" class="size-full wp-image-56182 alignnone" src="https://www.percona.com/wp-content/uploads/2026/11/1_split_brain_same_weigh.jpg" alt="" width="462" height="345" srcset="https://www.percona.com/wp-content/uploads/2026/11/1_split_brain_same_weigh.jpg 462w, https://www.percona.com/wp-content/uploads/2026/11/1_split_brain_same_weigh-300x224.jpg 300w" sizes="auto, (max-width: 462px) 100vw, 462px"></a></span></p>
<p><span>To prevent this, we may decide to use one node with different weight:</span></p>
<p><a href="https://www.percona.com/wp-content/uploads/2026/11/1_split_brain_diff_weigh.jpg"><img loading="lazy" decoding="async" class="size-full wp-image-56181 alignnone" src="https://www.percona.com/wp-content/uploads/2026/11/1_split_brain_diff_weigh.jpg" alt="" width="463" height="348" srcset="https://www.percona.com/wp-content/uploads/2026/11/1_split_brain_diff_weigh.jpg 463w, https://www.percona.com/wp-content/uploads/2026/11/1_split_brain_diff_weigh-300x225.jpg 300w" sizes="auto, (max-width: 463px) 100vw, 463px"></a></p>
<p><span>Here we go, problem solved… really?</span></p>
<p><span>No, because changing weights may cause other issues, for example if we calculate the weights wrongly with respect to the number of nodes:</span></p>
<p><a href="https://www.percona.com/wp-content/uploads/2026/11/3_unbalnced_weight.jpg"><img loading="lazy" decoding="async" class="size-full wp-image-56185 alignnone" src="https://www.percona.com/wp-content/uploads/2026/11/3_unbalnced_weight.jpg" alt="" width="461" height="345" srcset="https://www.percona.com/wp-content/uploads/2026/11/3_unbalnced_weight.jpg 461w, https://www.percona.com/wp-content/uploads/2026/11/3_unbalnced_weight-300x225.jpg 300w" sizes="auto, (max-width: 461px) 100vw, 461px"></a></p>
<p><span>In this case one single node could bring down the whole cluster. </span></p>
<p><span>Of course, if we perform standard operations, or crashes happen in cascade and the cluster has time to rebuild the Primary Component, all will work well:</span></p>
<p><a href="https://www.percona.com/wp-content/uploads/2026/11/2_crash_on_cascade.jpg"><img loading="lazy" decoding="async" class="size-full wp-image-56184 alignnone" src="https://www.percona.com/wp-content/uploads/2026/11/2_crash_on_cascade.jpg" alt="" width="462" height="347" srcset="https://www.percona.com/wp-content/uploads/2026/11/2_crash_on_cascade.jpg 462w, https://www.percona.com/wp-content/uploads/2026/11/2_crash_on_cascade-300x225.jpg 300w" sizes="auto, (max-width: 462px) 100vw, 462px"></a></p>
<p><a href="https://www.percona.com/wp-content/uploads/2026/11/2_crash_on_cascade_b.jpg"><img loading="lazy" decoding="async" class="size-full wp-image-56183 alignnone" src="https://www.percona.com/wp-content/uploads/2026/11/2_crash_on_cascade_b.jpg" alt="" width="463" height="347" srcset="https://www.percona.com/wp-content/uploads/2026/11/2_crash_on_cascade_b.jpg 463w, https://www.percona.com/wp-content/uploads/2026/11/2_crash_on_cascade_b-300x225.jpg 300w" sizes="auto, (max-width: 463px) 100vw, 463px"></a></p>
<p><span>The problem is that:</span></p>
<ul>
<li aria-level="1"><span>Cluster dimensions are dynamic and change in relation to the active Primary Component</span></li>
<li aria-level="1"><span>In environments managed by operators (e.g., Kubernetes), scaling the number of nodes up or down is even more dynamic and difficult to predict.</span></li>
<li aria-level="1"><span>In real life we may have an incredible variety of scenarios, I have built just a few (see </span><a href="https://github.com/Tusamarco/blogs/blob/master/quorum_calculations/quorum_calculation.pdf">PDF</a><span>)  </span></li>
</ul>
<p><span>The real question is not if we can do it but whether it is worth it: does using an odd number of nodes help or not?</span></p>
<p><span>The answer is partly. </span></p>
<p><span>Odd nodes get rid of the exact-tie case in the simple situation. They don’t completely stop the cluster losing PRIMARY state.</span></p>
<p><b>What matters is the parity of the total weight, not the node count.</b> <span><br>
</span><span>A tie happens when present = partitioned, which means the counted weight (Σw(PC) − Left) splits exactly in half. </span></p>
<p><span>That is only possible when the counted weight is even. So:</span></p>
<ul>
<li aria-level="1"><b>5 nodes of w=1, any two-way split:</b><span> 3|2 or 4|1. One side always strictly wins.</span></li>
<li aria-level="1"><b>6 nodes with one node at w=2 (total 7)</b><span> behave exactly like an odd cluster. Adding the Galera Arbitrator (garbd) as a 7th voter does the same thing.</span></li>
<li aria-level="1"><b>3 nodes weighted 2,1,1 (total 4)</b><span> is an odd node count with an even total, so it can tie again.</span></li>
</ul>
<p><span>Where odd nodes still lose PRIMARY:</span></p>
<ol>
<li aria-level="1"><b>The PC shrinks after each failure, and its parity changes.</b><span> 5 nodes lose 1 and stay PRIMARY, but the new PC has 4 nodes. A 2|2 split after that ties. A rolling failure still ends at 2 nodes and losing one of those ties.</span></li>
<li aria-level="1"><b>Graceful leaves change the parity.</b><span> If 1 of 5 shuts down cleanly, the counted weight is 4, so a 2|2 split ties.</span></li>
<li aria-level="1"><b>Splits into three or more groups.</b><span> A 2|2|1 split of 5 has no tie, but no group is bigger than the rest combined (2 &lt; 3), so every group goes NON-PRIMARY.</span></li>
<li aria-level="1"><b>Losing a majority is still losing.</b><span> 5 nodes survive 2 failures but not 3, so odd doesn’t add fault tolerance. 5 and 6 nodes both tolerate only 2 failures. The 6th node adds cost and a tie risk, and no extra resilience.</span></li>
<li aria-level="1"><b>Two data centers.</b><span> With 3+2 nodes, the 3-node site survives a link cut, but if that site goes down, the 2-node site stops. To survive the loss of either site we need a third location, even if it only runs garbd.</span></li>
</ol>
<p><b>In practice:</b><span> aim for an odd total weight in the PC and use a third site or an arbitrator for multi-DC setups. Even then, a cluster that has already shrunk can become even again, so the last-two-nodes tie can’t be designed away. </span></p>
<p><span>Only pc.ignore_sb or a manual bootstrap gets past it. </span><b>But you do not want to go there</b><span>.</span></p>
<p><span>Given this, and given that what counts is NOT the number of nodes but the balance in the weight calculation, is using weights the right way to compensate?</span></p>
<p><span><br>
</span><span>Well, this may be my personal opinion, but in my experience, weights make the cluster harder to manage, and the stability they add is narrow.</span></p>
<p><b>A weight-2 node acts like two nodes that always fail together.</b><span> Four nodes weighted 2,1,1,1 behave like a 5-node cluster where two of the nodes share one box. </span><span><br>
</span><span>We trade a split-brain tie for a single point of fate:</span></p>
<ul>
<li aria-level="1"><b>Gain:</b><span> a clean 2|2 split now has a fixed winner, the side holding the heavy node.</span></li>
<li aria-level="1"><b>Loss:</b><span> losing the heavy node plus one other is fatal. Over-weighting makes it worse: with w=3, losing that node alone causes a tie.</span></li>
</ul>
<p><span>The set of survivable failures shifts rather than grows. We still tolerate only one arbitrary failure plus some specific splits.</span></p>
<p><span>What weights add to day-to-day operations:</span></p>
<ol>
<li aria-level="1"><b>Nodes are no longer equal.</b><span> Everyone must know which node is “special”. Runbooks, monitoring and on-call decisions depend on it.</span></li>
<li aria-level="1"><b>Maintenance order matters.</b><span> Shut the heavy node down cleanly so its weight counts for neither side. </span><span><br>
</span><span>If it crashes during maintenance on another node, the cluster can stop. Patching or rebooting it may first need its weight moved to another node, a step that’s easy to forget.</span></li>
<li aria-level="1"><b>Weight is a configuration that can drift.</b><span> It has to be consistent across config files, rebuilds and node replacements. A rebuilt node that comes back with the default w=1 silently changes the quorum maths.</span></li>
<li aria-level="1"><b>The tie-breaker site is fixed.</b><span> With two data centers, the weighted side always wins. That’s no different from simply putting 3 nodes in one DC and 2 in the other, and in both cases losing that DC stops the cluster.</span></li>
<li aria-level="1"><b>Shrinking still breaks it.</b><span> After failures the PC changes, so a setup tuned for 4 nodes may tie at 3 or 2.</span></li>
</ol>
<p><span>Simple to most complex for the same goal:</span></p>
<ol>
<li aria-level="1"><b>An odd number of equal nodes:</b><span> symmetric, nothing special to manage.</span></li>
<li aria-level="1"><b>An even number of nodes plus the Galera Arbitrator (garbd):</b><span> the data nodes stay equal, and the arbitrator holds the tie-breaking vote. If it runs at a third site, the cluster survives losing either main site, which weights alone can’t give you.</span></li>
<li aria-level="1"><b>An even number of nodes with weights:</b><span> use this only when you can’t add a node or an arbitrator and you deliberately want one side to always win a split.</span></li>
</ol>
<p><span>Weights don’t make the cluster generally more stable, and they add operational complexity. They make sense as a deliberate tiebreaker, not as a fix for even node counts.</span></p>
<h2><span>Conclusions</span></h2>
<p><span>I stand by the evergreen KISS motto: Keep It Simple and Straightforward.<br>
</span><span>Having a PXC cluster with more than 3 nodes makes sense in very few cases, and only to scale reads or if you plan to kill 2 nodes at the same time on a cluster of 5.<br>
Given the significant complications the increase of nodes or the use of weight brings, I do not really see why someone in a healthy state of mind should look for trouble.</span></p>
<p><span>Just because it can be done, it doesn’t mean “do it”, especially if the gain is minimal and the complexity increases. </span></p>
<p><span>Do yourself a favor: keep your cluster to the minimum, to what is really needed and do not try to stretch it. You will save yourself from spending miserable nights doing cluster recovery or worse data reconciliation.</span></p>
<p><span>Great MySQL to everyone.</span></p>
<p>The post <a href="https://www.percona.com/blog/just-because-you-can-doesnt-mean-you-should-re-evaluating-galera-node-weights/">Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights</a> appeared first on <a href="https://www.percona.com/">Percona</a>.</p>]]></content:encoded>
    <pubDate>Fri, 13 Nov 2026 14:01:47 +0000</pubDate>
    <dc:creator>Marco Tusa</dc:creator>
    <category>Best Practices</category>
    <category>MySQL</category>
    <category>Open Source</category>
    <category>Uncategorized</category>
    <category>XtraDB Cluster (PXC)</category>
    <category>Cluster Management</category>
    <category>Database Administration (DBA)</category>
    <category>Fault Tolerance</category>
    <category>Galera Arbitrator (garbd)</category>
    <category>Galera Cluster</category>
    <category>High Availability (HA)</category>
    <category>Node Weighting (pc.weight)</category>
    <category>Percona XtraDB Cluster (PXC)</category>
    <category>Pr</category>
  </item>

  <item>
    <title>Just Because You Can, Doesn't Mean You Should: Re-evaluating Galera Node Weights</title>
    <guid isPermaLink="false">http://www.tusacentral.com/joomla/index.php/mysql-blogs/274-just-because-you-can-doesnt-mean-you-should-re-evaluating-galera-node-weights</guid>
    <link>http://www.tusacentral.com/joomla/index.php/mysql-blogs/274-just-because-you-can-doesnt-mean-you-should-re-evaluating-galera-node-weights</link>
    <description>We all know that when using a cluster based on Galera (Codership) such as Percona XtraDB Cluster (PXC), it is recommended to deploy an odd number of nodes. 
The generic answer to this recommendation is that otherwise the risk of having an inconsistent cluster (like split-brain) is higher.  (see Get started with Percona XtraDB Cluster - Percona XtraDB Cluster ) 
As a reminder, when split-brain occurs, a cluster will stop accepting queries, which is what a safe solution should do while waiting for the DBA to review and decide which of the two nodes is the leader. I do not even want to describe the infamous wsrep_provider_options=&quot;pc.ignore_sb = yes&quot;, which has been the cause of many, many nights spent trying to reconcile the corrupted datasets. If you have it, remove it.
However, in Galera, we also have the setting wsrep_provider_options=&quot;pc.weight=N&quot;. With it, we can define a specific weight in the quorum calculation for each node.  The question naturally arises: if we can assign a different weight to each node, why can't we have an even number of nodes and assign them different weights?
The answer to the question is simple, yes, we can. 
What is not simple is to identify who should have a different weight and how to distribute it.
To understand the problem, we need to review how the quorum is calculated (I have touched on the topic long ago here TusaCentral - Geographic replication and quorum Calculation in MySQL/Galera) and what a View is.
In Percona XtraDB Cluster (PXC) and Galera, a View (or cluster view) is the agreed-upon membership of the cluster at any specific moment in time. It represents the exact list of nodes that are currently connected, synchronized, and actively communicating with each other.
Whenever a node joins the cluster, leaves gracefully, or drops out due to a network failure, the remaining connected nodes negotiate a new View and assign it a unique View ID.
Its relationship to the Primary Component (PC) is based on quorum:

The Quorum Check: Every time a new View is formed, the nodes inside it calculate whether they hold the majority of the cluster's total votes.
The Primary Component: If the new View holds the majority, it is designated as the Primary Component. Only the Primary Component is allowed to accept and process transactions.
Non-Primary Views: If a network split occurs, isolated nodes will form their own separate View. Because this smaller View lacks a majority, it becomes a non-primary component and immediately halts database operations to prevent split-brain data corruption.

Essentially, a View defines who is currently connected to each other, while the Primary Component is the specific View that has enough members to safely keep the database running.
2026-11-10T13:45:01.498927Z 0 [Note] [MY-000000] [Galera] Node 49d8df08-bad6 state primary
2026-11-10T13:45:01.500967Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,49d8df08-bad6,93)
memb {
	49d8df08-bad6,2
	5ce1a9aa-8e06,2
	8458c8f0-9b8f,2
	}
joined {}
left {}
partitioned {	85a6e937-b8c4,2}
)

The formula for calculating the quorum is:
  Σ p·w  &amp;lt;  2·(Σ m·w + Σ l·w)
Where: p = members of the last PRIMARY view  l = members that left gracefully (LEFT) m = members of the component being evaluated w = pc.weight (0–255).
In short, the total weight of the previous Primary view must be less than twice the weight of the members of the new component plus the weight of the members that left gracefully.  To be clear, a crashed / unreachable node (DOWN or in another partition) still counts in Σ p·w, so it pushes the threshold up.  A graceful leave (LEFT), instead, is credited to the new component. Also, any node that was not in the last PC is dropped from calculation, so a node that has just joined or rejoined contributes nothing to this check.
Look for the function:
bool gcomm::pc::Proto::have_quorum(const View&amp;amp; view, const View&amp;amp; pc_view) const
{
    // Compare only against members and left which were part of the pc_view.
    gcomm::NodeList memb_intersection(
        node_list_intersection(view.members(), pc_view.members()));
    gcomm::NodeList left_intersection(
        node_list_intersection(view.left(), pc_view.members()));
    if (have_weights(...))
        return (weighted_sum(memb_intersection, instances_) * 2
                + weighted_sum(left_intersection, instances_) &amp;gt;
                weighted_sum(pc_view.members(), instances_));
    else
        return (memb_intersection.size()*2 + left_intersection.size() &amp;gt;
                pc_view.members().size());
}

(repo https://github.com/percona/galera/tree/a6b54d887d58bae43ae3f46f5696195329405b7e) 
Now let's assume that we have a cluster of 4 nodes, there are many possible combinations where things will work and where instead they will hit us badly.

The most common is the split-brain scenario:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
To prevent this, we may decide to use one node with different weight:

 
 
 
 
 
 
 
 
 
 
 
 
 
Here we go, problem solved… really?
No, because changing weights may cause other issues, for example if we calculate the weights wrongly with respect to the number of nodes:

 
 
 
 
 
 
 
 
 
 
 
 
 
In this case one single node could bring down the whole cluster. 
Of course, if we perform standard operations, or crashes happen in cascade and the cluster has time to rebuild the Primary Component, all will work well:

 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
The problem is that:

Cluster dimensions are dynamic and change in relation to the active Primary Component
In environments managed by operators (e.g., Kubernetes), scaling the number of nodes up or down is even more dynamic and difficult to predict.
In real life we may have an incredible variety of scenarios, I have built just a few (see PDF)  

The real question is not if we can do it but whether it is worth it: does using an odd number of nodes help or not?
The answer is partly. 
Odd nodes get rid of the exact-tie case in the simple situation. They don't completely stop the cluster losing PRIMARY state.
What matters is the parity of the total weight, not the node count.
 A tie happens when present = partitioned, which means the counted weight (Σw(PC) − Left) splits exactly in half. 
That is only possible when the counted weight is even. So:

5 nodes of w=1, any two-way split: 3|2 or 4|1. One side always strictly wins.
6 nodes with one node at w=2 (total 7) behave exactly like an odd cluster. Adding the Galera Arbitrator (garbd) as a 7th voter does the same thing.
3 nodes weighted 2,1,1 (total 4) is an odd node count with an even total, so it can tie again.

Where odd nodes still lose PRIMARY:

The PC shrinks after each failure, and its parity changes. 5 nodes lose 1 and stay PRIMARY, but the new PC has 4 nodes. A 2|2 split after that ties. A rolling failure still ends at 2 nodes and losing one of those ties.
Graceful leaves change the parity. If 1 of 5 shuts down cleanly, the counted weight is 4, so a 2|2 split ties.
Splits into three or more groups. A 2|2|1 split of 5 has no tie, but no group is bigger than the rest combined (2 &amp;lt; 3), so every group goes NON-PRIMARY.
Losing a majority is still losing. 5 nodes survive 2 failures but not 3, so odd doesn't add fault tolerance. 5 and 6 nodes both tolerate only 2 failures. The 6th node adds cost and a tie risk, and no extra resilience.
Two data centers. With 3+2 nodes, the 3-node site survives a link cut, but if that site goes down, the 2-node site stops. To survive the loss of either site we need a third location, even if it only runs garbd.

In practice: aim for an odd total weight in the PC and use a third site or an arbitrator for multi-DC setups. Even then, a cluster that has already shrunk can become even again, so the last-two-nodes tie can't be designed away. 
Only pc.ignore_sb or a manual bootstrap gets past it. But you do not want to go there.
Given this, and given that what counts is NOT the number of nodes but the balance in the weight calculation, is using weights the right way to compensate?
 Well, this may be my personal opinion, but in my experience, weights make the cluster harder to manage, and the stability they add is narrow.
A weight-2 node acts like two nodes that always fail together. Four nodes weighted 2,1,1,1 behave like a 5-node cluster where two of the nodes share one box.  We trade a split-brain tie for a single point of fate:

Gain: a clean 2|2 split now has a fixed winner, the side holding the heavy node.
Loss: losing the heavy node plus one other is fatal. Over-weighting makes it worse: with w=3, losing that node alone causes a tie.

The set of survivable failures shifts rather than grows. We still tolerate only one arbitrary failure plus some specific splits.
What weights add to day-to-day operations:

Nodes are no longer equal. Everyone must know which node is &quot;special&quot;. Runbooks, monitoring and on-call decisions depend on it.
Maintenance order matters. Shut the heavy node down cleanly so its weight counts for neither side.  If it crashes during maintenance on another node, the cluster can stop. Patching or rebooting it may first need its weight moved to another node, a step that's easy to forget.
Weight is a configuration that can drift. It has to be consistent across config files, rebuilds and node replacements. A rebuilt node that comes back with the default w=1 silently changes the quorum maths.
The tie-breaker site is fixed. With two data centers, the weighted side always wins. That's no different from simply putting 3 nodes in one DC and 2 in the other, and in both cases losing that DC stops the cluster.
Shrinking still breaks it. After failures the PC changes, so a setup tuned for 4 nodes may tie at 3 or 2.

Simple to most complex for the same goal:

An odd number of equal nodes: symmetric, nothing special to manage.
An even number of nodes plus the Galera Arbitrator (garbd): the data nodes stay equal, and the arbitrator holds the tie-breaking vote. If it runs at a third site, the cluster survives losing either main site, which weights alone can't give you.
An even number of nodes with weights: use this only when you can't add a node or an arbitrator and you deliberately want one side to always win a split.

Weights don't make the cluster generally more stable, and they add operational complexity. They make sense as a deliberate tiebreaker, not as a fix for even node counts.
Conclusions
I stand by the evergreen KISS motto: Keep It Simple and Straightforward. Having a PXC cluster with more than 3 nodes makes sense in very few cases, and only to scale reads or if you plan to kill 2 nodes at the same time on a cluster of 5. Given the significant complications the increase of nodes or the use of weight brings, I do not really see why someone in a healthy state of mind should look for trouble.
Just because it can be done, it doesn’t mean “do it”, especially if the gain is minimal and the complexity increases. 
Do yourself a favor: keep your cluster to the minimum, to what is really needed and do not try to stretch it. You will save yourself from spending miserable nights doing cluster recovery or worse data reconciliation.
Great MySQL to everyone.</description>
    <content:encoded><![CDATA[<p><span>We all know that when using a cluster based on Galera (Codership) such as Percona XtraDB Cluster (PXC), it is recommended to deploy an odd number of nodes. <a href="https://www.percona.com/wp-content/uploads/2026/11/goat_small.jpg"><img src="https://www.percona.com/wp-content/uploads/2026/11/goat_small.jpg" alt="" width="401" height="269" class="alignright wp-image-56186"></a></span></p>
<p><span>The generic answer to this recommendation is that otherwise the risk of having an inconsistent cluster (like split-brain) is higher.  (see </span><a href="https://docs.percona.com/percona-xtradb-cluster/8.4/get-started-cluster.html"><span>Get started with Percona XtraDB Cluster - Percona XtraDB Cluster</span></a><span> ) </span></p>
<p><span>As a reminder, when split-brain occurs, a cluster will stop accepting queries, which is what a safe solution should do while waiting for the DBA to review and decide which of the two nodes is the leader. I do not even want to describe the infamous wsrep_provider_options="pc.ignore_sb = yes", which has been the cause of many, many nights spent trying to reconcile the corrupted datasets. If you have it, remove it.</span></p>
<p><span>However, in Galera, we also have the setting wsrep_provider_options="pc.weight=N". With it, we can define a specific weight in the quorum calculation for each node. </span><span> </span><span>The question naturally arises: </span><i><span>if we can assign a different weight to each node, why can't we have an even number of nodes and assign them different weights?</span></i><span></span></p>
<p><span>The answer to the question is simple, yes, we can. </span></p>
<p><span>What is not simple is to identify who should have a different weight and how to distribute it.</span></p>
<p><span>To understand the problem, we need to review how the quorum is calculated (I have touched on the topic long ago here </span><a href="http://www.tusacentral.com/joomla/index.php/mysql-blogs/170-geographic-replication-and-quorum-calculation-in-mysqlgalera"><span>TusaCentral - Geographic replication and quorum Calculation in MySQL/Galera</span></a><span>) and what a View is.</span></p>
<p><span>In Percona XtraDB Cluster (PXC) and Galera, a </span><b>View</b><span> (or cluster view) is the agreed-upon membership of the cluster at any specific moment in time. It represents the exact list of nodes that are currently connected, synchronized, and actively communicating with each other.</span></p>
<p><span>Whenever a node joins the cluster, leaves gracefully, or </span><b>drops out due to a network failure</b><span>, the remaining connected nodes negotiate a new View and assign it a unique View ID.</span></p>
<p><span>Its relationship to the </span><b>Primary Component (PC)</b><span> is based on quorum:</span></p>
<ul>
<li aria-level="1"><b>The Quorum Check:</b><span> Every time a new View is formed, the nodes inside it calculate whether they hold the majority of the cluster's total votes.</span></li>
<li aria-level="1"><b>The Primary Component:</b><span> If the new View holds the majority, it is designated as the </span><b>Primary Component</b><span>. Only the Primary Component is allowed to accept and process transactions.</span></li>
<li aria-level="1"><b>Non-Primary Views:</b><span> If a network split occurs, isolated nodes will form their own separate View. Because this smaller View lacks a majority, it becomes a non-primary component and immediately halts database operations to prevent split-brain data corruption.</span></li>
</ul>
<p><span>Essentially, a View defines </span><i><span>who</span></i><span> is currently connected to each other, while the Primary Component is the specific View that has enough members to safely keep the database running.</span></p>
<pre class="lang:default decode:true">2026-11-10T13:45:01.498927Z 0 [Note] [MY-000000] [Galera] Node 49d8df08-bad6 state primary
2026-11-10T13:45:01.500967Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,49d8df08-bad6,93)
memb {
	49d8df08-bad6,2
	5ce1a9aa-8e06,2
	8458c8f0-9b8f,2
	}
joined {}
left {}
partitioned {	85a6e937-b8c4,2}
)
</pre>
<p><span>The formula for calculating the quorum is:</span></p>
<p><span>  Σ p·w  &lt;  2·(Σ m·w + Σ l·w)</span></p>
<p><span>Where:</span><span> </span><span>p = members of the last PRIMARY view </span><span> </span><span>l = members that left gracefully (LEFT)</span><span> </span><span>m = members of the component being evaluated</span><span> </span><span>w = pc.weight (0–255).</span></p>
<p><span>In short, the total weight of the previous Primary view must be less than twice the weight of the members of the new component plus the weight of the members that left gracefully.  To be clear, a crashed / unreachable node (DOWN or in another partition) still counts in Σ p·w, so it pushes the threshold up. </span><span> </span><span>A graceful leave (LEFT), instead, is credited to the new component. Also, any node that was not in the last PC is dropped from calculation, so a node that has just joined or rejoined contributes nothing to this check.</span></p>
<p><span>Look for the function:</span></p>
<pre class="lang:c decode:true">bool gcomm::pc::Proto::have_quorum(const View&amp; view, const View&amp; pc_view) const
{
    // Compare only against members and left which were part of the pc_view.
    gcomm::NodeList memb_intersection(
        node_list_intersection(view.members(), pc_view.members()));
    gcomm::NodeList left_intersection(
        node_list_intersection(view.left(), pc_view.members()));
    if (have_weights(...))
        return (weighted_sum(memb_intersection, instances_) * 2
                + weighted_sum(left_intersection, instances_) &gt;
                weighted_sum(pc_view.members(), instances_));
    else
        return (memb_intersection.size()*2 + left_intersection.size() &gt;
                pc_view.members().size());
}
</pre>
<p><span>(repo </span><a href="https://github.com/percona/galera/tree/a6b54d887d58bae43ae3f46f5696195329405b7e"><span>https://github.com/percona/galera/tree/a6b54d887d58bae43ae3f46f5696195329405b7e</span></a><span>) </span></p>
<p><span>Now let's assume that we have a cluster of 4 nodes, there are many possible combinations where things will work and where instead they will hit us badly.</span></p>
<p><a href="http://www.tusacentral.com/joomla/images/stories/PXC_generic/legend.jpg"><img src="http://www.tusacentral.com/joomla/images/stories/PXC_generic/legend.jpg" alt="legend" width="800" height="45" class="alignleft size-full wp-image-56187"></a></p>
<p><span>The most common is the split-brain scenario:</span></p>
<p><span> <a href="http://www.tusacentral.com/joomla/images/stories/PXC_generic/1_split_brain_same_weigh.jpg"><img src="http://www.tusacentral.com/joomla/images/stories/PXC_generic/1_split_brain_same_weigh.jpg" alt="1 split brain same weigh" width="462" height="345" class="size-full wp-image-56182 alignnone"><br></a></span></p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p><span>To prevent this, we may decide to use one node with different weight:</span></p>
<p><a href="http://www.tusacentral.com/joomla/images/stories/PXC_generic/1_split_brain_diff_weigh.jpg"><img src="http://www.tusacentral.com/joomla/images/stories/PXC_generic/1_split_brain_diff_weigh.jpg" alt="1 split brain diff weigh" width="463" height="348" class="size-full wp-image-56181 alignnone"><br></a></p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p><span>Here we go, problem solved… really?</span></p>
<p><span>No, because changing weights may cause other issues, for example if we calculate the weights wrongly with respect to the number of nodes:</span></p>
<p><a href="http://www.tusacentral.com/joomla/images/stories/PXC_generic/3_unbalnced_weight.jpg"><img src="http://www.tusacentral.com/joomla/images/stories/PXC_generic/3_unbalnced_weight.jpg" alt="3 unbalnced weight" width="461" height="345" class="size-full wp-image-56185 alignnone"><br></a></p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p><span>In this case one single node could bring down the whole cluster. </span></p>
<p><span>Of course, if we perform standard operations, or crashes happen in cascade and the cluster has time to rebuild the Primary Component, all will work well:</span></p>
<p><a href="http://www.tusacentral.com/joomla/images/stories/PXC_generic/2_crash_on_cascade.jpg"><img src="http://www.tusacentral.com/joomla/images/stories/PXC_generic/2_crash_on_cascade.jpg" alt="2 crash on cascade" width="462" height="347" class="size-full wp-image-56184 alignnone"><br></a></p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p><a href="http://www.tusacentral.com/joomla/images/stories/PXC_generic/2_crash_on_cascade_b.jpg"><img src="http://www.tusacentral.com/joomla/images/stories/PXC_generic/2_crash_on_cascade_b.jpg" alt="2 crash on cascade b" width="463" height="347" class="size-full wp-image-56183 alignnone"><br></a></p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p><span>The problem is that:</span></p>
<ul>
<li aria-level="1"><span>Cluster dimensions are dynamic and change in relation to the active Primary Component</span></li>
<li aria-level="1"><span>In environments managed by operators (e.g., Kubernetes), scaling the number of nodes up or down is even more dynamic and difficult to predict.</span></li>
<li aria-level="1"><span>In real life we may have an incredible variety of scenarios, I have built just a few (see </span><a href="https://github.com/Tusamarco/blogs/blob/master/quorum_calculations/quorum_calculation.pdf">PDF</a><span>)  </span></li>
</ul>
<p><span>The real question is not if we can do it but whether it is worth it: does using an odd number of nodes help or not?</span></p>
<p><span>The answer is partly. </span></p>
<p><span>Odd nodes get rid of the exact-tie case in the simple situation. They don't completely stop the cluster losing PRIMARY state.</span></p>
<p><b>What matters is the parity of the total weight, not the node count.</b></p>
<p><span> </span><span>A tie happens when present = partitioned, which means the counted weight (Σw(PC) − Left) splits exactly in half. </span></p>
<p><span>That is only possible when the counted weight is even. So:</span></p>
<ul>
<li aria-level="1"><b>5 nodes of w=1, any two-way split:</b><span> 3|2 or 4|1. One side always strictly wins.</span></li>
<li aria-level="1"><b>6 nodes with one node at w=2 (total 7)</b><span> behave exactly like an odd cluster. Adding the Galera Arbitrator (garbd) as a 7th voter does the same thing.</span></li>
<li aria-level="1"><b>3 nodes weighted 2,1,1 (total 4)</b><span> is an odd node count with an even total, so it can tie again.</span></li>
</ul>
<p><span>Where odd nodes still lose PRIMARY:</span></p>
<ol>
<li aria-level="1"><b>The PC shrinks after each failure, and its parity changes.</b><span> 5 nodes lose 1 and stay PRIMARY, but the new PC has 4 nodes. A 2|2 split after that ties. A rolling failure still ends at 2 nodes and losing one of those ties.</span></li>
<li aria-level="1"><b>Graceful leaves change the parity.</b><span> If 1 of 5 shuts down cleanly, the counted weight is 4, so a 2|2 split ties.</span></li>
<li aria-level="1"><b>Splits into three or more groups.</b><span> A 2|2|1 split of 5 has no tie, but no group is bigger than the rest combined (2 &lt; 3), so every group goes NON-PRIMARY.</span></li>
<li aria-level="1"><b>Losing a majority is still losing.</b><span> 5 nodes survive 2 failures but not 3, so odd doesn't add fault tolerance. 5 and 6 nodes both tolerate only 2 failures. The 6th node adds cost and a tie risk, and no extra resilience.</span></li>
<li aria-level="1"><b>Two data centers.</b><span> With 3+2 nodes, the 3-node site survives a link cut, but if that site goes down, the 2-node site stops. To survive the loss of either site we need a third location, even if it only runs garbd.</span></li>
</ol>
<p><b>In practice:</b><span> aim for an odd total weight in the PC and use a third site or an arbitrator for multi-DC setups. Even then, a cluster that has already shrunk can become even again, so the last-two-nodes tie can't be designed away. </span></p>
<p><span>Only pc.ignore_sb or a manual bootstrap gets past it. </span><b>But you do not want to go there</b><span>.</span></p>
<p><span>Given this, and given that what counts is NOT the number of nodes but the balance in the weight calculation, is using weights the right way to compensate?</span></p>
<p><span> </span><span>Well, this may be my personal opinion, but in my experience, weights make the cluster harder to manage, and the stability they add is narrow.</span></p>
<p><b>A weight-2 node acts like two nodes that always fail together.</b><span> Four nodes weighted 2,1,1,1 behave like a 5-node cluster where two of the nodes share one box. </span><span> </span><span>We trade a split-brain tie for a single point of fate:</span></p>
<ul>
<li aria-level="1"><b>Gain:</b><span> a clean 2|2 split now has a fixed winner, the side holding the heavy node.</span></li>
<li aria-level="1"><b>Loss:</b><span> losing the heavy node plus one other is fatal. Over-weighting makes it worse: with w=3, losing that node alone causes a tie.</span></li>
</ul>
<p><span>The set of survivable failures shifts rather than grows. We still tolerate only one arbitrary failure plus some specific splits.</span></p>
<p><span>What weights add to day-to-day operations:</span></p>
<ol>
<li aria-level="1"><b>Nodes are no longer equal.</b><span> Everyone must know which node is "special". Runbooks, monitoring and on-call decisions depend on it.</span></li>
<li aria-level="1"><b>Maintenance order matters.</b><span> Shut the heavy node down cleanly so its weight counts for neither side. </span><span> </span><span>If it crashes during maintenance on another node, the cluster can stop. Patching or rebooting it may first need its weight moved to another node, a step that's easy to forget.</span></li>
<li aria-level="1"><b>Weight is a configuration that can drift.</b><span> It has to be consistent across config files, rebuilds and node replacements. A rebuilt node that comes back with the default w=1 silently changes the quorum maths.</span></li>
<li aria-level="1"><b>The tie-breaker site is fixed.</b><span> With two data centers, the weighted side always wins. That's no different from simply putting 3 nodes in one DC and 2 in the other, and in both cases losing that DC stops the cluster.</span></li>
<li aria-level="1"><b>Shrinking still breaks it.</b><span> After failures the PC changes, so a setup tuned for 4 nodes may tie at 3 or 2.</span></li>
</ol>
<p><span>Simple to most complex for the same goal:</span></p>
<ol>
<li aria-level="1"><b>An odd number of equal nodes:</b><span> symmetric, nothing special to manage.</span></li>
<li aria-level="1"><b>An even number of nodes plus the Galera Arbitrator (garbd):</b><span> the data nodes stay equal, and the arbitrator holds the tie-breaking vote. If it runs at a third site, the cluster survives losing either main site, which weights alone can't give you.</span></li>
<li aria-level="1"><b>An even number of nodes with weights:</b><span> use this only when you can't add a node or an arbitrator and you deliberately want one side to always win a split.</span></li>
</ol>
<p><span>Weights don't make the cluster generally more stable, and they add operational complexity. They make sense as a deliberate tiebreaker, not as a fix for even node counts.</span></p>
<h2><span>Conclusions</span></h2>
<p><span>I stand by the evergreen KISS motto: Keep It Simple and Straightforward. </span><span>Having a PXC cluster with more than 3 nodes makes sense in very few cases, and only to scale reads or if you plan to kill 2 nodes at the same time on a cluster of 5. Given the significant complications the increase of nodes or the use of weight brings, I do not really see why someone in a healthy state of mind should look for trouble.</span></p>
<p><span>Just because it can be done, it doesn’t mean “do it”, especially if the gain is minimal and the complexity increases. </span></p>
<p><span>Do yourself a favor: keep your cluster to the minimum, to what is really needed and do not try to stretch it. You will save yourself from spending miserable nights doing cluster recovery or worse data reconciliation.</span></p>
<p><span>Great MySQL to everyone.</span></p>]]></content:encoded>
    <pubDate>Fri, 13 Nov 2026 13:58:31 +0000</pubDate>
    <dc:creator>Marco Tusa</dc:creator>
    <category>MySQL</category>
  </item>

  <item>
    <title>MySQL Shorts - Episode 106 - MySQL LAG() Window Function for Comparing Sales Data</title>
    <guid isPermaLink="false">planet-manual-f4e440ea6b82b33d214c0e3ff3fb6e0b</guid>
    <link>https://www.youtube.com/watch?v=ODW1Mqdjc5o&amp;list=PLWx5a9Tn2EvG4C90YFJ9eU61IpALeE0SN&amp;index=2</link>
    <description>MySQL's LAG() window function helps you compare sales totals across rows and calculate changes between consecutive records. See how to retrieve previous row values, order sales data by date, and calculate sales differences using the LAG() function in MySQL: https://dev.mysql.com/doc/refman/26.10/en/window-function-descriptions.html#function_lag</description>
    <pubDate>Thu, 12 Nov 2026 21:22:53 +0000</pubDate>
    <dc:creator>MySQL Community Team</dc:creator>
  </item>

  <item>
    <title>Introducing Percona Database Service (PDS): An Experiment in Simplifying Database Management</title>
    <guid isPermaLink="false">https://www.percona.com/?p=56139</guid>
    <link>https://www.percona.com/blog/percona-database-service-pds/</link>
    <description>It’s Thursday afternoon, and a developer has an idea.
Nothing complicated. They are working on a new application feature and want to test an idea before committing to a larger implementation. A quick prototype is already running locally. So far, so good. Next, they want to deploy the application to a shared environment so their colleagues can try it out. The application needs a database too, but it needs to be accessible from the shared environment, not just their laptop. And that’s where things get more complicated.
Now they need to submit a request to the infrastructure team, wait for an environment to be provisioned, or spend the afternoon working through Kubernetes configurations. What should have been a quick experiment becomes another task on somebody else’s backlog.
This is a common problem for engineering teams. Getting a database running in a shared environment shouldn’t require extensive coordination or infrastructure expertise. But there is another issue: when an application moves to staging or production, the requirements grow. The challenge isn’t just making provisioning easier, but finding a consistent way to manage databases as applications evolve, without requiring developers and infrastructure teams to go over the same things all over again.
Even when the underlying infrastructure is in place, creating a database with the right configuration, access controls, and operational settings can require several manual steps. And provisioning is only the beginning. Someone still needs to manage backups, monitor resource utilization, handle scaling, and maintain the database throughout its lifecycle.
At Percona, we’ve been exploring how to simplify these workflows while allowing organizations to retain control over their infrastructure. The result is Percona Database Service (PDS), a platform for provisioning and managing databases. We’re starting with MySQL and PostgreSQL on Kubernetes, and we’re making the initial implementation available for users to explore and provide feedback.
The idea is straightforward: bring the simplicity of a managed database experience to infrastructure you control, without the usual vendor lock-in.
How PDS works
PDS follows a relatively simple model: Kubernetes environments connect to a centralized control plane, allowing databases to be deployed and managed within the customer’s infrastructure.PDS builds on OpenEverest, the open-source database management platform that provides the underlying Kubernetes-based database provisioning and management capabilities. PDS extends that foundation with a centralized, more opinionated experience designed to simplify how developers interact with their databases.
Using Amazon Elastic Kubernetes Service (EKS), PDS provisions databases directly within your AWS account, rather than moving them into a Percona-hosted environment. This allows you to retain ownership of your infrastructure and databases while using PDS as a centralized management platform.
The PDS approach to database creation is intentionally opinionated. PDS provides standardized starting points for common workloads, yet still allows customization when needed. The goal isn’t to hide every technical detail. It’s to simplify and standardize routine database operations while keeping the underlying systems accessible to the teams that operate them.
PDS is currently an experimental initiative, not a generally available product. We’re using it to explore how database provisioning and management can be simplified, test different approaches with early users, and gather feedback before deciding how the platform should evolve. The capabilities described below reflect what we’re experimenting with today and may change as we learn more.
Connecting an environment is a command, not a form
It starts with a name, which doubles as the EKS cluster name, and one generated command, run locally on a machine that already has aws, eksctl, kubectl, helm, and python3. Nothing gets pasted or copied into a web form; the command does the work, and the environment reports itself in once it is up and running.
That command asks which AWS region to use. Leave it blank, and it uses your AWS CLI’s configured region, or us-west-2 if you haven’t set one. Then it creates an EKS cluster with four t3.2xlarge nodes and an S3 bucket, installs PDS onto it, and registers it. If you point the command at a cluster that’s already running PDS, it reuses that install instead of stacking a second one on top, and turns off that cluster’s own sign-in page in the process.
A database as a profile, not a spec sheet
Creating a database means picking a profile, not filling in a resource calculator: Dev, Staging, or Prod.
Each profile comes with configurations and operational defaults established by Percona’s experts, bringing best practices into the provisioning process. High availability (HA) is the default from Staging up, so it’s not a box you have to know to check. You retain control over your databases and infrastructure, while PDS helps ensure they’re configured according to proven operational standards.
Access is Private or Public. Public adds a load balancer in your own cloud account, billed by your provider, and an allowlist of IPs or CIDR ranges.
Your AI agent can drive it
It is really easy to connect an AI assistant to PDS. PDS provides setup instructions for supported clients, such as Cursor and Claude, enabling you to integrate database management into your existing workflows.
Authentication happens separately from the AI client configuration. Sign in through a browser-based flow, without storing passwords in configuration files or sharing credentials with the assistant. Once authenticated, the agent can start operating. The agent’s actions are recorded, so nothing goes unnoticed.
And then there are permissions you control. Grant read-only access to database information, allow operational tasks such as provisioning, scaling, and backups, or separately authorize actions such as deleting or restoring databases. Allowing an agent to modify databases does not automatically give it permission to delete them.
Managing databases after deployment
Creating a database is only one part of its lifecycle.
PDS provides a database management interface where users can inspect deployment status, review resource metrics, access connection information, and perform supported operational tasks. For example, users can scale database resources, configure scheduled backups, or initiate an on-demand backup. Backup schedules include configurable frequency, execution time, and retention settings. PDS also provides an activity history to help track operations performed through the platform. The interface currently exposes basic operational metrics, including CPU utilization, disk usage, and database connections.
These capabilities are intended to cover common management tasks without requiring users to interact directly with Kubernetes resources for every operation.
Supporting teams and shared environments
In most organizations, the people requesting databases are not necessarily the people responsible for operating the underlying infrastructure.
A development team may need a PostgreSQL instance for an application, while platform or DBA teams remain responsible for infrastructure, security, and operational standards. PDS supports this separation through organizations, teams, and user management. Users can register, create teams, and invite colleagues. Database environments can be organized around teams, creating a foundation for shared management and controlled access.
This is an area we’re developing based on customer feedback. One recurring requirement is to give application teams more independence without losing centralized oversight. That includes questions around permissions, resource limits, and which operations should require additional authorization.
The current implementation provides the initial team and user management capabilities. More granular governance and operational controls are areas we’re continuing to explore.
Integration with Percona Monitoring and Management (PMM)
PDS also integrates with Percona Monitoring and Management (PMM). This gives teams access to deeper database performance monitoring and diagnostics. PMM enables more detailed analysis of query performance, database activity, and potential bottlenecks for PDS users.
Together, they combine simplified database management with the monitoring capabilities Percona users already know and trust.
Try PDS and tell us what you think
We’re making PDS available to learn more about how it fits into real database workflows.
Visit pds.percona.com, create an account, and connect an Amazon EKS environment in your AWS account. From there, you can provision MySQL or PostgreSQL databases, create teams, invite users, configure backups, and explore integrations. We look forward to receiving your feedback:

Which provisioning steps create too much work?
What should a standardized configuration include?
Which operational tasks should be automated?
Where should the boundaries between developer self-service and managed database be?

This is an early preview, and some functionality will change as development continues. Our goal is to build a practical, open source approach to database management that gives teams more control over their infrastructure, not less.
Try PDS Today
The post Introducing Percona Database Service (PDS): An Experiment in Simplifying Database Management appeared first on Percona.</description>
    <content:encoded><![CDATA[<p>It’s Thursday afternoon, and a developer has an idea.</p>
<p>Nothing complicated. They are working on a new application feature and want to test an idea before committing to a larger implementation. A quick prototype is already running locally. So far, so good. Next, they want to deploy the application to a shared environment so their colleagues can try it out. The application needs a database too, but it needs to be accessible from the shared environment, not just their laptop. And that’s where things get more complicated.</p>
<p>Now they need to submit a request to the infrastructure team, wait for an environment to be provisioned, or spend the afternoon working through Kubernetes configurations. What should have been a quick experiment becomes another task on somebody else’s backlog.</p>
<p>This is a common problem for engineering teams. Getting a database running in a shared environment shouldn’t require extensive coordination or infrastructure expertise. But there is another issue: when an application moves to staging or production, the requirements grow. The challenge isn’t just making provisioning easier, but finding a consistent way to manage databases as applications evolve, without requiring developers and infrastructure teams to go over the same things all over again.</p>
<p>Even when the underlying infrastructure is in place, creating a database with the right configuration, access controls, and operational settings can require several manual steps. And provisioning is only the beginning. Someone still needs to manage backups, monitor resource utilization, handle scaling, and maintain the database throughout its lifecycle.</p>
<p>At Percona, we’ve been exploring how to simplify these workflows while allowing organizations to retain control over their infrastructure. The result is <strong>Percona Database Service (PDS)</strong>, a platform for provisioning and managing databases. We’re starting with MySQL and PostgreSQL on Kubernetes, and we’re making the initial implementation available for users to explore and provide feedback.</p>
<p>The idea is straightforward: bring the simplicity of a managed database experience to infrastructure you control, without the usual vendor lock-in.</p>
<h2>How PDS works</h2>
<p>PDS follows a relatively simple model: Kubernetes environments connect to a centralized control plane, allowing databases to be deployed and managed within the customer’s infrastructure.<br aria-hidden="true">PDS builds on <a href="https://openeverest.io/">OpenEverest</a>, the open-source database management platform that provides the underlying Kubernetes-based database provisioning and management capabilities. PDS extends that foundation with a centralized, more opinionated experience designed to simplify how developers interact with their databases.</p>
<p>Using Amazon Elastic Kubernetes Service (EKS), PDS provisions databases directly within your AWS account, rather than moving them into a Percona-hosted environment. This allows you to retain ownership of your infrastructure and databases while using PDS as a centralized management platform.</p>
<p>The PDS approach to database creation is intentionally opinionated. PDS provides standardized starting points for common workloads, yet still allows customization when needed. The goal isn’t to hide every technical detail. It’s to simplify and standardize routine database operations while keeping the underlying systems accessible to the teams that operate them.</p>
<p>PDS is currently an experimental initiative, not a generally available product. We’re using it to explore how database provisioning and management can be simplified, test different approaches with early users, and gather feedback before deciding how the platform should evolve. The capabilities described below reflect what we’re experimenting with today and may change as we learn more.</p>
<h2>Connecting an environment is a command, not a form</h2>
<p>It starts with a name, which doubles as the EKS cluster name, and one generated command, run locally on a machine that already has <code>aws</code>, <code>eksctl</code>, <code>kubectl</code>, <code>helm</code>, and <code>python3</code>. Nothing gets pasted or copied into a web form; the command does the work, and the environment reports itself in once it is up and running.</p>
<p>That command asks which AWS region to use. Leave it blank, and it uses your AWS CLI’s configured region, or <code>us-west-2</code> if you haven’t set one. Then it creates an EKS cluster with four <code>t3.2xlarge</code> nodes and an S3 bucket, installs PDS onto it, and registers it. If you point the command at a cluster that’s already running PDS, it reuses that install instead of stacking a second one on top, and turns off that cluster’s own sign-in page in the process.</p>
<h2>A database as a profile, not a spec sheet</h2>
<p>Creating a database means picking a profile, not filling in a resource calculator: <strong>Dev</strong>, <strong>Staging</strong>, or <strong>Prod</strong>.</p>
<p>Each profile comes with configurations and operational defaults established by Percona’s experts, bringing best practices into the provisioning process. High availability (HA) is the default from Staging up, so it’s not a box you have to know to check. You retain control over your databases and infrastructure, while PDS helps ensure they’re configured according to proven operational standards.</p>
<p>Access is <strong>Private</strong> or <strong>Public</strong>. Public adds a load balancer in your own cloud account, billed by your provider, and an allowlist of IPs or CIDR ranges.</p>
<h2>Your AI agent can drive it</h2>
<p>It is really easy to connect an AI assistant to PDS. PDS provides setup instructions for supported clients, such as Cursor and Claude, enabling you to integrate database management into your existing workflows.</p>
<p>Authentication happens separately from the AI client configuration. Sign in through a browser-based flow, without storing passwords in configuration files or sharing credentials with the assistant. Once authenticated, the agent can start operating. The agent’s actions are recorded, so nothing goes unnoticed.</p>
<p>And then there are permissions you control. Grant read-only access to database information, allow operational tasks such as provisioning, scaling, and backups, or separately authorize actions such as deleting or restoring databases. Allowing an agent to modify databases does not automatically give it permission to delete them.</p>
<h2>Managing databases after deployment</h2>
<p>Creating a database is only one part of its lifecycle.</p>
<p>PDS provides a database management interface where users can inspect deployment status, review resource metrics, access connection information, and perform supported operational tasks. For example, users can scale database resources, configure scheduled backups, or initiate an on-demand backup. Backup schedules include configurable frequency, execution time, and retention settings. PDS also provides an activity history to help track operations performed through the platform. The interface currently exposes basic operational metrics, including CPU utilization, disk usage, and database connections.</p>
<p>These capabilities are intended to cover common management tasks without requiring users to interact directly with Kubernetes resources for every operation.</p>
<h2>Supporting teams and shared environments</h2>
<p>In most organizations, the people requesting databases are not necessarily the people responsible for operating the underlying infrastructure.</p>
<p>A development team may need a PostgreSQL instance for an application, while platform or DBA teams remain responsible for infrastructure, security, and operational standards. PDS supports this separation through organizations, teams, and user management. Users can register, create teams, and invite colleagues. Database environments can be organized around teams, creating a foundation for shared management and controlled access.</p>
<p>This is an area we’re developing based on customer feedback. One recurring requirement is to give application teams more independence without losing centralized oversight. That includes questions around permissions, resource limits, and which operations should require additional authorization.</p>
<p>The current implementation provides the initial team and user management capabilities. More granular governance and operational controls are areas we’re continuing to explore.</p>
<h2>Integration with Percona Monitoring and Management (PMM)</h2>
<p>PDS also integrates with <a href="https://www.percona.com/software/database-tools/percona-monitoring-and-management">Percona Monitoring and Management (PMM)</a>. This gives teams access to deeper database performance monitoring and diagnostics. PMM enables more detailed analysis of query performance, database activity, and potential bottlenecks for PDS users.</p>
<p>Together, they combine simplified database management with the monitoring capabilities Percona users already know and trust.</p>
<h2>Try PDS and tell us what you think</h2>
<p>We’re making PDS available to learn more about how it fits into real database workflows.</p>
<p>Visit <a href="https://pds.percona.com/" target="_blank" rel="noopener">pds.percona.com</a>, create an account, and connect an Amazon EKS environment in your AWS account. From there, you can provision MySQL or PostgreSQL databases, create teams, invite users, configure backups, and explore integrations. We look forward to receiving your feedback:</p>
<ul>
<li>Which provisioning steps create too much work?</li>
<li>What should a standardized configuration include?</li>
<li>Which operational tasks should be automated?</li>
<li>Where should the boundaries between developer self-service and managed database be?</li>
</ul>
<p>This is an early preview, and some functionality will change as development continues. Our goal is to build a practical, open source approach to database management that gives teams more control over their infrastructure, not less.</p>
<p><strong><a href="https://pds.percona.com/">Try PDS Today</a></strong></p>
<p>The post <a href="https://www.percona.com/blog/percona-database-service-pds/">Introducing Percona Database Service (PDS): An Experiment in Simplifying Database Management</a> appeared first on <a href="https://www.percona.com/">Percona</a>.</p>]]></content:encoded>
    <pubDate>Thu, 12 Nov 2026 16:43:13 +0000</pubDate>
    <dc:creator>MySQL Performance Blog</dc:creator>
    <category>MySQL</category>
    <category>Percona Announcements</category>
    <category>Percona Software</category>
    <category>PostgreSQL</category>
  </item>

  <item>
    <title>Git commits on MySQL tree, impact by AI</title>
    <guid isPermaLink="false">tag:blogger.com,1999:blog-14455177.post-3668793100719948153</guid>
    <link>https://mikaelronstrom.blogspot.com/2026/11/git-commits-on-mysql-tree-impact-by-ai.html</link>
    <description></description>
    <content:encoded><![CDATA[<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzaRpn7CRtIDfhiAxb4wkhbqfBZ14FEln13N3VyYDzk_bV3lW_NsW0b7dab6HK5bK9hszXNPUd1aUIs45EIky19Q6pKqdbKMmCr5BPGuIbfx0Wr3IHm_cdqx0onEobzB8bTMEeeWPmjd1J2m0V0tUOM6L36QQMyClq-wCYp26zuuDzDvRXJVqy/s0/mikael-commits-2004-2026.png"><img alt="" border="0" data-original-height="1425" data-original-width="1950" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzaRpn7CRtIDfhiAxb4wkhbqfBZ14FEln13N3VyYDzk_bV3lW_NsW0b7dab6HK5bK9hszXNPUd1aUIs45EIky19Q6pKqdbKMmCr5BPGuIbfx0Wr3IHm_cdqx0onEobzB8bTMEeeWPmjd1J2m0V0tUOM6L36QQMyClq-wCYp26zuuDzDvRXJVqy/s0/mikael-commits-2004-2026.png" width="800"></a></div>]]></content:encoded>
    <pubDate>Thu, 12 Nov 2026 16:06:39 +0000</pubDate>
    <dc:creator>Mikael Ronström</dc:creator>
  </item>

  <item>
    <title>Migrating Azure SQL Database to Nutanix Database Service with a Planned Cutover Window</title>
    <guid isPermaLink="false">https://www.nutanix.com/blog/migrating-azure-sql-database-to-nutanix-database-service</guid>
    <link>https://www.nutanix.com/blog/migrating-azure-sql-database-to-nutanix-database-service</link>
    <description>Migrate Azure SQL Database to SQL Server on Nutanix Database Service (NDB) using Azure SQL Data Sync or Azure Data Factory, with a planned cutover and data-integrity checks.</description>
    <pubDate>Thu, 12 Nov 2026 08:39:36 +0000</pubDate>
    <dc:creator>Nutanix</dc:creator>
    <category>MOF - Interest</category>
    <category>Blog Category</category>
    <category>Nutanix Database Service (NDB)</category>
    <category>Private Cloud</category>
    <category>VMware Alternative   Broadcom Compete</category>
  </item>

  <item>
    <title>Refresh a single tenant across accounts with Aurora MySQL and Amazon S3</title>
    <guid isPermaLink="false">7e74ffca4941e90d7a2f7af676d116105d249796</guid>
    <link>https://aws.amazon.com/blogs/database/refresh-a-single-tenant-across-accounts-with-aurora-mysql-and-amazon-s3/</link>
    <description>Multi-tenant software as a service (SaaS) applications regularly need to copy one tenant’s data from production into a lower environment. Teams do this to reproduce a defect, validate a schema change, or give quality assurance (QA) a representative dataset. When production and non-production run in separate AWS accounts, their virtual private clouds (VPCs) often have no route to each other. That turns the copy into a networking problem as much as a data problem, and security teams are rightly reluctant to open a path between the two. 
In this post, we show how to move a single tenant’s data between Amazon Aurora MySQL-Compatible Edition clusters in two separate AWS accounts, using Aurora cloning, SELECT INTO OUTFILE S3, and LOAD DATA FROM S3. The databases don’t connect directly. Each account reaches Amazon Simple Storage Service (Amazon S3) independently through its own gateway VPC endpoint, and the accounts share only one staging bucket as their touchpoint. 
This post builds on two others. Managed database backup and recovery in a multi-tenant SaaS application covers segregating a tenant across the silo, bridge, and pool models. In its terms, what follows is extraction-phase segregation in a pool model. Best practices for exporting and importing data from Amazon Aurora MySQL to Amazon S3 covers the export and import mechanics. What’s new here is moving a tenant-scoped extract between accounts with no network path, plus the access and encryption model that keeps the staging bucket safe. 
Many SaaS teams handle this with a logical dump-and-load pipeline that runs for hours per tenant and can’t resume after a failure. This pattern replaces that single long-running job with independent per-table steps: tables export and import in parallel, and a failed table is retried on its own instead of restarting the whole refresh. 
Why not cross-account cloning? 
If you’re moving data between accounts, cross-account cloning through AWS Resource Access Manager (AWS RAM) is the AWS-recommended approach, and it’s worth evaluating first. It’s the right tool for copying a whole cluster into another account, such as a migration or a full-environment refresh. It doesn’t fit the job in this post, for three reasons. We’re moving a single tenant’s data, not the entire cluster volume with every tenant on it. We’re loading an existing non-production cluster that QA already points at, not standing up a new one. And our source cluster is encrypted with the default RDS key, which cross-account cloning can’t share. So the rest of this post builds a different pattern. 
Solution overview 
The pattern has four phases. The central idea is that Amazon S3 is a Regional service reachable from inside a VPC through a gateway endpoint without traversing the internet, so two VPCs with no route between them can both reach the same bucket. 
The following diagram shows the data flow. The two clusters have no direct connection. Every interaction between the accounts happens through the staging bucket. 

 
 Figure 1: Data flow for a cross-account tenant refresh through a shared Amazon S3 staging bucket
 
 
 Clone the production cluster – Aurora cloning is copy-on-write at the storage layer, so creation takes minutes largely independent of database size. The clone is a separate cluster in the production account that you can drive hard without affecting production. 
 Export the tenant’s rows to Amazon S3 – On the clone, run SELECT INTO OUTFILE S3 with a predicate on the tenant ID. Aurora writes delimited text files to the bucket, splitting into part files past a threshold (6 GB by default) and emitting a manifest. 
 Import into the non-production cluster – LOAD DATA FROM S3 reads the manifest from that same bucket and ingests the part files. 
 Clean up – Delete the clone and the exported objects. 
 
Because each table exports and imports as an independent unit, both phases parallelize across tables, and a failure affects only that table. 
This pattern changes the shape of the connectivity requirement rather than removing it. There is no VPC peering, transit gateway, or VPC-to-VPC path between the accounts. Each account needs only an S3 gateway VPC endpoint so its cluster can reach S3 privately. Access between the accounts lives in resource policy, not routing, so production doesn’t become reachable from the lower environment. That’s what makes the design straightforward to clear through security review. 
Prerequisites 
Have the following in place before you start, in roughly this setup order: 
 
 Two AWS accounts (one production, one non-production), each with its own VPC and no network route between them. (A third shared account can own the staging bucket. This post keeps the bucket in the production account.) 
 The AWS Command Line Interface (AWS CLI), configured with credentials in each account that can manage Amazon Relational Database Service (Amazon RDS), Amazon S3, AWS Identity and Access Management (IAM), and AWS Key Management Service (AWS KMS). 
 Aurora MySQL-Compatible Edition, version 3 (MySQL 8.0-compatible) or later on both clusters. This post uses the Aurora MySQL statements SELECT INTO OUTFILE S3 and LOAD DATA FROM S3. (Aurora PostgreSQL offers an equivalent path through the aws_s3 extension. The cross-account bridge applies there too, but the SQL in this post is MySQL-specific.) 
 An existing non-production cluster with a schema that already matches production. LOAD DATA carries rows, not data definition language (DDL), so the destination tables must exist with the same structure. Reconcile any schema drift before you load. 
 An index on the tenant ID column in every participating table. Without one, the WHERE clause degrades to a full table scan. See Optimizing and tuning queries in Amazon Aurora MySQL. 
 An S3 gateway VPC endpoint in each VPC, so each cluster can reach Amazon S3 privately without a route between the two VPCs. 
 Both clusters and the staging bucket in the same AWS Region. A clone can’t be created in an AWS Region different from its source, so the entire pattern operates within a single Region. See Cloning an Amazon Aurora DB cluster volume. 
 A maintenance window for the first-time parameter change on the non-production cluster. Associating the IAM role uses the static DB cluster parameter aws_default_s3_role, which takes effect only after the writer reboots. (The per-refresh clone avoids this reboot by associating the role before its writer starts. See Step 1.) See Working with DB cluster parameter groups. 
 
Walkthrough 
The walkthrough has five parts. First, configure the cross-account bridge once. Then run the four phases of each refresh: clone, export, import, and clean up. The bridge is the one-time setup. The four phases are what you repeat on every refresh. 
Configure the cross-account bridge 
Set this up once. The bridge is made of three things, and no network route is among them. The production account owns an encrypted S3 bucket and grants the non-production account’s import role read access to it. Each cluster has an IAM role that lets Aurora read from or write to S3 on its behalf. And a customer managed AWS KMS key, held in the production account, grants decrypt permission to both roles so the cross-account read can succeed. The following sections configure each piece in turn: the Aurora roles, the bucket ownership and policy, the encryption key, and the data-store controls on the bucket. 
Wire up the Aurora roles 
Each statement needs an IAM role associated with its cluster, that role named in a DB cluster parameter, and a database grant to the issuing user. The export/import best practices post covers the role’s permissions in detail. 
In this one-time setup, wire up the non-production (import) cluster only. The clone is created fresh each refresh, so its export-role association belongs in Step 1: 
 
 # Non-production account — import
aws rds add-role-to-db-cluster \
  --db-cluster-identifier nonprod-cluster \
  --role-arn arn:aws:iam::444455556666:role/AuroraS3ImportRole 
 
Then set aws_default_s3_role in the cluster’s parameter group to the role’s Amazon Resource Name (ARN) and reboot the writer (it’s a static parameter). On Aurora MySQL version 3 and 8.4 this is the parameter to use. The statement-specific aurora_select_into_s3_role and aurora_load_from_s3_role aren’t available there. 
Finally, grant the import user the right to issue the statement. On version 3 and 8.4 this is a role, not a privilege: 
 
 -- Non-production cluster
GRANT AWS_LOAD_S3_ACCESS TO 'refresh_user'@'%'; 
 
The matching GRANT AWS_SELECT_S3_ACCESS runs on the clone in Step 1. Because these are roles, you must activate them: set activate_all_roles_on_login or issue SET ROLE ALL per session. Otherwise a pipeline opening a fresh connection per table fails on permissions while SHOW GRANTS looks correct. 
Two export-role permissions caused a misleading Access Denied in our testing. First, don’t put an s3:prefix condition on the export role’s s3:ListBucket. SELECT INTO OUTFILE S3 lists the bucket without a matching prefix, so the condition denies the export. Scope the object actions to the prefix, but leave ListBucket and GetBucketLocation unconditioned. (The import role is the exception: LOAD DATA FROM S3 MANIFEST reads a specific key, so its ListBucket can keep the prefix condition.) Second, with an S3 Bucket Key enabled, the export role also needs kms:Decrypt. 
Decide which account owns the bucket 
Put the staging bucket in the production account, the account that writes the objects. Because the writer is also the bucket owner, S3 Object Ownership doesn’t come into question. Ownership mismatch is a common failure with cross-account buckets. This bucket policy is the connectivity step for the pattern: it grants the non-production import role read access across the account boundary, scoped to the refresh prefix. Scope the grant to a role ARN and bound it with a condition key: aws:PrincipalOrgID (shown here), or aws:PrincipalAccount with the non-production account ID if you don’t use AWS Organizations. That way the grant can’t be exercised outside that boundary even if the role’s trust is later widened. Because this bucket holds production tenant data, the ARN alone isn’t a sufficient boundary: 
 
 {
  &quot;Version&quot;: &quot;2012-10-17&quot;,
  &quot;Statement&quot;: [
    {
      &quot;Sid&quot;: &quot;ListRefreshPrefixOnly&quot;,
      &quot;Effect&quot;: &quot;Allow&quot;,
      &quot;Principal&quot;: {&quot;AWS&quot;: &quot;arn:aws:iam::444455556666:role/AuroraS3ImportRole&quot;},
      &quot;Action&quot;: &quot;s3:ListBucket&quot;,
      &quot;Resource&quot;: &quot;arn:aws:s3:::amzn-s3-demo-tenant-refresh&quot;,
      &quot;Condition&quot;: {
        &quot;StringLike&quot;: {&quot;s3:prefix&quot;: &quot;refresh-*&quot;},
        &quot;StringEquals&quot;: {&quot;aws:PrincipalOrgID&quot;: &quot;o-exampleorgid&quot;}
      }
    },
    {
      &quot;Sid&quot;: &quot;ReadRefreshObjects&quot;,
      &quot;Effect&quot;: &quot;Allow&quot;,
      &quot;Principal&quot;: {&quot;AWS&quot;: &quot;arn:aws:iam::444455556666:role/AuroraS3ImportRole&quot;},
      &quot;Action&quot;: &quot;s3:GetObject&quot;,
      &quot;Resource&quot;: &quot;arn:aws:s3:::amzn-s3-demo-tenant-refresh/refresh-*&quot;,
      &quot;Condition&quot;: {
        &quot;StringEquals&quot;: {&quot;aws:PrincipalOrgID&quot;: &quot;o-exampleorgid&quot;}
      }
    }
  ]
} 
 
If you instead put the bucket in the non-production account, check its S3 Object Ownership setting. Buckets created after April 2023 default to Bucket owner enforced, which disables ACLs and gives the owner every object, so the import works. On an older ACL-enabled bucket the reader gets AccessDenied, and because SELECT INTO OUTFILE S3 can’t set a bucket-owner-full-control ACL, the fix is to set Bucket owner enforced. 
Encrypt with a customer managed key 
Encrypt the bucket with SSE-KMS using a customer managed key. This is a requirement of the cross-account design, not a hardening step: the AWS managed aws/s3 key policy can’t be modified to grant a cross-account principal the kms:Decrypt the import needs. Keep the key in the production account. Its key policy must grant kms:GenerateDataKey and kms:Decrypt to the export role and kms:Decrypt to the import role. A missing grant here is the most common failure in cross-account variants of this pattern. 
On Aurora MySQL 3.05 and later you can also name the key in the statement with an ENCRYPTION SSE_KMS 'cmk_id' clause, so the encryption is visible in the SQL rather than resting on a bucket setting. 
Treat the staging bucket as a production data store 
The bucket holds production tenant data as plain text, so it inherits production’s requirements: 
 
 Block Public Access. 
 A bucket policy Deny on aws:SecureTransport: false. 
 AWS CloudTrail data events on the refresh prefix. 
 A dedicated bucket for staging. 
 Explicit deletion after a successful import, with a 24-hour Lifecycle rule as a backstop rather than the primary mechanism. 
 
With the bridge configured, you run the four phases on each refresh. The following steps walk through each phase with the commands to run and the pitfalls to avoid. 
Step 1: Create the clone 
Cloning is a point-in-time restore with copy-on-write semantics. Pass the cluster parameter group that names aws_default_s3_role. Without --db-cluster-parameter-group-name, the clone comes up on the default parameter group, which has no role set, and the export fails: 
 
 aws rds restore-db-cluster-to-point-in-time \
  --source-db-cluster-identifier prod-cluster \
  --db-cluster-identifier tenant-refresh-clone \
  --restore-type copy-on-write \
  --use-latest-restorable-time \
  --vpc-security-group-ids sg-0abc123 \
  --db-subnet-group-name prod-subnet-group \
  --db-cluster-parameter-group-name tenant-refresh-s3-params 
 
This restores the cluster volume only. The clone doesn’t inherit the source’s IAM role association, so associate the export role before creating the writer. In our testing, this ordering let the writer read aws_default_s3_role at first start and the export ran without a reboot: 
 
 # Associate the export role with the clone (it is not inherited from the source)
aws rds add-role-to-db-cluster \
  --db-cluster-identifier tenant-refresh-clone \
  --role-arn arn:aws:iam::111122223333:role/AuroraS3ExportRole

# Then create the writer, sized for export throughput rather than steady-state workload
aws rds create-db-instance \
  --db-instance-identifier tenant-refresh-clone-writer \
  --db-cluster-identifier tenant-refresh-clone \
  --db-instance-class db.r6g.8xlarge \
  --engine aurora-mysql 
 
Confirm the association is active before exporting. describe-db-clusters should show &quot;Status&quot;: &quot;ACTIVE&quot; under AssociatedRoles. If you associate the role after the writer is already running, the first export fails with Missing Credentials: Cannot instantiate S3 Client. Reboot the writer to clear it. Then grant the export role to the database user on the clone: 
 
 -- On the clone
GRANT AWS_SELECT_S3_ACCESS TO 'refresh_user'@'%'; 
 
Step 2: Export the tenant’s data 
Run one statement per table, concurrently: 
 
 SELECT * FROM orders
  WHERE tenant_id = 4711
  INTO OUTFILE S3 's3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/orders'
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '&quot;'
  LINES TERMINATED BY '\n'
  MANIFEST ON
  OVERWRITE ON; 
 
On success, the statement returns a rows-affected count (Query OK, 4821233 rows affected). MANIFEST ON writes an orders.manifest JSON file listing every part file, which the import consumes directly: 
 
 {
  &quot;entries&quot;: [
    { &quot;url&quot;: &quot;s3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/orders.part_00000&quot; },
    { &quot;url&quot;: &quot;s3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/orders.part_00001&quot; }
  ]
} 
 
OVERWRITE ON makes retries idempotent. In our testing, 8–20 concurrent exports saturated instance network bandwidth, but the ceiling depends on instance size and table width, so measure rather than assume. Output is uncompressed text split into part files at the 6 GB default threshold, and parallelism is across tables, not within a table. See Considerations and limitations for the large-single-table case. 
Scope every export to exactly one tenant. This is the correctness property the whole pattern rests on, and looking for a tenant_id column isn’t sufficient to guarantee it. Classify every table into one of three buckets, and treat anything unclassified as a blocker: 
 
 Tenant-scoped: carries the tenant ID. The preceding predicate applies directly. 
 Tenant-derived: holds tenant-owned rows but has no tenant column, typically a child table keyed to its parent. Copy it wholesale and you export every tenant’s data. Skip it and the refresh is incomplete. Reach it through a join: 
   
   SELECT li.* FROM line_items li
  JOIN orders o ON o.order_id = li.order_id
  WHERE o.tenant_id = 4711
  INTO OUTFILE S3 's3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/line_items'
  ... 
    
 Shared: reference data common to all tenants, safe to copy in full, but as a named allowlist rather than by omission. 
 
Generate these statements from the classification rather than hand-authoring predicates, and make isolation a preventive control that runs before any S3 write. But check what can actually go wrong, not a tautology. Asserting COUNT(DISTINCT tenant_id) = 1 on a query already filtered by WHERE tenant_id = 4711 consistently passes and proves nothing. The real failure modes are an unclassified table, a tenant-derived table exported without its join (which leaks every tenant’s rows), and a wrong join path. Guard those: 
 
 Classification coverage. Assert that every base table in information_schema.tables appears in your classification. Treat an unclassified table as a hard stop. It’s the most likely way to leak or ship an incomplete tenant. 
 Tenant-derived join correctness. For each tenant-derived table, compare the join’s row count against the count expected by walking from the parent (for example, line_items whose order_id belongs to the tenant’s orders). A mismatch means the join path is wrong. 
 
A failing check aborts the export, so over-scoped data doesn’t reach the lower environment. Checking only after import is too late. 
Step 3: Import into non-production 
The non-production cluster is a long-lived environment that other tenants share, so you can’t load into it blind. Prepare the target first, then load. 
Prepare the target: remove the tenant’s existing rows. Before loading, delete the tenant’s current rows from the non-production tables, in child-to-parent order so foreign keys stay satisfied. Use a tenant-scoped DELETE, never TRUNCATE. TRUNCATE empties the whole table and would wipe every other tenant’s rows: 
 
 -- Child (tenant-derived) tables first, via the join to the parent
DELETE li FROM line_items li
  JOIN orders o ON o.order_id = li.order_id
  WHERE o.tenant_id = 4711;

-- Then the tenant-scoped parent tables
DELETE FROM orders WHERE tenant_id = 4711; 
 
Assume keys can’t collide. This load assumes the production primary keys for this tenant won’t collide with rows other tenants already own in non-production. That holds when non-production was originally seeded from production (so the key spaces line up), or when the schema uses UUIDs or non-overlapping auto-increment ranges per environment. If non-production auto-increment sequences have drifted independently from production’s, reconcile that before loading. Otherwise LOAD DATA can hit duplicate-key errors or, with unique_checks off, silently create duplicates. 
Handle shared tables deliberately. Reference tables shared across all tenants aren’t tenant-scoped. Loading them “in full” overwrites data every other non-production tenant depends on. Treat shared tables as a separate decision: skip them if non-production already has the correct reference data, or reconcile them explicitly (for example, upsert only new reference rows) rather than overwriting. 
With the tenant’s old rows removed, load one LOAD DATA FROM S3 statement per table, reading the manifests the export wrote. The following example loads the orders table: 
 
 SET SESSION foreign_key_checks = 0;
SET SESSION unique_checks = 0;

LOAD DATA FROM S3 MANIFEST
  's3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/orders.manifest'
  INTO TABLE orders
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '&quot;'
  LINES TERMINATED BY '\n';

SET SESSION unique_checks = 1;
SET SESSION foreign_key_checks = 1; 
 
This is one LOAD DATA FROM S3 MANIFEST per table, matching the per-table export. It’s a matrix of operations, not a single bulk load. Either load parent tables before children, or (as shown) disable foreign_key_checks and unique_checks for the load and re-enable them afterward. 
Be precise about what re-enabling those checks does: it doesn’t re-validate the rows you just loaded. MySQL applies the checks only to new changes and doesn’t rescan existing rows. The load is safe here only because the data was consistent on the clone and parents load before children. Running with unique_checks = 0 against a table that already holds other tenants’ rows can also admit duplicates into a secondary unique index without error, so remove the tenant’s existing rows first (above) and make sure production key ranges can’t collide. 
On success, LOAD DATA reports the rows loaded (Query OK, 4821233 rows affected). To speed up InnoDB bulk loads, drop secondary indexes before loading and rebuild them afterward (ALTER TABLE ... DISABLE KEYS is a no-op on InnoDB). 
Validate the refresh 
After loading, compare row counts for the tenant against the source clone (not the live primary, whose data might have changed since the clone was taken): 
 
 -- Run on the clone, then on the non-production target; the counts must match.
SELECT COUNT(*) FROM orders WHERE tenant_id = 4711;   -- e.g. 4821233 on both 
 
Do this for every table. Also confirm every manifest part file was consumed, and that other tenants’ row counts in the shared tables are unchanged. That is the check that proves you refreshed one tenant without disturbing the rest. 
Step 4: Clean up 
Delete the cloned cluster and its instance, then the exported objects. An orphaned clone bills for every storage page it diverges from the source, and an orphaned export leaves tenant data at rest. 
Performance, cost, and failure handling 
Refresh time depends on your data, so measure a representative tenant before you commit to a service level agreement (SLA). Four things drive it. Export runs in parallel across tables but not within one, so the largest table sets the wall-clock time. The export is uncompressed text, so the full data volume moves through Amazon S3. Import time includes both the load and any secondary index rebuilds. And the instance sizes of the clone writer and the target writer cap throughput on each side, so size both for the refresh window rather than steady-state workload. 
Speed isn’t the main win, though. Under a monolithic dump-and-load, a failure discards all completed work. Here, a failed export affects one table while the rest sit durable in Amazon S3, and you retry a failed import one table at a time (delete the tenant’s rows, as in Step 3, then re-read the manifest). Because all export work runs against an ephemeral clone, a pathological job can’t degrade live workloads. 
Considerations and limitations 
Weigh these before adopting the pattern: 
 
 Tenant shape matters. Export parallelizes across tables but not within one, so wall-clock time depends on the single largest table, not the aggregate size. A tenant spread across many tables refreshes far faster than one dominated by a single large table. If one table holds most of the volume, shard its export by primary-key range. 
 Transient storage. The export is uncompressed text, so a large tenant lands a large volume in Amazon S3 for the life of the refresh. Plan for it, and delete on completion. 
 Right-size the clone. Export throughput is often network-bound, so check DB instance networking figures rather than reusing the production class. The clone exists only for the refresh. 
 Clone limit. A cluster supports up to 15 copy-on-write clones. The sixteenth becomes a full copy: slower, full storage, no error. Delete each clone after its refresh. See Limitations of Aurora cloning. 
 Protect personal data. Encryption isn’t the whole story. Production tenant data might contain personal or regulated data, so confirm your customer agreements and compliance permit copying it into non-production, and mask or anonymize sensitive columns on the clone before SELECT INTO OUTFILE S3 runs. That keeps unmasked data out of the bucket. 
 When another approach fits better. AWS Database Migration Service (AWS DMS) is the choice for a one-time migration or change data capture, though cross-account it needs a cross-VPC path through AWS PrivateLink or AWS Transit Gateway. Cross-account cloning through AWS RAM fits copying a whole cluster rather than one tenant. 
 
Conclusion 
In this post, we showed how to refresh a single tenant’s data from a production account into an isolated non-production account with no connectivity between them. By chaining Aurora cloning, SELECT INTO OUTFILE S3, and LOAD DATA FROM S3, we turned a cross-account data movement problem into a Regional one: the clusters don’t connect, and one encrypted staging bucket is all they share. Because the pattern reads from an isolated clone and retries at the table level, it can help improve operational stability when failures occur. 
To go deeper, see Cloning an Amazon Aurora DB cluster volume, Saving data into text files in Amazon S3, and Loading data from text files in an Amazon S3 bucket. 
After you’ve run the refresh by hand a few times, the natural next step is to automate it. The manual steps map onto AWS Step Functions and AWS Lambda so a refresh runs on a schedule with nothing billed between runs. A design that keeps export and cleanup in the production account, runs the import in the non-production account, and coordinates the two with a cross-account Amazon EventBridge event keeps either account from needing a role to act on the other’s resources. But building that orchestration is a larger exercise than this post covers. 
 
About the author 
 
  
  
   
   
  Mansi Panchal 
  Mansi Panchal is a Senior Database Specialist Solutions Architect at AWS, where she works with customers on Amazon Aurora architecture, performance, and operational automation. She is particularly interested in helping multi-tenant SaaS teams build secure, repeatable data operations across isolated environments. 
  
</description>
    <content:encoded><![CDATA[<p>Multi-tenant software as a service (SaaS) applications regularly need to copy one tenant’s data from production into a lower environment. Teams do this to reproduce a defect, validate a schema change, or give quality assurance (QA) a representative dataset. When production and non-production run in separate AWS accounts, their virtual private clouds (VPCs) often have no route to each other. That turns the copy into a networking problem as much as a data problem, and security teams are rightly reluctant to open a path between the two.</p> 
<p>In this post, we show how to move a single tenant’s data between Amazon Aurora MySQL-Compatible Edition clusters in two separate AWS accounts, using Aurora cloning, <code>SELECT INTO OUTFILE S3</code>, and <code>LOAD DATA FROM S3</code>. The databases don’t connect directly. Each account reaches Amazon Simple Storage Service (Amazon S3) independently through its own gateway VPC endpoint, and the accounts share only one staging bucket as their touchpoint.</p> 
<p>This post builds on two others. <a href="https://aws.amazon.com/blogs/database/managed-database-backup-and-recovery-in-a-multi-tenant-saas-application/" target="_blank" rel="noopener">Managed database backup and recovery in a multi-tenant SaaS application</a> covers segregating a tenant across the silo, bridge, and pool models. In its terms, what follows is extraction-phase segregation in a pool model. <a href="https://aws.amazon.com/blogs/database/best-practices-for-exporting-and-importing-data-from-amazon-aurora-mysql-to-amazon-s3/" target="_blank" rel="noopener">Best practices for exporting and importing data from Amazon Aurora MySQL to Amazon S3</a> covers the export and import mechanics. What’s new here is moving a tenant-scoped extract between accounts with no network path, plus the access and encryption model that keeps the staging bucket safe.</p> 
<p>Many SaaS teams handle this with a logical dump-and-load pipeline that runs for hours per tenant and can’t resume after a failure. This pattern replaces that single long-running job with independent per-table steps: tables export and import in parallel, and a failed table is retried on its own instead of restarting the whole refresh.</p> 
<h3>Why not cross-account cloning?</h3> 
<p>If you’re moving data between accounts, cross-account cloning through AWS Resource Access Manager (AWS RAM) is the AWS-recommended approach, and it’s worth evaluating first. It’s the right tool for copying a whole cluster into another account, such as a migration or a full-environment refresh. It doesn’t fit the job in this post, for three reasons. We’re moving a <strong>single tenant’s</strong> data, not the entire cluster volume with every tenant on it. We’re loading an <strong>existing non-production cluster</strong> that QA already points at, not standing up a new one. And our source cluster is encrypted with the default RDS key, which cross-account cloning can’t share. So the rest of this post builds a different pattern.</p> 
<h2>Solution overview</h2> 
<p>The pattern has four phases. The central idea is that Amazon S3 is a Regional service reachable from inside a VPC through a gateway endpoint without traversing the internet, so two VPCs with no route between them can both reach the same bucket.</p> 
<p>The following diagram shows the data flow. The two clusters have no direct connection. Every interaction between the accounts happens through the staging bucket.</p> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5886-1.png" target="_blank" rel="noopener"><img loading="lazy" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5886-1.png" alt="Architecture diagram of a cross-account tenant refresh. The production account contains an Aurora MySQL source cluster, an ephemeral clone, and the Amazon S3 staging bucket it owns. The non-production account contains the Aurora MySQL target cluster. There is no network route between the accounts, and both accounts and the bucket are in the same Region. Numbered arrows show the data flow: 1, the production cluster is cloned; 2, the clone exports one tenant’s rows too the S3 bucket; and 3, the non-production cluster imports those rows from the bucket. A cleanup step (not shown) then removes the clone and the exported objects." width="800" height="451"></a>
 <p class="wp-caption-text">Figure 1: Data flow for a cross-account tenant refresh through a shared Amazon S3 staging bucket</p>
</div> 
<ol type="1"> 
 <li><strong>Clone the production cluster –</strong> Aurora cloning is copy-on-write at the storage layer, so creation takes minutes largely independent of database size. The clone is a separate cluster in the production account that you can drive hard without affecting production.</li> 
 <li><strong>Export the tenant’s rows to Amazon S3 –</strong> On the clone, run <code>SELECT INTO OUTFILE S3</code> with a predicate on the tenant ID. Aurora writes delimited text files to the bucket, splitting into part files past a threshold (6 GB by default) and emitting a manifest.</li> 
 <li><strong>Import into the non-production cluster –</strong> <code>LOAD DATA FROM S3</code> reads the manifest from that same bucket and ingests the part files.</li> 
 <li><strong>Clean up –</strong> Delete the clone and the exported objects.</li> 
</ol> 
<p>Because each table exports and imports as an independent unit, both phases parallelize across tables, and a failure affects only that table.</p> 
<p>This pattern changes the shape of the connectivity requirement rather than removing it. There is no VPC peering, transit gateway, or VPC-to-VPC path between the accounts. Each account needs only an S3 gateway VPC endpoint so its cluster can reach S3 privately. Access between the accounts lives in resource policy, not routing, so production doesn’t become reachable from the lower environment. That’s what makes the design straightforward to clear through security review.</p> 
<h2>Prerequisites</h2> 
<p>Have the following in place before you start, in roughly this setup order:</p> 
<ol type="1"> 
 <li><strong>Two AWS accounts</strong> (one production, one non-production), each with its own VPC and no network route between them. (A third shared account can own the staging bucket. This post keeps the bucket in the production account.)</li> 
 <li><strong>The AWS Command Line Interface (AWS CLI)</strong>, configured with credentials in each account that can manage Amazon Relational Database Service (Amazon RDS), Amazon S3, AWS Identity and Access Management (IAM), and AWS Key Management Service (AWS KMS).</li> 
 <li><strong>Aurora MySQL-Compatible Edition, version 3 (MySQL 8.0-compatible) or later</strong> on both clusters. This post uses the Aurora MySQL statements <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.SaveIntoS3.html" target="_blank" rel="noopener">SELECT INTO OUTFILE S3</a> and <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.LoadFromS3.html" target="_blank" rel="noopener">LOAD DATA FROM S3</a>. (Aurora PostgreSQL offers an equivalent path through the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/postgresql-s3-export.html" target="_blank" rel="noopener">aws_s3 extension</a>. The cross-account bridge applies there too, but the SQL in this post is MySQL-specific.)</li> 
 <li><strong>An existing non-production cluster</strong> with a schema that already matches production. <code>LOAD DATA</code> carries rows, not data definition language (DDL), so the destination tables must exist with the same structure. Reconcile any schema drift before you load.</li> 
 <li><strong>An index on the tenant ID column</strong> in every participating table. Without one, the <code>WHERE</code> clause degrades to a full table scan. See <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.BestPractices.html" target="_blank" rel="noopener">Optimizing and tuning queries in Amazon Aurora MySQL</a>.</li> 
 <li><strong>An S3 gateway VPC endpoint in each VPC</strong>, so each cluster can reach Amazon S3 privately without a route between the two VPCs.</li> 
 <li><strong>Both clusters and the staging bucket in the same AWS Region.</strong> A clone can’t be created in an AWS Region different from its source, so the entire pattern operates within a single Region. See <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Clone.html" target="_blank" rel="noopener">Cloning an Amazon Aurora DB cluster volume</a>.</li> 
 <li><strong>A maintenance window for the first-time parameter change on the non-production cluster.</strong> Associating the IAM role uses the static DB cluster parameter <code>aws_default_s3_role</code>, which takes effect only after the writer reboots. (The per-refresh clone avoids this reboot by associating the role before its writer starts. See Step 1.) See <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithDBClusterParamGroups.html" target="_blank" rel="noopener">Working with DB cluster parameter groups</a>.</li> 
</ol> 
<h2>Walkthrough</h2> 
<p>The walkthrough has five parts. First, configure the cross-account bridge once. Then run the four phases of each refresh: clone, export, import, and clean up. The bridge is the one-time setup. The four phases are what you repeat on every refresh.</p> 
<h3>Configure the cross-account bridge</h3> 
<p>Set this up once. The bridge is made of three things, and no network route is among them. The production account owns an encrypted S3 bucket and grants the non-production account’s import role read access to it. Each cluster has an IAM role that lets Aurora read from or write to S3 on its behalf. And a customer managed AWS KMS key, held in the production account, grants decrypt permission to both roles so the cross-account read can succeed. The following sections configure each piece in turn: the Aurora roles, the bucket ownership and policy, the encryption key, and the data-store controls on the bucket.</p> 
<h4>Wire up the Aurora roles</h4> 
<p>Each statement needs an IAM role associated with its cluster, that role named in a DB cluster parameter, and a database grant to the issuing user. The <a href="https://aws.amazon.com/blogs/database/best-practices-for-exporting-and-importing-data-from-amazon-aurora-mysql-to-amazon-s3/" target="_blank" rel="noopener">export/import best practices post</a> covers the role’s permissions in detail.</p> 
<p>In this one-time setup, wire up the <strong>non-production (import) cluster</strong> only. The clone is created fresh each refresh, so its export-role association belongs in Step 1:</p> 
<div class="hide-language"> 
 <pre><code class="language-bash"># Non-production account — import
aws rds add-role-to-db-cluster \
  --db-cluster-identifier nonprod-cluster \
  --role-arn arn:aws:iam::444455556666:role/AuroraS3ImportRole</code></pre> 
</div> 
<p>Then set <code>aws_default_s3_role</code> in the cluster’s parameter group to the role’s Amazon Resource Name (ARN) and reboot the writer (it’s a static parameter). On Aurora MySQL version 3 and 8.4 this is the parameter to use. The statement-specific <code>aurora_select_into_s3_role</code> and <code>aurora_load_from_s3_role</code> aren’t available there.</p> 
<p>Finally, grant the import user the right to issue the statement. On version 3 and 8.4 this is a role, not a privilege:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">-- Non-production cluster
GRANT AWS_LOAD_S3_ACCESS TO 'refresh_user'@'%';</code></pre> 
</div> 
<p>The matching <code>GRANT AWS_SELECT_S3_ACCESS</code> runs on the clone in Step 1. Because these are roles, you must activate them: set <code>activate_all_roles_on_login</code> or issue <code>SET ROLE ALL</code> per session. Otherwise a pipeline opening a fresh connection per table fails on permissions while <code>SHOW GRANTS</code> looks correct.</p> 
<p>Two export-role permissions caused a misleading <code>Access Denied</code> in our testing. First, don’t put an <code>s3:prefix</code> condition on the export role’s <code>s3:ListBucket</code>. <code>SELECT INTO OUTFILE S3</code> lists the bucket without a matching prefix, so the condition denies the export. Scope the object actions to the prefix, but leave <code>ListBucket</code> and <code>GetBucketLocation</code> unconditioned. (The import role is the exception: <code>LOAD DATA FROM S3 MANIFEST</code> reads a specific key, so its <code>ListBucket</code> can keep the prefix condition.) Second, with an S3 Bucket Key enabled, the export role also needs <code>kms:Decrypt</code>.</p> 
<h4>Decide which account owns the bucket</h4> 
<p>Put the staging bucket in the <strong>production account</strong>, the account that writes the objects. Because the writer is also the bucket owner, S3 Object Ownership doesn’t come into question. Ownership mismatch is a common failure with cross-account buckets. This bucket policy is the connectivity step for the pattern: it grants the non-production import role read access across the account boundary, scoped to the refresh prefix. Scope the grant to a role ARN <em>and</em> bound it with a condition key: <code>aws:PrincipalOrgID</code> (shown here), or <code>aws:PrincipalAccount</code> with the non-production account ID if you don’t use AWS Organizations. That way the grant can’t be exercised outside that boundary even if the role’s trust is later widened. Because this bucket holds production tenant data, the ARN alone isn’t a sufficient boundary:</p> 
<div class="hide-language"> 
 <pre><code class="language-json">{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListRefreshPrefixOnly",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::444455556666:role/AuroraS3ImportRole"},
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::amzn-s3-demo-tenant-refresh",
      "Condition": {
        "StringLike": {"s3:prefix": "refresh-*"},
        "StringEquals": {"aws:PrincipalOrgID": "o-exampleorgid"}
      }
    },
    {
      "Sid": "ReadRefreshObjects",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::444455556666:role/AuroraS3ImportRole"},
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::amzn-s3-demo-tenant-refresh/refresh-*",
      "Condition": {
        "StringEquals": {"aws:PrincipalOrgID": "o-exampleorgid"}
      }
    }
  ]
}</code></pre> 
</div> 
<p>If you instead put the bucket in the non-production account, check its S3 Object Ownership setting. Buckets created after April 2023 default to <strong>Bucket owner enforced</strong>, which disables ACLs and gives the owner every object, so the import works. On an older ACL-enabled bucket the reader gets <code>AccessDenied</code>, and because <code>SELECT INTO OUTFILE S3</code> can’t set a <code>bucket-owner-full-control</code> ACL, the fix is to set Bucket owner enforced.</p> 
<h4>Encrypt with a customer managed key</h4> 
<p>Encrypt the bucket with SSE-KMS using a customer managed key. This is a requirement of the cross-account design, not a hardening step: the AWS managed <code>aws/s3</code> key policy can’t be modified to grant a cross-account principal the <code>kms:Decrypt</code> the import needs. Keep the key in the production account. Its key policy must grant <code>kms:GenerateDataKey</code> and <code>kms:Decrypt</code> to the export role and <code>kms:Decrypt</code> to the import role. A missing grant here is the most common failure in cross-account variants of this pattern.</p> 
<p>On Aurora MySQL 3.05 and later you can also name the key in the statement with an <code>ENCRYPTION SSE_KMS 'cmk_id'</code> clause, so the encryption is visible in the SQL rather than resting on a bucket setting.</p> 
<h4>Treat the staging bucket as a production data store</h4> 
<p>The bucket holds production tenant data as plain text, so it inherits production’s requirements:</p> 
<ul> 
 <li>Block Public Access.</li> 
 <li>A bucket policy <code>Deny</code> on <code>aws:SecureTransport: false</code>.</li> 
 <li>AWS CloudTrail data events on the refresh prefix.</li> 
 <li>A dedicated bucket for staging.</li> 
 <li>Explicit deletion after a successful import, with a 24-hour Lifecycle rule as a backstop rather than the primary mechanism.</li> 
</ul> 
<p>With the bridge configured, you run the four phases on each refresh. The following steps walk through each phase with the commands to run and the pitfalls to avoid.</p> 
<h3>Step 1: Create the clone</h3> 
<p>Cloning is a point-in-time restore with copy-on-write semantics. Pass the cluster parameter group that names <code>aws_default_s3_role</code>. Without <code>--db-cluster-parameter-group-name</code>, the clone comes up on the default parameter group, which has no role set, and the export fails:</p> 
<div class="hide-language"> 
 <pre><code class="language-bash">aws rds restore-db-cluster-to-point-in-time \
  --source-db-cluster-identifier prod-cluster \
  --db-cluster-identifier tenant-refresh-clone \
  --restore-type copy-on-write \
  --use-latest-restorable-time \
  --vpc-security-group-ids sg-0abc123 \
  --db-subnet-group-name prod-subnet-group \
  --db-cluster-parameter-group-name tenant-refresh-s3-params</code></pre> 
</div> 
<p>This restores the cluster volume only. The clone doesn’t inherit the source’s IAM role association, so associate the export role <strong>before</strong> creating the writer. In our testing, this ordering let the writer read <code>aws_default_s3_role</code> at first start and the export ran without a reboot:</p> 
<div class="hide-language"> 
 <pre><code class="language-bash"># Associate the export role with the clone (it is not inherited from the source)
aws rds add-role-to-db-cluster \
  --db-cluster-identifier tenant-refresh-clone \
  --role-arn arn:aws:iam::111122223333:role/AuroraS3ExportRole

# Then create the writer, sized for export throughput rather than steady-state workload
aws rds create-db-instance \
  --db-instance-identifier tenant-refresh-clone-writer \
  --db-cluster-identifier tenant-refresh-clone \
  --db-instance-class db.r6g.8xlarge \
  --engine aurora-mysql</code></pre> 
</div> 
<p>Confirm the association is active before exporting. <code>describe-db-clusters</code> should show <code>"Status": "ACTIVE"</code> under <code>AssociatedRoles</code>. If you associate the role <em>after</em> the writer is already running, the first export fails with <code>Missing Credentials: Cannot instantiate S3 Client</code>. Reboot the writer to clear it. Then grant the export role to the database user on the clone:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">-- On the clone
GRANT AWS_SELECT_S3_ACCESS TO 'refresh_user'@'%';</code></pre> 
</div> 
<h3>Step 2: Export the tenant’s data</h3> 
<p>Run one statement per table, concurrently:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SELECT * FROM orders
  WHERE tenant_id = 4711
  INTO OUTFILE S3 's3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/orders'
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
  LINES TERMINATED BY '\n'
  MANIFEST ON
  OVERWRITE ON;</code></pre> 
</div> 
<p>On success, the statement returns a rows-affected count (<code>Query OK, 4821233 rows affected</code>). <code>MANIFEST ON</code> writes an <code>orders.manifest</code> JSON file listing every part file, which the import consumes directly:</p> 
<div class="hide-language"> 
 <pre><code class="language-json">{
  "entries": [
    { "url": "s3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/orders.part_00000" },
    { "url": "s3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/orders.part_00001" }
  ]
}</code></pre> 
</div> 
<p><code>OVERWRITE ON</code> makes retries idempotent. In our testing, 8–20 concurrent exports saturated instance network bandwidth, but the ceiling depends on instance size and table width, so measure rather than assume. Output is uncompressed text split into part files at the 6 GB default threshold, and parallelism is across tables, not within a table. See Considerations and limitations for the large-single-table case.</p> 
<p><strong>Scope every export to exactly one tenant.</strong> This is the correctness property the whole pattern rests on, and looking for a <code>tenant_id</code> column isn’t sufficient to guarantee it. Classify every table into one of three buckets, and treat anything unclassified as a blocker:</p> 
<ul> 
 <li><strong>Tenant-scoped:</strong> carries the tenant ID. The preceding predicate applies directly.</li> 
 <li><strong>Tenant-derived:</strong> holds tenant-owned rows but has no tenant column, typically a child table keyed to its parent. Copy it wholesale and you export every tenant’s data. Skip it and the refresh is incomplete. Reach it through a join: 
  <div class="hide-language"> 
   <pre><code class="language-sql">SELECT li.* FROM line_items li
  JOIN orders o ON o.order_id = li.order_id
  WHERE o.tenant_id = 4711
  INTO OUTFILE S3 's3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/line_items'
  ...</code></pre> 
  </div> </li> 
 <li><strong>Shared:</strong> reference data common to all tenants, safe to copy in full, but as a named allowlist rather than by omission.</li> 
</ul> 
<p>Generate these statements from the classification rather than hand-authoring predicates, and make isolation a <em>preventive</em> control that runs before any S3 write. But check what can actually go wrong, not a tautology. Asserting <code>COUNT(DISTINCT tenant_id) = 1</code> on a query already filtered by <code>WHERE tenant_id = 4711</code> consistently passes and proves nothing. The real failure modes are an unclassified table, a tenant-derived table exported without its join (which leaks every tenant’s rows), and a wrong join path. Guard those:</p> 
<ul> 
 <li><strong>Classification coverage.</strong> Assert that every base table in <code>information_schema.tables</code> appears in your classification. Treat an unclassified table as a hard stop. It’s the most likely way to leak or ship an incomplete tenant.</li> 
 <li><strong>Tenant-derived join correctness.</strong> For each tenant-derived table, compare the join’s row count against the count expected by walking from the parent (for example, <code>line_items</code> whose <code>order_id</code> belongs to the tenant’s <code>orders</code>). A mismatch means the join path is wrong.</li> 
</ul> 
<p>A failing check aborts the export, so over-scoped data doesn’t reach the lower environment. Checking only after import is too late.</p> 
<h3>Step 3: Import into non-production</h3> 
<p>The non-production cluster is a long-lived environment that other tenants share, so you can’t load into it blind. Prepare the target first, then load.</p> 
<p><strong>Prepare the target: remove the tenant’s existing rows.</strong> Before loading, delete the tenant’s current rows from the non-production tables, in child-to-parent order so foreign keys stay satisfied. Use a tenant-scoped <code>DELETE</code>, never <code>TRUNCATE</code>. <code>TRUNCATE</code> empties the whole table and would wipe every other tenant’s rows:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">-- Child (tenant-derived) tables first, via the join to the parent
DELETE li FROM line_items li
  JOIN orders o ON o.order_id = li.order_id
  WHERE o.tenant_id = 4711;

-- Then the tenant-scoped parent tables
DELETE FROM orders WHERE tenant_id = 4711;</code></pre> 
</div> 
<p><strong>Assume keys can’t collide.</strong> This load assumes the production primary keys for this tenant won’t collide with rows other tenants already own in non-production. That holds when non-production was originally seeded from production (so the key spaces line up), or when the schema uses UUIDs or non-overlapping auto-increment ranges per environment. If non-production auto-increment sequences have drifted independently from production’s, reconcile that before loading. Otherwise <code>LOAD DATA</code> can hit duplicate-key errors or, with <code>unique_checks</code> off, silently create duplicates.</p> 
<p><strong>Handle shared tables deliberately.</strong> Reference tables shared across all tenants aren’t tenant-scoped. Loading them “in full” overwrites data every other non-production tenant depends on. Treat shared tables as a separate decision: skip them if non-production already has the correct reference data, or reconcile them explicitly (for example, upsert only new reference rows) rather than overwriting.</p> 
<p>With the tenant’s old rows removed, load one <code>LOAD DATA FROM S3</code> statement per table, reading the manifests the export wrote. The following example loads the <code>orders</code> table:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SET SESSION foreign_key_checks = 0;
SET SESSION unique_checks = 0;

LOAD DATA FROM S3 MANIFEST
  's3://amzn-s3-demo-tenant-refresh/refresh-2026-08-13/orders.manifest'
  INTO TABLE orders
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
  LINES TERMINATED BY '\n';

SET SESSION unique_checks = 1;
SET SESSION foreign_key_checks = 1;</code></pre> 
</div> 
<p>This is one <code>LOAD DATA FROM S3 MANIFEST</code> per table, matching the per-table export. It’s a matrix of operations, not a single bulk load. Either load parent tables before children, or (as shown) disable <code>foreign_key_checks</code> and <code>unique_checks</code> for the load and re-enable them afterward.</p> 
<p>Be precise about what re-enabling those checks does: it doesn’t re-validate the rows you just loaded. MySQL applies the checks only to new changes and doesn’t rescan existing rows. The load is safe here only because the data was consistent on the clone and parents load before children. Running with <code>unique_checks = 0</code> against a table that already holds other tenants’ rows can also admit duplicates into a secondary unique index without error, so remove the tenant’s existing rows first (above) and make sure production key ranges can’t collide.</p> 
<p>On success, <code>LOAD DATA</code> reports the rows loaded (<code>Query OK, 4821233 rows affected</code>). To speed up InnoDB bulk loads, drop secondary indexes before loading and rebuild them afterward (<code>ALTER TABLE ... DISABLE KEYS</code> is a no-op on InnoDB).</p> 
<h3>Validate the refresh</h3> 
<p>After loading, compare row counts for the tenant against the source clone (not the live primary, whose data might have changed since the clone was taken):</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">-- Run on the clone, then on the non-production target; the counts must match.
SELECT COUNT(*) FROM orders WHERE tenant_id = 4711;   -- e.g. 4821233 on both</code></pre> 
</div> 
<p>Do this for every table. Also confirm every manifest part file was consumed, and that other tenants’ row counts in the shared tables are unchanged. That is the check that proves you refreshed one tenant without disturbing the rest.</p> 
<h3>Step 4: Clean up</h3> 
<p>Delete the cloned cluster and its instance, then the exported objects. An orphaned clone bills for every storage page it diverges from the source, and an orphaned export leaves tenant data at rest.</p> 
<h2>Performance, cost, and failure handling</h2> 
<p>Refresh time depends on your data, so measure a representative tenant before you commit to a service level agreement (SLA). Four things drive it. Export runs in parallel across tables but not within one, so the largest table sets the wall-clock time. The export is uncompressed text, so the full data volume moves through Amazon S3. Import time includes both the load and any secondary index rebuilds. And the instance sizes of the clone writer and the target writer cap throughput on each side, so size both for the refresh window rather than steady-state workload.</p> 
<p>Speed isn’t the main win, though. Under a monolithic dump-and-load, a failure discards all completed work. Here, a failed export affects one table while the rest sit durable in Amazon S3, and you retry a failed import one table at a time (delete the tenant’s rows, as in Step 3, then re-read the manifest). Because all export work runs against an ephemeral clone, a pathological job can’t degrade live workloads.</p> 
<h2>Considerations and limitations</h2> 
<p>Weigh these before adopting the pattern:</p> 
<ul> 
 <li><strong>Tenant shape matters.</strong> Export parallelizes across tables but not within one, so wall-clock time depends on the single largest table, not the aggregate size. A tenant spread across many tables refreshes far faster than one dominated by a single large table. If one table holds most of the volume, shard its export by primary-key range.</li> 
 <li><strong>Transient storage.</strong> The export is uncompressed text, so a large tenant lands a large volume in Amazon S3 for the life of the refresh. Plan for it, and delete on completion.</li> 
 <li><strong>Right-size the clone.</strong> Export throughput is often network-bound, so check <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Concepts.DBInstanceClass.Summary.html" target="_blank" rel="noopener">DB instance networking figures</a> rather than reusing the production class. The clone exists only for the refresh.</li> 
 <li><strong>Clone limit.</strong> A cluster supports up to 15 copy-on-write clones. The sixteenth becomes a full copy: slower, full storage, no error. Delete each clone after its refresh. See <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Clone.html#Aurora.Managing.Clone.Limitations" target="_blank" rel="noopener">Limitations of Aurora cloning</a>.</li> 
 <li><strong>Protect personal data.</strong> Encryption isn’t the whole story. Production tenant data might contain personal or regulated data, so confirm your customer agreements and compliance permit copying it into non-production, and mask or anonymize sensitive columns on the clone before <code>SELECT INTO OUTFILE S3</code> runs. That keeps unmasked data out of the bucket.</li> 
 <li><strong>When another approach fits better.</strong> <a href="https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html" target="_blank" rel="noopener">AWS Database Migration Service (AWS DMS)</a> is the choice for a one-time migration or change data capture, though cross-account it needs a cross-VPC path through AWS PrivateLink or AWS Transit Gateway. <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Clone.Cross-Account.html" target="_blank" rel="noopener">Cross-account cloning through AWS RAM</a> fits copying a whole cluster rather than one tenant.</li> 
</ul> 
<h2>Conclusion</h2> 
<p>In this post, we showed how to refresh a single tenant’s data from a production account into an isolated non-production account with no connectivity between them. By chaining Aurora cloning, <code>SELECT INTO OUTFILE S3</code>, and <code>LOAD DATA FROM S3</code>, we turned a cross-account data movement problem into a Regional one: the clusters don’t connect, and one encrypted staging bucket is all they share. Because the pattern reads from an isolated clone and retries at the table level, it can help improve operational stability when failures occur.</p> 
<p>To go deeper, see <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Clone.html" target="_blank" rel="noopener">Cloning an Amazon Aurora DB cluster volume</a>, <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.SaveIntoS3.html" target="_blank" rel="noopener">Saving data into text files in Amazon S3</a>, and <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.LoadFromS3.html" target="_blank" rel="noopener">Loading data from text files in an Amazon S3 bucket</a>.</p> 
<p>After you’ve run the refresh by hand a few times, the natural next step is to automate it. The manual steps map onto <a href="https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html" target="_blank" rel="noopener">AWS Step Functions</a> and <a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html" target="_blank" rel="noopener">AWS Lambda</a> so a refresh runs on a schedule with nothing billed between runs. A design that keeps export and cleanup in the production account, runs the import in the non-production account, and coordinates the two with a cross-account <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html" target="_blank" rel="noopener">Amazon EventBridge</a> event keeps either account from needing a role to act on the other’s resources. But building that orchestration is a larger exercise than this post covers.</p> 
<hr> 
<h2>About the author</h2> 
<footer> 
 <div class="blog-author-box"> 
  <div class="blog-author-image">
   <img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5886-2.jpg" alt="Mansi Panchal" width="100" height="133">
  </div> 
  <h3 class="lb-h4">Mansi Panchal</h3> 
  <p>Mansi Panchal is a Senior Database Specialist Solutions Architect at AWS, where she works with customers on Amazon Aurora architecture, performance, and operational automation. She is particularly interested in helping multi-tenant SaaS teams build secure, repeatable data operations across isolated environments.</p> 
 </div> 
</footer>]]></content:encoded>
    <pubDate>Wed, 11 Nov 2026 22:08:37 +0000</pubDate>
    <dc:creator>AWS Database Blog</dc:creator>
    <category>Advanced (300)</category>
    <category>Amazon Aurora</category>
    <category>Amazon Simple Storage Service (S3)</category>
    <category>Technical How-to</category>
  </item>

  <item>
    <title>How Oracle changed the future of MySQL: Dominic Preuss on Inside the Silicon Mind</title>
    <guid isPermaLink="false">6af4a3873a14a1000188058c</guid>
    <link>https://villagesql.com/blog/silicon-mind/</link>
    <description>Shopify, GitHub, and Uber run on MySQL, yet, as Dominic Preuss puts it, the hype, innovation, and investment historically went to PostgreSQL. Inside the Silicon Mind published Firas Sozan's interview with Dominic, co-founder and CEO of VillageSQL, about why that happened, what Oracle is doing to address it, and how VillageSQL plans to bring innovation back to MySQL.Dominic covers how MySQL went from a Swedish startup to Oracle, why Oracle is now opening MySQL development to the community, including AWS and Google, and why he started a company on MySQL when everyone was running in the direction of PostgreSQL.Watch more VillageSQL talks on the Talks page, or get started at villagesql.com.</description>
    <content:encoded><![CDATA[<p>Shopify, GitHub, and Uber run on MySQL, yet, as Dominic Preuss puts it, the hype, innovation, and investment historically went to PostgreSQL. Inside the Silicon Mind published <a href="https://www.youtube.com/watch?v=iAiaAQW1jEo">Firas Sozan's interview</a> with Dominic, co-founder and CEO of VillageSQL, about why that happened, what Oracle is doing to address it, and how VillageSQL plans to bring innovation back to MySQL.</p><figure class="kg-card kg-embed-card"></figure><p>Dominic covers how MySQL went from a Swedish startup to Oracle, why Oracle is now opening MySQL development to the community, including AWS and Google, and why he started a company on MySQL when everyone was running in the direction of PostgreSQL.</p><p>Watch more VillageSQL talks on the <a href="https://villagesql.com/talks">Talks page</a>, or get started at <a href="https://villagesql.com/">villagesql.com</a>.</p>]]></content:encoded>
    <pubDate>Wed, 11 Nov 2026 19:17:59 +0000</pubDate>
    <dc:creator>VillageSQL</dc:creator>
    <category>Open Source</category>
    <category>VillageSQL</category>
    <category>MySQL</category>
    <category>AI</category>
  </item>

  <item>
    <title>Help prevent out-of-memory restarts on Aurora MySQL 8.4</title>
    <guid isPermaLink="false">685702552467a817ac2bc486472882121136c392</guid>
    <link>https://aws.amazon.com/blogs/database/help-prevent-out-of-memory-restarts-on-aurora-mysql-8-4/</link>
    <description>When an Amazon Aurora MySQL-Compatible Edition instance runs critically low on memory, the operating system terminates the mysql process. This drops every connection, rolls back in-flight transactions, and triggers a cluster restart. The instinctive fixes (move to a larger instance, or lower max_connections) treat the symptom, not the underlying memory pressure your workload generates. 
Aurora has long included out-of-memory (OOM) avoidance that watches system memory and takes recovery actions before the OS steps in. On Aurora MySQL version 3, use the aurora_oom_response parameter to configure these actions; Aurora provides sensible defaults that you can customize. Aurora MySQL version 8.4 adds a new instance-level parameter, aurora_enable_memory_management, on by default, that can help Aurora choose and adjust recovery actions dynamically based on current conditions. 
For the complete parameter reference, see the Aurora documentation. Here, we reproduce a memory-pressure event, read the signals Aurora exposes while it happens, and watch the feature keep the database alive that would otherwise have restarted. Then we cover the decisions: when to leave automatic management on versus taking manual control, and what changes when you upgrade from version 3 to 8.4. 
Prerequisites 
To follow along, you need: 
 
 An Aurora MySQL DB cluster. Automatic memory management (aurora_enable_memory_management) is available starting in Aurora MySQL version 8.4. On version 3, use the separate aurora_oom_response parameter instead to configure recovery actions manually. 
 Permissions to modify DB parameter groups and to view Amazon CloudWatch metrics. 
 The Performance Schema enabled, so you can inspect the heaviest memory consumers. On Aurora MySQL version 3, Aurora enables memory instrumentation by default once you enable Performance Schema. On both Aurora MySQL versions, enabling Amazon CloudWatch Database Insights automatically turns on Performance Schema. 
 
Reproducing the OOM scenario is optional and should only be done on a non-production test cluster. 
How Aurora MySQL uses memory 
An Aurora MySQL instance divides its memory between shared structures and per-session structures. The largest shared consumer is the InnoDB buffer pool, which Aurora sizes automatically from instance memory. On top of that, every connection reserves memory for its own buffers, and queries that sort, group, or join can allocate internal temporary tables that consume memory when they exceed their limits. 
Three consumers drive almost every OOM event, and they are also where you fix it: 
 
 The buffer pool is sized to the instance and rarely the sole culprit, but it leaves less headroom on smaller classes. 
 Per-connection memory scales with connection count. A fleet that opens thousands of mostly idle connections turns connection growth into memory growth. 
 Temporary tables from large GROUP BY, ORDER BY, DISTINCT, or UNION operations can allocate significant memory per query. 
 
A bigger instance gives you more total memory, so it takes longer for buffer pool growth, connection buffers, or temporary tables to exhaust it. But whichever of the three is driving the pressure keeps growing, and will eventually hit the new, higher ceiling too. Identifying which one is the real driver is what lets you fix the cause instead of only paying for more headroom. 
How Aurora protects itself: The recovery-action ladder 
The OOM avoidance in Aurora escalates through a set of actions, from least to most disruptive. On Aurora MySQL 8.4 with automatic management on, Aurora selects and adjusts these for you. On Aurora MySQL version 3, they are the entries in aurora_oom_response. Some actions are also restricted by deployment type: provisioned clusters run on instances you size yourself, while Aurora Serverless v2 clusters scale capacity automatically and don’t support the two actions that resize memory structures directly. 
 
  
   
   Action 
   What it does 
   Notes 
   
   
   print 
   Logs the memory-intensive queries and connections to the error log. Nothing is terminated. 
   Versions 3 and 8.4. 
   
   
   tune 
   Shrinks internal table caches (table_open_cache, table_definition_cache). These restore automatically when memory recovers. 
   Provisioned only. Not supported on Aurora Serverless v2, which scales capacity automatically instead. 
   
   
   tune_buffer_pool 
   Shrinks the InnoDB buffer pool. It restores automatically when memory recovers. 
   Provisioned instances with 2 vCPUs only. Not supported on Aurora Serverless v2. Must be paired with kill_query or kill_connect or it does nothing. 
   
   
   decline 
   Rejects new queries with an error while memory is low. 
   Versions 3 and 8.4. 
   
   
   kill_query 
   Terminates running SELECT queries, highest memory consumers first. DDL, other DML, and transactions are unaffected. 
   If both kill_query and kill_connect are set, only kill_connect activates. 
   
   
   kill_connect 
   Terminates user connections, rolling back their transactions and terminating DDL. 
   In 3.11+ and 8.4, also terminates connections that try to allocate memory during pressure. 
   
  
 
The ladder matters because it escalates from least to most disruptive: Aurora tunes caches and the buffer pool first, and only terminates queries or connections if pressure continues. Getting that order right by hand, along with constraints such as tune_buffer_pool needing a terminate action to take effect, is a common pitfall. That is exactly what automatic management removes. 
Aurora MySQL version 3 today, and what changes when you upgrade to version 8.4 
On version 3, Aurora auto-configures aurora_oom_response from engine version, instance class, and memory. Per the Aurora documentation on out-of-memory issues, smaller instances on 3.08+ default to a protective set like print,tune,tune_buffer_pool,decline,kill_connect, and from 3.09 larger instances default to print,decline,kill_connect. Defaults vary by engine version and instance class, so check the effective value with SHOW GLOBAL STATUS LIKE 'aurora_oom%' rather than assuming. 
The upgrade to version 8.4 introduces one behavior worth planning for. When aurora_enable_memory_management is ON (the default), your existing aurora_oom_response is ignored, which is expected and generally an improvement. But if you turn automatic management OFF on version 8.4, aurora_oom_response is empty by default, unlike version 3’s populated defaults. A version 8.4 cluster with automatic management disabled and nothing configured can therefore have fewer recovery actions enabled than the version 3 cluster it was upgraded from. If you plan to run with manual control on version 8.4, set aurora_oom_response explicitly as part of the upgrade, and validate the memory-health metrics on a non-production cluster first. 
Reproduce and read a memory-pressure event 
A parameter list doesn’t show you what an incident looks like. This section forces one on a test cluster, reads the signals Aurora exposes in order, and then shows the same workload with protection turned off so you can see what the feature is actually helping to prevent. 
Force memory pressure 
On a non-production Aurora MySQL version 8.4 test cluster (here a db.r7g.large with 16 GB and a table of about 6.1 million rows, roughly 3.7 GB in size), drive many concurrent sessions through large in-memory sorts. Each session gets its own sort buffer, the per-connection memory MySQL uses to sort result rows for operations like ORDER BY when no index can produce the order directly. Give each session a large sort buffer, then run a full-table ORDER BY on an unindexed column, from roughly 40 sessions at once. The exact number that triggers pressure varies by engine patch version and instance state. Too few sessions do not saturate memory, and too many can overwhelm the instance faster than automatic management can shed load, so ramp up gradually (for example 40, then 60, then 100) until AuroraMemoryHealthState climbs above 0: 
 
 SET SESSION sort_buffer_size = 1073741824; -- 1 GB per session
SELECT id, grp, rand_val, payload FROM big_table ORDER BY rand_val DESC; 
 
The instance has 16 GB, but 40 sessions each able to claim up to a 1 GB sort buffer can demand up to 40 GB, far more memory than exists. As concurrency rises, two CloudWatch metrics move together: FreeableMemory (the memory on the instance available for new allocations, including OS and database caches) falls, and SwapUsage (how much of that memory has spilled to disk-backed swap because physical memory ran out) climbs. On its own, falling FreeableMemory isn’t an alarm. MySQL and the OS use free memory for caching. The pattern that signals danger is a rapid FreeableMemory decline combined with rising SwapUsage. Under this load, between 20:51 and 20:54 UTC (the same window AuroraMemoryHealthState spiked, see Figure 3), FreeableMemory repeatedly fell from a steady approximately 6.7 GB down to as low as 1.3 GB while automatic management held the line. SwapUsage rose to roughly 184 MB. 

 
 Figure 1: CloudWatch FreeableMemory for the instance. It holds steady near 6.7 GB, then dips as low as 1.3 GB between 20:51 and 20:54 UTC as the workload saturates memory, recovering each time automatic management sheds load. The run with automatic management turned off (described in the following section) shows a different shape: a single sharp drop followed by the OS terminating the process.
 

 
 Figure 2: CloudWatch SwapUsage over the same window, rising to about 184 MB as memory tightens.
 
Read the signals in order 
When memory tightens, Aurora exposes the event through a consistent set of signals. Reading them in order tells you what happened and what to do. 
1. Start with memory health. AuroraMemoryHealthState reports 0 (healthy), 5 (moderate pressure), or 10 (critical). Find it in the CloudWatch console under the Amazon Relational Database Service (Amazon RDS) metrics for your DB instance, or query it directly with the CloudWatch GetMetricData API. A move from 0 to 5 to 10 is the event beginning. Alarm on any sustained value above 0 so you see pressure before a restart, not after. 

 
 Figure 3: CloudWatch AuroraMemoryHealthState. It sits flat at 0 and spikes to 5 (moderate pressure) at the moment the workload saturates memory.
 
2. Confirm which actions are in effect. Check the runtime OOM status. Connect to the instance with a MySQL client (for example, the mysql command-line tool or MySQL Workbench) and run this on an Aurora MySQL version 8.4.8 instance at rest: 
 
 SHOW GLOBAL STATUS LIKE 'aurora_oom%'; 
 
 
 +-------------------------------------+----------------------------+
| Variable_name | Value |
+-------------------------------------+----------------------------+
| Aurora_oom_response | print,decline,kill_connect |
| aurora_oom_avoidance_recovery_state | INACTIVE |
+-------------------------------------+----------------------------+ 
 
Aurora_oom_response shows the actions Aurora is actually using (here print,decline,kill_connect), chosen automatically even though the settable aurora_oom_response parameter is empty on version 8.4 (see the preceding version 8.4 section). aurora_oom_avoidance_recovery_state is INACTIVE when memory is healthy. Polling the same query under load shows the transition mid-event: 
 
 +-------------------------------------+----------------------------+
| Variable_name | Value |
+-------------------------------------+----------------------------+
| Aurora_oom_response | print,decline,kill_connect |
| aurora_oom_avoidance_recovery_state | ACTIVE |
+-------------------------------------+----------------------------+ 
 
ACTIVE means Aurora is intervening right now. 
3. See what Aurora did during the event. Under the load, AuroraMemoryHealthState rose from 0 to 5 and Aurora began shedding work. Applications saw the recovery actions directly, in their own database client’s error output, as MySQL error 1041: 
 
 (1041, 'Query declined - system memory is critically low. This action was taken to protect system stability.')
(1041, 'Connection terminated - system memory is critically low. This action was taken to protect system stability.') 
 
Those counts are also tracked as CloudWatch metrics, AuroraMemoryNumDeclinedSqlTotal and AuroraMemoryNumKillConnTotal, so you don’t need to rely on client-side error counting to know how much work the feature shed. Over this pressure window they climbed in repeated bursts rather than a single steady ramp, matching the error log’s pattern of distinct detect-and-recover cycles a few seconds apart: 

 
 Figure 4: CloudWatch AuroraMemoryNumDeclinedSqlTotal and AuroraMemoryNumKillConnTotal over the pressure window, peaking at 21 declined queries and 14 terminated connections in the 20:52 UTC data point.
 
The number of terminated connections can exceed the number of connections open at a single instant, because clients that get disconnected often reconnect immediately and get caught by the next cycle. The cumulative count across a multi-cycle event reflects that repeated churn, not a one-time mass termination. Those are the decline and kill_connect actions from the ladder doing their job: new and heaviest work is rejected so the instance stays up. Critically, the database did not restart. 
4. Read the error log. Download it from the RDS console (go to your instance, then Logs &amp;amp; events, then Logs) or with aws rds download-db-log-file-portion. Recovery messages are prefixed Aurora memory management: on version 8.4 (and OOM crash avoidance: on version 3), and they name the specific queries and connections involved: 
 
 Aurora memory management: Memory pressure detected. Total: 16087316 KB,
available: 1580244 KB. Memory recovery actions include declining queries,
terminating connections.
Memory report: {&quot;system_report&quot;:{&quot;total_mem_kb&quot;:16087316,&quot;available_mem_kb&quot;:1577088, ...},
&quot;user_connections_memory_usage&quot;:{&quot;connection_count&quot;:5,&quot;current_used_bytes&quot;:5369504400},
&quot;top_memory_consuming_connections&quot;:[{&quot;id&quot;:87,&quot;used_bytes&quot;:1073900880, ...}]}
Aurora memory management: Memory recovered. Actions taken: queries declined: 21,
connections killed: 2. Total: 16087316 KB, available: 1803672 KB. 
 
The memory report names the biggest consumers. Here, 5 surviving user connections held about 5.37 GB between them, each one close to the full 1 GB sort_buffer_size setting. That provides a route from detecting pressure to identifying its root cause, and the paired “Memory recovered” line confirms what the recovery cost: 21 declined queries and 2 terminated connections for this single cycle. 
5. Attribute it to a consumer. Use the Performance Schema memory summary to find the components and connections holding the most memory, then map it back to buffer pool, per-connection memory, or temporary tables: 
 
 SELECT event_name,
current_number_of_bytes_used/1024/1024 AS mb_used
FROM performance_schema.memory_summary_global_by_event_name
ORDER BY current_number_of_bytes_used DESC
LIMIT 10; 
 
 
 +-----------------------------------------------------------------------------+---------+
| event_name | mb_used |
+-----------------------------------------------------------------------------+---------+
| memory/sql/Filesort_buffer::sort_keys | 4064.74 |
| memory/innodb/hash0hash | 93.96 |
| memory/performance_schema/events_statements_summary_by_digest | 40.28 |
| memory/innodb/lock0lock | 19.58 |
| memory/mysys/KEY_CACHE | 16.00 |
| memory/performance_schema/events_statements_history_long | 14.42 |
| memory/performance_schema/events_errors_summary_by_thread_by_error | 14.21 |
| memory/performance_schema/events_statements_summary_by_thread_by_event_name | 11.87 |
| memory/performance_schema/events_statements_history_long.sql_text | 9.77 |
| memory/performance_schema/events_statements_history_long.digest_text | 9.77 |
+-----------------------------------------------------------------------------+---------+ 
 
This output comes from a lighter run of the same query with 4 concurrent sessions, which kept memory below the level where Aurora declines the diagnostic query itself. Filesort_buffer::sort_keys alone accounts for about 4.0 GB, close to 1 GB per session, matching the sort_buffer_size setting. That is roughly 18 times every other component combined, and about 43 times the next-largest single component (hash0hash, at about 94 MB). That single line confirms the cause: the concurrent ORDER BY sorts allocating large per-session sort buffers, not the buffer pool or a leak elsewhere. Under critical pressure, Aurora can decline this diagnostic query along with the workload, so run it as soon as pressure eases, or rely on the error log’s memory report during the peak. 
6. Confirm the outcome. AuroraMemoryHealthState returns to 0 and aurora_oom_avoidance_recovery_state goes back to INACTIVE once the pressure clears. That means Aurora rode out the event. If instead os.memory.outOfMemoryKillCount (a Performance Insights operating system counter) is non-zero, the OS terminated the process and a restart followed. Recovery didn’t keep up, and the workload or instance size needs attention. 
The same workload with protection off 
To see what the feature helps prevent, set aurora_enable_memory_management = OFF with an empty aurora_oom_response, reboot to apply (changing this parameter takes effect only after a reboot), and run the identical workload in a separate test window from the preceding protected run. With no recovery actions, memory pressure runs to exhaustion and the OS terminates the database process: 

 
 Figure 5: CloudWatch FreeableMemory for the same workload with automatic memory management off. Unlike the protected run’s repeated recovery, memory drops once to about 645 MB at 21:25 UTC and then snaps back to baseline as the instance restarts.
 

 
 Figure 6: RDS events for the instance with automatic memory management turned off. The 17:25 restart has no preceding shutdown entry, unlike the planned parameter-group reboot preceding it. The RDS console displays times in your browser’s local time zone (here UTC-04:00), so 17:25 in this figure is the same event as 21:25 UTC in Figure 5.
 
The database was unreachable for about 30 seconds: the new mysqld process started roughly 29 seconds after the load launched and was accepting connections again a few seconds after that. Every connection drops and the cluster restarts. Instead of the graceful 1041 declines, applications see connection resets and “can’t connect” errors. That illustrates how the feature can help maintain database availability by shedding load during memory pressure events. With automatic management back on (the default), the same workload produces the recovery sequence from the previous section: some declined queries and terminated connections, and a database that stays up. That contrast is the whole value of the feature. 
After an OFF test, turn the feature back on, reboot, and verify it: 
 
 SHOW GLOBAL VARIABLES LIKE 'aurora_enable_memory_management'; 
 
 
 +---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| aurora_enable_memory_management | ON |
+---------------------------------+-------+ 
 
Leaving the parameter group on OFF after a test is how an unprotected production cluster happens by accident. 
Decide: Keep automatic management on, or take manual control 
For the large majority of Aurora MySQL version 8.4 workloads, the right choice is the default: leave aurora_enable_memory_management set to ON. Aurora adjusts the actions to conditions, and you avoid the constraints that make manual configuration error-prone. Verifying it’s on and staying on a current patch level is the whole task. 
There are narrower cases for turning it OFF and setting aurora_oom_response yourself: 
 
 You need a deterministic, fixed action set for change-controlled environments where dynamic behavior is undesirable. 
 Your workload is latency-sensitive online transaction processing (OLTP) and you would rather terminate a heavy query (kill_query) than reject all new queries (decline), or the reverse. 
 You must protect a specific class of work and want to avoid kill_connect entirely. 
 
If you do take manual control, two rules matter: pair tune_buffer_pool with kill_query or kill_connect or it won’t resize the pool, and never leave aurora_oom_response empty in production, which disables recovery entirely. 
Aurora Serverless v2 differences 
The preceding reproduction used a provisioned instance. Aurora Serverless v2 responds differently, per the Aurora documentation: it responds to memory pressure first by scaling up Aurora Capacity Units (ACUs) to add memory. If pressure persists while scaling is in progress, Aurora can terminate existing connections to recover. Termination of connections that try to allocate memory happens only once the instance has reached its configured maximum ACU and can scale no further. The tune and tune_buffer_pool actions are not supported on Aurora Serverless v2. The others behave as on provisioned instances. Two implications: size the maximum ACU with real memory headroom so scaling has room to work, and make sure clients reconnect gracefully, since error 1041 can appear here too (see Limitations). 
Take immediate action 
When you catch memory pressure in progress: 
 
 Confirm protection is active. SHOW GLOBAL STATUS LIKE 'aurora_oom%' shows aurora_oom_avoidance_recovery_state = ACTIVE during an event. To check whether automatic management itself is on, use SHOW GLOBAL VARIABLES LIKE 'aurora_enable_memory_management' instead, since that’s a parameter, not a status. 
 Find and stop the heaviest query. Identify it from the error log or Performance Schema and cancel it if it’s non-critical with CALL mysql.rds_kill_query(&amp;lt;id&amp;gt;);. 
 Stabilize connections. Pool with Amazon RDS Proxy so a connection surge doesn’t become a memory surge. 
 Reduce pressure at the source. Index the queries that build large temporary tables, and right-size max_connections to the instance. 
 
Summary 
The following table sums up how OOM protection differs between the two Aurora MySQL versions: 
 
  
   
   Capability 
   Aurora MySQL version 3 
   Aurora MySQL version 8.4 
   
   
   Prevention parameter 
   aurora_oom_response (auto-set defaults) 
   aurora_enable_memory_management (ON by default) 
   
   
   Mechanism 
   Fixed recovery-action ladder 
   Dynamic recovery-action ladder 
   
   
   Default protection 
   Populated defaults by instance class 
   Managed automatically 
   
   
   Manual override 
   Edit aurora_oom_response 
   Set OFF, then set aurora_oom_response (empty by default) 
   
   
   Key metric 
   AuroraMemoryHealthState 
   AuroraMemoryHealthState 
   
  
 
Limitations 
 
 If you turn automatic management OFF on version 8.4, aurora_oom_response is empty by default, so no recovery actions run until you configure them explicitly. 
 tune_buffer_pool is supported on provisioned instances with 2 vCPUs only, and needs to be paired with kill_query or kill_connect. On its own it does nothing. 
 If both kill_query and kill_connect are set, only kill_connect activates. 
 kill_connect rolls back open transactions on the connections it terminates. 
 Applications need retry handling for MySQL error 1041 wherever they connect to Aurora MySQL. A provisioned instance under automatic management can decline queries and terminate connections the same way Aurora Serverless v2 does. 
 The feature aurora_enable_memory_management is available only in Aurora MySQL 8.4. OOM avoidance is specific to Aurora MySQL and does not exist on RDS for MySQL. 
 
Conclusion 
Out-of-memory restarts are preventable, and on Aurora MySQL they’re increasingly avoided for you. We reproduced a memory-pressure event, read it through health state, the recovery counters, and the error log, and saw the same workload restart the database once protection was removed. We then covered when to keep automatic memory management on versus taking manual control, what changes when you move from version 3 to 8.4, and the Aurora Serverless v2 differences. 
A safe posture on version 8.4 is the default: keep aurora_enable_memory_management on, stay current on patches, and alarm on AuroraMemoryHealthState so a protected instance does not quietly becomes a stressed one. And because recovery actions are a safety net, reduce the pressure at its source: pool connections so connection growth doesn’t become memory growth, and index the queries that build large temporary tables. 
To learn more, see Troubleshooting out-of-memory issues for Aurora MySQL databases, Troubleshooting memory usage issues for Aurora MySQL databases, and Best practices for Aurora MySQL performance and scaling. If you have questions or suggestions, leave a comment, or visit AWS re:Post. 
  
 
About the authors 
 
  
  
   
   
  Aayushi Mittal 
  Aayushi is a Technical Account Manager at AWS, where she helps enterprise customers keep Amazon Aurora and Amazon RDS databases healthy and performant. She specializes in memory and performance troubleshooting, and is driven by turning complex operational incidents into clear, actionable guidance her customers can act on with confidence. 
  
  
  
   
   
  Archana Dhere 
  Archana is a Technical Account Manager at AWS, where she partners with enterprise customers to optimize their AWS environments across databases, security, and cloud operations. She specializes in Amazon RDS and Aurora performance and migration planning, and is passionate about translating complex operational challenges into clear, actionable guidance for her customers. 
  
</description>
    <content:encoded><![CDATA[<p>When an <a href="https://aws.amazon.com/rds/aurora/" target="_blank" rel="noopener">Amazon Aurora MySQL-Compatible Edition</a> instance runs critically low on memory, the operating system terminates the mysql process. This drops every connection, rolls back in-flight transactions, and triggers a cluster restart. The instinctive fixes (move to a larger instance, or lower <code>max_connections</code>) treat the symptom, not the underlying memory pressure your workload generates.</p> 
<p>Aurora has long included out-of-memory (OOM) avoidance that watches system memory and takes recovery actions before the OS steps in. On Aurora MySQL version 3, use the <code>aurora_oom_response</code> parameter to configure these actions; Aurora provides sensible defaults that you can customize. <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraMySQLReleaseNotes/AuroraMySQL.Updates.847.html" target="_blank" rel="noopener">Aurora MySQL version 8.4</a> adds a new instance-level parameter, <code>aurora_enable_memory_management</code>, on by default, that can help Aurora choose and adjust recovery actions dynamically based on current conditions.</p> 
<p>For the complete parameter reference, see the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQLOOM.html" target="_blank" rel="noopener">Aurora documentation</a>. Here, we reproduce a memory-pressure event, read the signals Aurora exposes while it happens, and watch the feature keep the database alive that would otherwise have restarted. Then we cover the decisions: when to leave automatic management on versus taking manual control, and what changes when you upgrade from version 3 to 8.4.</p> 
<h2>Prerequisites</h2> 
<p>To follow along, you need:</p> 
<ol type="1"> 
 <li>An Aurora MySQL DB cluster. Automatic memory management (<code>aurora_enable_memory_management</code>) is available starting in Aurora MySQL version 8.4. On version 3, use the separate <code>aurora_oom_response</code> parameter instead to configure recovery actions manually.</li> 
 <li>Permissions to modify DB parameter groups and to view Amazon CloudWatch metrics.</li> 
 <li>The Performance Schema enabled, so you can inspect the heaviest memory consumers. On Aurora MySQL version 3, Aurora enables memory instrumentation by default once you enable Performance Schema. On both Aurora MySQL versions, enabling Amazon CloudWatch Database Insights automatically turns on Performance Schema.</li> 
</ol> 
<p>Reproducing the OOM scenario is optional and should only be done on a non-production test cluster.</p> 
<h2>How Aurora MySQL uses memory</h2> 
<p>An Aurora MySQL instance divides its memory between shared structures and per-session structures. The largest shared consumer is the InnoDB buffer pool, which Aurora sizes automatically from instance memory. On top of that, every connection reserves memory for its own buffers, and queries that sort, group, or join can allocate internal temporary tables that consume memory when they exceed their limits.</p> 
<p>Three consumers drive almost every OOM event, and they are also where you fix it:</p> 
<ul> 
 <li>The buffer pool is sized to the instance and rarely the sole culprit, but it leaves less headroom on smaller classes.</li> 
 <li>Per-connection memory scales with connection count. A fleet that opens thousands of mostly idle connections turns connection growth into memory growth.</li> 
 <li>Temporary tables from large <code>GROUP BY</code>, <code>ORDER BY</code>, <code>DISTINCT</code>, or <code>UNION</code> operations can allocate significant memory per query.</li> 
</ul> 
<p>A bigger instance gives you more total memory, so it takes longer for buffer pool growth, connection buffers, or temporary tables to exhaust it. But whichever of the three is driving the pressure keeps growing, and will eventually hit the new, higher ceiling too. Identifying which one is the real driver is what lets you fix the cause instead of only paying for more headroom.</p> 
<h2>How Aurora protects itself: The recovery-action ladder</h2> 
<p>The OOM avoidance in Aurora escalates through a set of actions, from least to most disruptive. On Aurora MySQL 8.4 with automatic management on, Aurora selects and adjusts these for you. On Aurora MySQL version 3, they are the entries in <code>aurora_oom_response</code>. Some actions are also restricted by deployment type: <em>provisioned</em> clusters run on instances you size yourself, while <em>Aurora Serverless v2</em> clusters scale capacity automatically and don’t support the two actions that resize memory structures directly.</p> 
<table border="1px" width="100%" cellpadding="10px"> 
 <tbody> 
  <tr> 
   <td><strong>Action</strong></td> 
   <td><strong>What it does</strong></td> 
   <td><strong>Notes</strong></td> 
  </tr> 
  <tr> 
   <td><code>print</code></td> 
   <td>Logs the memory-intensive queries and connections to the error log. Nothing is terminated.</td> 
   <td>Versions 3 and 8.4.</td> 
  </tr> 
  <tr> 
   <td><code>tune</code></td> 
   <td>Shrinks internal table caches (<code>table_open_cache</code>, <code>table_definition_cache</code>). These restore automatically when memory recovers.</td> 
   <td>Provisioned only. Not supported on Aurora Serverless v2, which scales capacity automatically instead.</td> 
  </tr> 
  <tr> 
   <td><code>tune_buffer_pool</code></td> 
   <td>Shrinks the InnoDB buffer pool. It restores automatically when memory recovers.</td> 
   <td>Provisioned instances with 2 vCPUs only. Not supported on Aurora Serverless v2. Must be paired with <code>kill_query</code> or <code>kill_connect</code> or it does nothing.</td> 
  </tr> 
  <tr> 
   <td><code>decline</code></td> 
   <td>Rejects new queries with an error while memory is low.</td> 
   <td>Versions 3 and 8.4.</td> 
  </tr> 
  <tr> 
   <td><code>kill_query</code></td> 
   <td>Terminates running <code>SELECT</code> queries, highest memory consumers first. DDL, other DML, and transactions are unaffected.</td> 
   <td>If both <code>kill_query</code> and <code>kill_connect</code> are set, only <code>kill_connect</code> activates.</td> 
  </tr> 
  <tr> 
   <td><code>kill_connect</code></td> 
   <td>Terminates user connections, rolling back their transactions and terminating DDL.</td> 
   <td>In 3.11+ and 8.4, also terminates connections that try to allocate memory during pressure.</td> 
  </tr> 
 </tbody> 
</table> 
<p>The ladder matters because it escalates from least to most disruptive: Aurora tunes caches and the buffer pool first, and only terminates queries or connections if pressure continues. Getting that order right by hand, along with constraints such as <code>tune_buffer_pool</code> needing a terminate action to take effect, is a common pitfall. That is exactly what automatic management removes.</p> 
<h2>Aurora MySQL version 3 today, and what changes when you upgrade to version 8.4</h2> 
<p>On version 3, Aurora auto-configures <code>aurora_oom_response</code> from engine version, instance class, and memory. Per the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQLOOM.html" target="_blank" rel="noopener">Aurora documentation on out-of-memory issues</a>, smaller instances on 3.08+ default to a protective set like <code>print,tune,tune_buffer_pool,decline,kill_connect</code>, and from 3.09 larger instances default to <code>print,decline,kill_connect</code>. Defaults vary by engine version and instance class, so check the effective value with <code>SHOW GLOBAL STATUS LIKE 'aurora_oom%'</code> rather than assuming.</p> 
<p>The upgrade to version 8.4 introduces one behavior worth planning for. When <code>aurora_enable_memory_management</code> is <code>ON</code> (the default), your existing <code>aurora_oom_response</code> is ignored, which is expected and generally an improvement. But if you turn automatic management <code>OFF</code> on version 8.4, <code>aurora_oom_response</code> is <strong>empty by default</strong>, unlike version 3’s populated defaults. A version 8.4 cluster with automatic management disabled and nothing configured can therefore have <em>fewer</em> recovery actions enabled than the version 3 cluster it was upgraded from. If you plan to run with manual control on version 8.4, set <code>aurora_oom_response</code> explicitly as part of the upgrade, and validate the memory-health metrics on a non-production cluster first.</p> 
<h2>Reproduce and read a memory-pressure event</h2> 
<p>A parameter list doesn’t show you what an incident looks like. This section forces one on a test cluster, reads the signals Aurora exposes in order, and then shows the same workload with protection turned off so you can see what the feature is actually helping to prevent.</p> 
<h3>Force memory pressure</h3> 
<p>On a non-production Aurora MySQL version 8.4 test cluster (here a <code>db.r7g.large</code> with 16 GB and a table of about 6.1 million rows, roughly 3.7 GB in size), drive many concurrent sessions through large in-memory sorts. Each session gets its own <em>sort buffer</em>, the per-connection memory MySQL uses to sort result rows for operations like <code>ORDER BY</code> when no index can produce the order directly. Give each session a large sort buffer, then run a full-table <code>ORDER BY</code> on an unindexed column, from roughly 40 sessions at once. The exact number that triggers pressure varies by engine patch version and instance state. Too few sessions do not saturate memory, and too many can overwhelm the instance faster than automatic management can shed load, so ramp up gradually (for example 40, then 60, then 100) until <code>AuroraMemoryHealthState</code> climbs above <code>0</code>:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SET SESSION sort_buffer_size = 1073741824; -- 1 GB per session
SELECT id, grp, rand_val, payload FROM big_table ORDER BY rand_val DESC;</code></pre> 
</div> 
<p>The instance has 16 GB, but 40 sessions each able to claim up to a 1 GB sort buffer can demand up to 40 GB, far more memory than exists. As concurrency rises, two CloudWatch metrics move together: <code>FreeableMemory</code> (the memory on the instance available for new allocations, including OS and database caches) falls, and <code>SwapUsage</code> (how much of that memory has spilled to disk-backed swap because physical memory ran out) climbs. On its own, falling <code>FreeableMemory</code> isn’t an alarm. MySQL and the OS use free memory for caching. The pattern that signals danger is a rapid <code>FreeableMemory</code> decline combined with rising <code>SwapUsage</code>. Under this load, between 20:51 and 20:54 UTC (the same window <code>AuroraMemoryHealthState</code> spiked, see Figure 3), <code>FreeableMemory</code> repeatedly fell from a steady approximately 6.7 GB down to as low as 1.3 GB while automatic management held the line. <code>SwapUsage</code> rose to roughly 184 MB.</p> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-1.png" alt="CloudWatch FreeableMemory holding near 6.7 GB then dipping to 1.3 GB during a memory-pressure event" width="800"></a>
 <p class="wp-caption-text">Figure 1: CloudWatch FreeableMemory for the instance. It holds steady near 6.7 GB, then dips as low as 1.3 GB between 20:51 and 20:54 UTC as the workload saturates memory, recovering each time automatic management sheds load. The run with automatic management turned off (described in the following section) shows a different shape: a single sharp drop followed by the OS terminating the process.</p>
</div> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-2.png" alt="CloudWatch SwapUsage rising to about 184 MB as memory tightens" width="800"></a>
 <p class="wp-caption-text">Figure 2: CloudWatch SwapUsage over the same window, rising to about 184 MB as memory tightens.</p>
</div> 
<h3>Read the signals in order</h3> 
<p>When memory tightens, Aurora exposes the event through a consistent set of signals. Reading them in order tells you what happened and what to do.</p> 
<p><strong>1. Start with memory health.</strong> <code>AuroraMemoryHealthState</code> reports <code>0</code> (healthy), <code>5</code> (moderate pressure), or <code>10</code> (critical). Find it in the CloudWatch console under the Amazon Relational Database Service (Amazon RDS) metrics for your DB instance, or query it directly with the CloudWatch <code>GetMetricData</code> API. A move from <code>0</code> to <code>5</code> to <code>10</code> is the event beginning. Alarm on any sustained value above <code>0</code> so you see pressure before a restart, not after.</p> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-3.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-3.png" alt="CloudWatch AuroraMemoryHealthState flat at 0 then spiking to 5 during memory pressure" width="800"></a>
 <p class="wp-caption-text">Figure 3: CloudWatch AuroraMemoryHealthState. It sits flat at 0 and spikes to 5 (moderate pressure) at the moment the workload saturates memory.</p>
</div> 
<p><strong>2. Confirm which actions are in effect.</strong> Check the runtime OOM status. Connect to the instance with a MySQL client (for example, the mysql command-line tool or MySQL Workbench) and run this on an Aurora MySQL version 8.4.8 instance at rest:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SHOW GLOBAL STATUS LIKE 'aurora_oom%';</code></pre> 
</div> 
<div class="hide-language"> 
 <pre><code class="language-text">+-------------------------------------+----------------------------+
| Variable_name | Value |
+-------------------------------------+----------------------------+
| Aurora_oom_response | print,decline,kill_connect |
| aurora_oom_avoidance_recovery_state | INACTIVE |
+-------------------------------------+----------------------------+</code></pre> 
</div> 
<p><code>Aurora_oom_response</code> shows the actions Aurora is actually using (here <code>print,decline,kill_connect</code>), chosen automatically even though the settable <code>aurora_oom_response</code> parameter is empty on version 8.4 (see the preceding version 8.4 section). <code>aurora_oom_avoidance_recovery_state</code> is <code>INACTIVE</code> when memory is healthy. Polling the same query under load shows the transition mid-event:</p> 
<div class="hide-language"> 
 <pre><code class="language-text">+-------------------------------------+----------------------------+
| Variable_name | Value |
+-------------------------------------+----------------------------+
| Aurora_oom_response | print,decline,kill_connect |
| aurora_oom_avoidance_recovery_state | ACTIVE |
+-------------------------------------+----------------------------+</code></pre> 
</div> 
<p><code>ACTIVE</code> means Aurora is intervening right now.</p> 
<p><strong>3. See what Aurora did during the event.</strong> Under the load, <code>AuroraMemoryHealthState</code> rose from <code>0</code> to <code>5</code> and Aurora began shedding work. Applications saw the recovery actions directly, in their own database client’s error output, as MySQL error <code>1041</code>:</p> 
<div class="hide-language"> 
 <pre><code class="language-text">(1041, 'Query declined - system memory is critically low. This action was taken to protect system stability.')
(1041, 'Connection terminated - system memory is critically low. This action was taken to protect system stability.')</code></pre> 
</div> 
<p>Those counts are also tracked as CloudWatch metrics, <code>AuroraMemoryNumDeclinedSqlTotal</code> and <code>AuroraMemoryNumKillConnTotal</code>, so you don’t need to rely on client-side error counting to know how much work the feature shed. Over this pressure window they climbed in repeated bursts rather than a single steady ramp, matching the error log’s pattern of distinct detect-and-recover cycles a few seconds apart:</p> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-4.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-4.png" alt="CloudWatch graph of declined queries and terminated connections peaking at 21 and 14" width="800"></a>
 <p class="wp-caption-text">Figure 4: CloudWatch AuroraMemoryNumDeclinedSqlTotal and AuroraMemoryNumKillConnTotal over the pressure window, peaking at 21 declined queries and 14 terminated connections in the 20:52 UTC data point.</p>
</div> 
<p>The number of terminated connections can exceed the number of connections open at a single instant, because clients that get disconnected often reconnect immediately and get caught by the next cycle. The cumulative count across a multi-cycle event reflects that repeated churn, not a one-time mass termination. Those are the <code>decline</code> and <code>kill_connect</code> actions from the ladder doing their job: new and heaviest work is rejected so the instance stays up. Critically, the database did <strong>not</strong> restart.</p> 
<p><strong>4. Read the error log.</strong> Download it from the RDS console (go to your instance, then <strong>Logs &amp; events</strong>, then <strong>Logs</strong>) or with <code>aws rds download-db-log-file-portion</code>. Recovery messages are prefixed Aurora memory management: on version 8.4 (and OOM crash avoidance: on version 3), and they name the specific queries and connections involved:</p> 
<div class="hide-language"> 
 <pre><code class="language-text">Aurora memory management: Memory pressure detected. Total: 16087316 KB,
available: 1580244 KB. Memory recovery actions include declining queries,
terminating connections.
Memory report: {"system_report":{"total_mem_kb":16087316,"available_mem_kb":1577088, ...},
"user_connections_memory_usage":{"connection_count":5,"current_used_bytes":5369504400},
"top_memory_consuming_connections":[{"id":87,"used_bytes":1073900880, ...}]}
Aurora memory management: Memory recovered. Actions taken: queries declined: 21,
connections killed: 2. Total: 16087316 KB, available: 1803672 KB.</code></pre> 
</div> 
<p>The memory report names the biggest consumers. Here, 5 surviving user connections held about 5.37 GB between them, each one close to the full 1 GB <code>sort_buffer_size</code> setting. That provides a route from detecting pressure to identifying its root cause, and the paired “Memory recovered” line confirms what the recovery cost: 21 declined queries and 2 terminated connections for this single cycle.</p> 
<p><strong>5. Attribute it to a consumer.</strong> Use the Performance Schema memory summary to find the components and connections holding the most memory, then map it back to buffer pool, per-connection memory, or temporary tables:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SELECT event_name,
current_number_of_bytes_used/1024/1024 AS mb_used
FROM performance_schema.memory_summary_global_by_event_name
ORDER BY current_number_of_bytes_used DESC
LIMIT 10;</code></pre> 
</div> 
<div class="hide-language"> 
 <pre><code class="language-text">+-----------------------------------------------------------------------------+---------+
| event_name | mb_used |
+-----------------------------------------------------------------------------+---------+
| memory/sql/Filesort_buffer::sort_keys | 4064.74 |
| memory/innodb/hash0hash | 93.96 |
| memory/performance_schema/events_statements_summary_by_digest | 40.28 |
| memory/innodb/lock0lock | 19.58 |
| memory/mysys/KEY_CACHE | 16.00 |
| memory/performance_schema/events_statements_history_long | 14.42 |
| memory/performance_schema/events_errors_summary_by_thread_by_error | 14.21 |
| memory/performance_schema/events_statements_summary_by_thread_by_event_name | 11.87 |
| memory/performance_schema/events_statements_history_long.sql_text | 9.77 |
| memory/performance_schema/events_statements_history_long.digest_text | 9.77 |
+-----------------------------------------------------------------------------+---------+</code></pre> 
</div> 
<p>This output comes from a lighter run of the same query with 4 concurrent sessions, which kept memory below the level where Aurora declines the diagnostic query itself. <code>Filesort_buffer::sort_keys</code> alone accounts for about 4.0 GB, close to 1 GB per session, matching the <code>sort_buffer_size</code> setting. That is roughly 18 times every other component combined, and about 43 times the next-largest single component (<code>hash0hash</code>, at about 94 MB). That single line confirms the cause: the concurrent <code>ORDER BY</code> sorts allocating large per-session sort buffers, not the buffer pool or a leak elsewhere. Under critical pressure, Aurora can decline this diagnostic query along with the workload, so run it as soon as pressure eases, or rely on the error log’s memory report during the peak.</p> 
<p><strong>6. Confirm the outcome.</strong> <code>AuroraMemoryHealthState</code> returns to <code>0</code> and <code>aurora_oom_avoidance_recovery_state</code> goes back to <code>INACTIVE</code> once the pressure clears. That means Aurora rode out the event. If instead <code>os.memory.outOfMemoryKillCount</code> (a Performance Insights operating system counter) is non-zero, the OS terminated the process and a restart followed. Recovery didn’t keep up, and the workload or instance size needs attention.</p> 
<h3>The same workload with protection off</h3> 
<p>To see what the feature helps prevent, set <code>aurora_enable_memory_management</code> = <code>OFF</code> with an empty <code>aurora_oom_response</code>, reboot to apply (changing this parameter takes effect only after a reboot), and run the identical workload in a separate test window from the preceding protected run. With no recovery actions, memory pressure runs to exhaustion and the OS terminates the database process:</p> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-5.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-5.png" alt="CloudWatch FreeableMemory with memory management off dropping to 645 MB then recovering as the instance restarts" width="800"></a>
 <p class="wp-caption-text">Figure 5: CloudWatch FreeableMemory for the same workload with automatic memory management off. Unlike the protected run’s repeated recovery, memory drops once to about 645 MB at 21:25 UTC and then snaps back to baseline as the instance restarts.</p>
</div> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-6.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-6.png" alt="RDS events list showing a 17:25 instance restart with no preceding shutdown entry" width="800"></a>
 <p class="wp-caption-text">Figure 6: RDS events for the instance with automatic memory management turned off. The 17:25 restart has no preceding shutdown entry, unlike the planned parameter-group reboot preceding it. The RDS console displays times in your browser’s local time zone (here UTC-04:00), so 17:25 in this figure is the same event as 21:25 UTC in Figure 5.</p>
</div> 
<p>The database was unreachable for about 30 seconds: the new <code>mysqld</code> process started roughly 29 seconds after the load launched and was accepting connections again a few seconds after that. Every connection drops and the cluster restarts. Instead of the graceful <code>1041</code> declines, applications see connection resets and “can’t connect” errors. That illustrates how the feature can help maintain database availability by shedding load during memory pressure events. With automatic management back on (the default), the same workload produces the recovery sequence from the previous section: some declined queries and terminated connections, and a database that stays up. That contrast is the whole value of the feature.</p> 
<p>After an <code>OFF</code> test, turn the feature back on, reboot, and verify it:</p> 
<div class="hide-language"> 
 <pre><code class="language-sql">SHOW GLOBAL VARIABLES LIKE 'aurora_enable_memory_management';</code></pre> 
</div> 
<div class="hide-language"> 
 <pre><code class="language-text">+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| aurora_enable_memory_management | ON |
+---------------------------------+-------+</code></pre> 
</div> 
<p>Leaving the parameter group on <code>OFF</code> after a test is how an unprotected production cluster happens by accident.</p> 
<h2>Decide: Keep automatic management on, or take manual control</h2> 
<p>For the large majority of Aurora MySQL version 8.4 workloads, the right choice is the default: leave <code>aurora_enable_memory_management</code> set to <code>ON</code>. Aurora adjusts the actions to conditions, and you avoid the constraints that make manual configuration error-prone. Verifying it’s on and staying on a current patch level is the whole task.</p> 
<p>There are narrower cases for turning it <code>OFF</code> and setting <code>aurora_oom_response</code> yourself:</p> 
<ul> 
 <li>You need a deterministic, fixed action set for change-controlled environments where dynamic behavior is undesirable.</li> 
 <li>Your workload is latency-sensitive online transaction processing (OLTP) and you would rather terminate a heavy query (<code>kill_query</code>) than reject all new queries (<code>decline</code>), or the reverse.</li> 
 <li>You must protect a specific class of work and want to avoid <code>kill_connect</code> entirely.</li> 
</ul> 
<p>If you do take manual control, two rules matter: pair <code>tune_buffer_pool</code> with <code>kill_query</code> or <code>kill_connect</code> or it won’t resize the pool, and never leave <code>aurora_oom_response</code> empty in production, which disables recovery entirely.</p> 
<h2>Aurora Serverless v2 differences</h2> 
<p>The preceding reproduction used a provisioned instance. Aurora Serverless v2 responds differently, per the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQLOOM.html" target="_blank" rel="noopener">Aurora documentation</a>: it responds to memory pressure first by scaling up Aurora Capacity Units (ACUs) to add memory. If pressure persists while scaling is in progress, Aurora can terminate existing connections to recover. Termination of connections that try to allocate memory happens only once the instance has reached its configured maximum ACU and can scale no further. The <code>tune</code> and <code>tune_buffer_pool</code> actions are not supported on Aurora Serverless v2. The others behave as on provisioned instances. Two implications: size the maximum ACU with real memory headroom so scaling has room to work, and make sure clients reconnect gracefully, since error <code>1041</code> can appear here too (see Limitations).</p> 
<h2>Take immediate action</h2> 
<p>When you catch memory pressure in progress:</p> 
<ol type="1"> 
 <li><strong>Confirm protection is active.</strong> <code>SHOW GLOBAL STATUS LIKE 'aurora_oom%'</code> shows <code>aurora_oom_avoidance_recovery_state = ACTIVE</code> during an event. To check whether automatic management itself is on, use <code>SHOW GLOBAL VARIABLES LIKE 'aurora_enable_memory_management'</code> instead, since that’s a parameter, not a status.</li> 
 <li><strong>Find and stop the heaviest query.</strong> Identify it from the error log or Performance Schema and cancel it if it’s non-critical with <code>CALL mysql.rds_kill_query(&lt;id&gt;);</code>.</li> 
 <li><strong>Stabilize connections.</strong> Pool with <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html" target="_blank" rel="noopener">Amazon RDS Proxy</a> so a connection surge doesn’t become a memory surge.</li> 
 <li><strong>Reduce pressure at the source.</strong> Index the queries that build large temporary tables, and right-size <code>max_connections</code> to the instance.</li> 
</ol> 
<h2>Summary</h2> 
<p>The following table sums up how OOM protection differs between the two Aurora MySQL versions:</p> 
<table border="1px" width="100%" cellpadding="10px"> 
 <tbody> 
  <tr> 
   <td><strong>Capability</strong></td> 
   <td><strong>Aurora MySQL version 3</strong></td> 
   <td><strong>Aurora MySQL version 8.4</strong></td> 
  </tr> 
  <tr> 
   <td>Prevention parameter</td> 
   <td><code>aurora_oom_response</code> (auto-set defaults)</td> 
   <td><code>aurora_enable_memory_management</code> (<code>ON</code> by default)</td> 
  </tr> 
  <tr> 
   <td>Mechanism</td> 
   <td>Fixed recovery-action ladder</td> 
   <td>Dynamic recovery-action ladder</td> 
  </tr> 
  <tr> 
   <td>Default protection</td> 
   <td>Populated defaults by instance class</td> 
   <td>Managed automatically</td> 
  </tr> 
  <tr> 
   <td>Manual override</td> 
   <td>Edit <code>aurora_oom_response</code></td> 
   <td>Set <code>OFF</code>, then set <code>aurora_oom_response</code> (empty by default)</td> 
  </tr> 
  <tr> 
   <td>Key metric</td> 
   <td><code>AuroraMemoryHealthState</code></td> 
   <td><code>AuroraMemoryHealthState</code></td> 
  </tr> 
 </tbody> 
</table> 
<h2>Limitations</h2> 
<ul> 
 <li>If you turn automatic management <code>OFF</code> on version 8.4, <code>aurora_oom_response</code> is empty by default, so no recovery actions run until you configure them explicitly.</li> 
 <li><code>tune_buffer_pool</code> is supported on provisioned instances with 2 vCPUs only, and needs to be paired with <code>kill_query</code> or <code>kill_connect</code>. On its own it does nothing.</li> 
 <li>If both <code>kill_query</code> and <code>kill_connect</code> are set, only <code>kill_connect</code> activates.</li> 
 <li><code>kill_connect</code> rolls back open transactions on the connections it terminates.</li> 
 <li>Applications need retry handling for MySQL error <code>1041</code> wherever they connect to Aurora MySQL. A provisioned instance under automatic management can decline queries and terminate connections the same way Aurora Serverless v2 does.</li> 
 <li>The feature <code>aurora_enable_memory_management</code> is available only in Aurora MySQL 8.4. OOM avoidance is specific to Aurora MySQL and does not exist on RDS for MySQL.</li> 
</ul> 
<h2>Conclusion</h2> 
<p>Out-of-memory restarts are preventable, and on Aurora MySQL they’re increasingly avoided for you. We reproduced a memory-pressure event, read it through health state, the recovery counters, and the error log, and saw the same workload restart the database once protection was removed. We then covered when to keep automatic memory management on versus taking manual control, what changes when you move from version 3 to 8.4, and the Aurora Serverless v2 differences.</p> 
<p>A safe posture on version 8.4 is the default: keep <code>aurora_enable_memory_management</code> on, stay current on patches, and alarm on <code>AuroraMemoryHealthState</code> so a protected instance does not quietly becomes a stressed one. And because recovery actions are a safety net, reduce the pressure at its source: pool connections so connection growth doesn’t become memory growth, and index the queries that build large temporary tables.</p> 
<p>To learn more, see <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQLOOM.html" target="_blank" rel="noopener">Troubleshooting out-of-memory issues for Aurora MySQL databases</a>, <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/ams-workload-memory.html" target="_blank" rel="noopener">Troubleshooting memory usage issues for Aurora MySQL databases</a>, and <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.BestPractices.Performance.html" target="_blank" rel="noopener">Best practices for Aurora MySQL performance and scaling</a>. If you have questions or suggestions, leave a comment, or visit <a href="https://repost.aws/" target="_blank" rel="noopener">AWS re:Post</a>.</p> 
<p> </p> 
<hr> 
<h2>About the authors</h2> 
<footer> 
 <div class="blog-author-box"> 
  <div class="blog-author-image">
   <img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-7.jpg" alt="Aayushi Mittal" width="100" height="133">
  </div> 
  <h3 class="lb-h4">Aayushi Mittal</h3> 
  <p>Aayushi is a Technical Account Manager at AWS, where she helps enterprise customers keep Amazon Aurora and Amazon RDS databases healthy and performant. She specializes in memory and performance troubleshooting, and is driven by turning complex operational incidents into clear, actionable guidance her customers can act on with confidence.</p> 
 </div> 
 <div class="blog-author-box"> 
  <div class="blog-author-image">
   <img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5897-8.jpg" alt="Archana Dhere" width="100" height="133">
  </div> 
  <h3 class="lb-h4">Archana Dhere</h3> 
  <p>Archana is a Technical Account Manager at AWS, where she partners with enterprise customers to optimize their AWS environments across databases, security, and cloud operations. She specializes in Amazon RDS and Aurora performance and migration planning, and is passionate about translating complex operational challenges into clear, actionable guidance for her customers.</p> 
 </div> 
</footer>]]></content:encoded>
    <pubDate>Wed, 11 Nov 2026 16:51:16 +0000</pubDate>
    <dc:creator>AWS Database Blog</dc:creator>
    <category>Advanced (300)</category>
    <category>Amazon Aurora</category>
    <category>Best Practices</category>
  </item>

  <item>
    <title>Diagnose Amazon Aurora MySQL performance using CloudWatch Database Insights</title>
    <guid isPermaLink="false">6b115351958f4c3065b0b6fb1c75823afdbdbf09</guid>
    <link>https://aws.amazon.com/blogs/database/diagnose-amazon-aurora-mysql-performance-using-cloudwatch-database-insights/</link>
    <description>When an Amazon Aurora MySQL-Compatible Edition or Amazon Relational Database Service (Amazon RDS) for MySQL database slows down, it can be hard to tell why. CPU graphs and slow-query logs rarely tell the whole story. The more direct answer usually lies in wait events. A wait event tells you what your sessions are waiting for: reading from storage, redo log flushes, acquiring an internal lock, or waiting for CPU time. Amazon CloudWatch Database Insights surfaces this data as DB load, so you can quickly see whether a database is constrained and, if so, on what. 
Earlier MySQL wait-event guidance relied on the Performance Insights dashboard. Our 2018 post Tuning Amazon RDS for MySQL with Performance Insights walked through that workflow. The Performance Insights console experience has since moved into CloudWatch Database Insights, and the standalone console now redirects there, while the Performance Insights API continues to work unchanged. This post refreshes that guidance for the current tool. We show how to read DB load in Database Insights for a MySQL workload and translate the most common wait events into concrete actions. That way, you can help identify the real bottleneck instead of scaling the instance without knowing why. 
Solution overview 
Database Insights measures database activity as average active sessions (AAS). An active session is a connection that submits work to the engine and is either running on CPU or waiting for a resource. Database Insights samples active sessions each second, averages them over time, and reports this as the DB load metric. It plots DB load against a Max vCPU line that shows the instance’s CPU capacity. 
This gives you a first read: 
 
 DB load below Max vCPU – The database generally has capacity. Bottlenecks are usually specific queries or I/O, not overall saturation. 
 DB load consistently above Max vCPU – More sessions are active than the instance has vCPUs, so sessions wait to run, queuing for CPU time, storage I/O, or a lock. The dominant wait events can help you identify which resource might be constrained. 
 
For MySQL engines, wait events come from the MySQL Performance Schema. Because DB load is only the average active sessions, Database Insights can break it down by the wait event each session is in. This shows which wait consumes the most session time and correlates it with the top SQL statements driving the load. The authoritative list of MySQL wait events and their meanings is in Tuning Aurora MySQL with wait events. This post focuses on how to act on the ones you will see most often. 
Prerequisites 
Before you begin, make sure you have the following: 
 
 An Aurora MySQL or RDS for MySQL instance with CloudWatch Database Insights enabled. Standard mode is on by default and is available at no cost. It provides 7 days of DB load history and the per-instance load, wait-event, and Top SQL views (including per-SQL statement statistics) used throughout this post. You can optionally extend Standard mode retention to between 1 and 24 months for an added cost, at the same price as Performance Insights. Advanced mode adds fleet-wide monitoring, so you can view and rank many database instances together from the Fleet Health Dashboard shown later. It also adds on-demand analysis of a past time window and includes 1 to 24 month retention at no extra cost. It carries an additional cost billed per vCPU-hour (per ACU-hour for Aurora Serverless). For the full list of what each mode includes, see the Standard and Advanced feature comparison in the CloudWatch Database Insights documentation, and then see Amazon CloudWatch pricing for cost details. 
 For Aurora MySQL, wait-event data comes from the Performance Schema. When you create the instance with Database Insights turned on, Database Insights manages the Performance Schema automatically. If you enabled Database Insights on an existing instance, you might need to turn on the Performance Schema manually and reboot. See Enabling the Performance Schema for Performance Insights on Aurora MySQL. 
 Permissions to view Database Insights in the Amazon CloudWatch console. At minimum this means read access to CloudWatch metrics (cloudwatch:GetMetricData, cloudwatch:GetMetricStatistics, cloudwatch:ListMetrics), Performance Insights actions (pi:GetResourceMetrics, pi:DescribeDimensionKeys, and related pi: actions), and rds:DescribeDBInstances and rds:DescribeDBClusters. For the complete AWS Identity and Access Management (IAM) policy, see the sample policy in Get started with CloudWatch Database Insights. 
 
Read the load: DB load sliced by waits and top SQL 
Open Database Insights for the instance, set the DB load chart to slice by waits, and choose the time window that covers the slowdown. (In Advanced mode you can run on-demand analysis for any past period, which helps when the slowdown has already passed.) Two questions guide every investigation: 
 
 Is the database saturated? Compare DB load to the Max vCPU line over the problem window. Sustained load above the line means sessions are queuing for resources. Load below the line doesn’t always mean healthy, though: if one or two sessions hold a lock that blocks others, throughput can stall while DB load stays low. Locking waits still appear in the wait breakdown even when overall load is below the line. 
 On what? Identify the one or two wait categories that make up most of the stacked load. Then select that wait to see the top SQL statements associated with it. This pivot, from a dominant wait to the specific statements causing it, is where you find most root causes. 
 
The following figure shows the Database load chart for an Aurora MySQL instance, sliced by wait event. The dashed line marks Max vCPU, and each color is a wait category contributing to load. 

 
 Figure 1: Database Insights DB load for an Aurora MySQL instance, sliced by wait event against the Max vCPU line
 
In this example, average active sessions sit right at the Max vCPU line, so the instance is running near its CPU capacity. The load is dominated by CPU (green), followed by wait/io/table/sql/handler (table I/O). That points the investigation toward the SQL consuming CPU and driving a high volume of row reads. 
Selecting the dominant wait and opening the Top SQL tab surfaces the statements driving that load, ranked by their contribution. 

 
 Figure 2: Top SQL ranked by load for the dominant wait, showing many near-identical point-select statements
 
Here the load is spread fairly evenly across many near-identical point-select statements (SELECT c FROM sbtest... WHERE id = ?), each contributing roughly 0.03–0.04 average active sessions. That pattern points to an aggregate CPU-bound online transaction processing (OLTP) workload rather than one expensive query. The remedy is to lower per-query CPU usage across the board through indexing, simplifying hot queries, and caching repeated reads, rather than hunting for a single outlier. To decide which remedy applies, use the Top SQL view. Statements that examine far more rows than they return are good index candidates, repeated identical reads are good caching candidates, and statements doing redundant work are candidates for simplification. The calls, rows-examined, and rows-sent statistics in the Top SQL view help you tell these apart. 
Database Insights groups statements into digests with literal values replaced by ?, but full SQL samples can still contain literals. If your queries embed sensitive data as literal values, prefer parameterized queries and review the captured SQL text in a non-production environment before relying on it. 
With that investigation method in mind, the next section covers what the common MySQL waits mean and what to do about each. 
Wait-event playbook 
The following are the wait events that most often dominate MySQL workloads. Treat the what to do items as starting points. Always confirm with the top SQL correlated to the wait before making changes. 
CPU 
The cpu wait event means a session is running on, or waiting for, a CPU. If cpu dominates and DB load sits above Max vCPU, the instance is CPU-bound. 
What to do: Find the top SQL by CPU and reduce it: add or improve indexes to avoid full scans, simplify expensive joins and sorts, and cache repeated read-heavy queries. If the workload is genuinely CPU-bound after tuning, scale the instance class or offload reads to Aurora replicas. 
io/aurora_redo_log_flush (Aurora MySQL-specific) 
This event is specific to Aurora MySQL. On RDS for MySQL, commit-time flushing appears as standard InnoDB redo log I/O waits. 
What to do: Batch small writes into larger transactions where the application allows, so you commit less frequently. Review whether every write path needs to be synchronous. For sustained high write I/O, evaluate Aurora I/O-Optimized cluster storage for cost predictability. 
io/table/sql/handler 
This event reflects table I/O through the storage engine handler, often the signature of large reads and full table scans. 
What to do: Inspect the correlated SQL and its execution plan with EXPLAIN. Add indexes that support the query’s filter and join columns, avoid SELECT * on wide tables, and partition very large tables so scans touch less data. Aim for a buffer pool cache hit ratio high enough that most reads are served from memory. 
SYNCH events (synch/mutex, synch/sxlock) 
SYNCH waits occur when multiple sessions contend for the same in-memory structure that a mutex or lock protects. They often rise sharply as concurrency increases and can drive CPU up with them. 
What to do: Reduce contention rather than only adding CPU. Look for hot rows or hot tables updated by many sessions at once, spread writes across more keys, and shorten transactions so locks are held briefly. See Troubleshoot SYNCH wait events for specific patterns. 
io/socket/sql/client_connection 
This event means the database is busy creating threads to handle incoming new connections. It indicates connection churn, where clients open and close connections rapidly. 
What to do: Introduce or right-size application-side connection pooling so connections are reused, and consider Amazon RDS Proxy to pool and reuse connections. This is an application connectivity problem rather than a query problem, so scaling the instance will not fix it. 
io/aurora_respond_to_client (Aurora MySQL-specific) 
This event is specific to Aurora MySQL. On RDS for MySQL, time spent returning results appears under standard network and socket waits. 
What to do: Return only the columns and rows the application needs, paginate large result sets, and check for network or client-side processing delays that keep the session open while it waits to return data. 
Locking waits 
When sessions wait on row or table locks, DB load rises even when CPU isn’t saturated, and throughput stalls under concurrency. 
What to do: Keep transactions short, access rows in a consistent order to avoid deadlocks, and avoid long-running transactions that hold locks. Review isolation level and application retry logic for lock-wait timeouts and deadlocks. 
Triage across a fleet with the Fleet Health Dashboard 
When you run many databases, work top down: start at the fleet, rank instances by load, then drop into the per-instance wait and SQL views for the busiest ones. That triage starts with the Database Insights Fleet Health Dashboard, which shows DB load and health across a group of databases, so you can rank instances by load and open the busiest ones first. 
The following figure shows the Fleet Health Dashboard ranking databases by DB load utilization, the ratio of DB load to the number of vCPUs. DB load shown here publishes to CloudWatch as the DBLoad metric, along with its DBLoadCPU and DBLoadNonCPU components. You can alarm on it and add it to CloudWatch dashboards next to the rest of your monitoring. 

 
 Figure 3: Fleet Health Dashboard ranking Aurora MySQL instances by DB load utilization
 
Here the writer instance tops the ranking at over 100 percent of its vCPU capacity, while the reader sits idle. Selecting the busiest resource surfaces its top queries and top wait events (in this case CPU at about 73 percent and io/table/sql/handler at about 20 percent). From there, you move from fleet triage straight into the per-instance playbook. Use fleet filters based on tags (for example, environment or application) to focus on the databases that matter. 
This flow, from fleet to instance to wait to SQL, keeps investigations fast and repeatable, and it scales as you add databases. 
Conclusion 
In this post, we showed how to diagnose Amazon Aurora MySQL-Compatible Edition and Amazon RDS for MySQL performance using Amazon CloudWatch Database Insights. We covered how Database Insights represents load as average active sessions against instance capacity, how to slice load by wait event and pivot to the top SQL, what the most common MySQL wait events mean, and how to triage across a fleet. Reading wait events this way helps you fix the actual bottleneck rather than defaulting to a larger instance. The cause might be a query, an index, a connection pattern, or contention. 
To learn more, see Tuning Aurora MySQL with wait events, Get started with CloudWatch Database Insights, and Essential concepts for Aurora MySQL tuning. For related guidance on the AWS Database Blog, see Amazon CloudWatch Database Insights applied in real scenarios and Analyze Amazon Aurora MySQL workloads with Performance Insights. If you have questions or suggestions, leave a comment, or visit AWS re:Post. 
  
 
About the author 
 
  
  
   
   
  Aayushi Mittal 
  Aayushi is a Technical Account Manager at AWS, where she helps enterprise customers keep Amazon Aurora and Amazon RDS databases healthy and performant. She specializes in memory and performance troubleshooting, and is driven by turning complex operational incidents into clear, actionable guidance her customers can act on with confidence. 
  
</description>
    <content:encoded><![CDATA[<p>When an <a href="https://aws.amazon.com/rds/aurora/" target="_blank" rel="noopener">Amazon Aurora MySQL-Compatible Edition</a> or <a href="https://aws.amazon.com/rds/mysql/" target="_blank" rel="noopener">Amazon Relational Database Service (Amazon RDS) for MySQL</a> database slows down, it can be hard to tell why. CPU graphs and slow-query logs rarely tell the whole story. The more direct answer usually lies in wait events. A wait event tells you what your sessions are waiting for: reading from storage, redo log flushes, acquiring an internal lock, or waiting for CPU time. <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Database-Insights.html" target="_blank" rel="noopener">Amazon CloudWatch Database Insights</a> surfaces this data as DB load, so you can quickly see whether a database is constrained and, if so, on what.</p> 
<p>Earlier MySQL wait-event guidance relied on the Performance Insights dashboard. Our 2018 post <a href="https://aws.amazon.com/blogs/database/tuning-amazon-rds-for-mysql-with-performance-insights/" target="_blank" rel="noopener">Tuning Amazon RDS for MySQL with Performance Insights</a> walked through that workflow. The Performance Insights console experience has since moved into CloudWatch Database Insights, and the standalone console now redirects there, while the Performance Insights API continues to work unchanged. This post refreshes that guidance for the current tool. We show how to read DB load in Database Insights for a MySQL workload and translate the most common wait events into concrete actions. That way, you can help identify the real bottleneck instead of scaling the instance without knowing why.</p> 
<h2>Solution overview</h2> 
<p>Database Insights measures database activity as average active sessions (AAS). An active session is a connection that submits work to the engine and is either running on CPU or waiting for a resource. Database Insights samples active sessions each second, averages them over time, and reports this as the DB load metric. It plots DB load against a Max vCPU line that shows the instance’s CPU capacity.</p> 
<p>This gives you a first read:</p> 
<ul> 
 <li><strong>DB load below Max vCPU –</strong> The database generally has capacity. Bottlenecks are usually specific queries or I/O, not overall saturation.</li> 
 <li><strong>DB load consistently above Max vCPU –</strong> More sessions are active than the instance has vCPUs, so sessions wait to run, queuing for CPU time, storage I/O, or a lock. The dominant wait events can help you identify which resource might be constrained.</li> 
</ul> 
<p>For MySQL engines, wait events come from the MySQL Performance Schema. Because DB load is only the average active sessions, Database Insights can break it down by the wait event each session is in. This shows which wait consumes the most session time and correlates it with the top SQL statements driving the load. The authoritative list of MySQL wait events and their meanings is in <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Managing.Tuning.wait-events.html" target="_blank" rel="noopener">Tuning Aurora MySQL with wait events</a>. This post focuses on how to act on the ones you will see most often.</p> 
<h2>Prerequisites</h2> 
<p>Before you begin, make sure you have the following:</p> 
<ul> 
 <li>An Aurora MySQL or RDS for MySQL instance with <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Database-Insights-Get-Started.html" target="_blank" rel="noopener">CloudWatch Database Insights enabled</a>. <em>Standard mode</em> is on by default and is available at no cost. It provides 7 days of DB load history and the per-instance load, wait-event, and Top SQL views (including per-SQL statement statistics) used throughout this post. You can optionally extend Standard mode retention to between 1 and 24 months for an added cost, at the same price as Performance Insights. <em>Advanced mode</em> adds fleet-wide monitoring, so you can view and rank many database instances together from the Fleet Health Dashboard shown later. It also adds on-demand analysis of a past time window and includes 1 to 24 month retention at no extra cost. It carries an additional cost billed per vCPU-hour (per ACU-hour for Aurora Serverless). For the full list of what each mode includes, see the <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Database-Insights.html" target="_blank" rel="noopener">Standard and Advanced feature comparison</a> in the CloudWatch Database Insights documentation, and then see <a href="https://aws.amazon.com/cloudwatch/pricing/" target="_blank" rel="noopener">Amazon CloudWatch pricing</a> for cost details.</li> 
 <li>For Aurora MySQL, wait-event data comes from the Performance Schema. When you create the instance with Database Insights turned on, Database Insights manages the Performance Schema automatically. If you enabled Database Insights on an existing instance, you might need to turn on the Performance Schema manually and reboot. See <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_PerfInsights.EnableMySQL.html" target="_blank" rel="noopener">Enabling the Performance Schema for Performance Insights on Aurora MySQL</a>.</li> 
 <li>Permissions to view Database Insights in the Amazon CloudWatch console. At minimum this means read access to CloudWatch metrics (cloudwatch:GetMetricData, cloudwatch:GetMetricStatistics, cloudwatch:ListMetrics), Performance Insights actions (pi:GetResourceMetrics, pi:DescribeDimensionKeys, and related pi: actions), and rds:DescribeDBInstances and rds:DescribeDBClusters. For the complete AWS Identity and Access Management (IAM) policy, see the sample policy in <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Database-Insights-Get-Started.html" target="_blank" rel="noopener">Get started with CloudWatch Database Insights</a>.</li> 
</ul> 
<h2>Read the load: DB load sliced by waits and top SQL</h2> 
<p>Open Database Insights for the instance, set the DB load chart to slice by waits, and choose the time window that covers the slowdown. (In Advanced mode you can run on-demand analysis for any past period, which helps when the slowdown has already passed.) Two questions guide every investigation:</p> 
<ol type="1"> 
 <li><strong>Is the database saturated?</strong> Compare DB load to the Max vCPU line over the problem window. Sustained load above the line means sessions are queuing for resources. Load below the line doesn’t always mean healthy, though: if one or two sessions hold a lock that blocks others, throughput can stall while DB load stays low. Locking waits still appear in the wait breakdown even when overall load is below the line.</li> 
 <li><strong>On what?</strong> Identify the one or two wait categories that make up most of the stacked load. Then select that wait to see the top SQL statements associated with it. This pivot, from a dominant wait to the specific statements causing it, is where you find most root causes.</li> 
</ol> 
<p>The following figure shows the Database load chart for an Aurora MySQL instance, sliced by wait event. The dashed line marks Max vCPU, and each color is a wait category contributing to load.</p> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5871-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5871-1.png" alt="CloudWatch Database Insights database load chart for an Aurora MySQL instance, sliced by wait event, with a dashed Max vCPU line and a legend listing wait categories such as CPU and io/table/sql/handler." width="800"></a>
 <p class="wp-caption-text">Figure 1: Database Insights DB load for an Aurora MySQL instance, sliced by wait event against the Max vCPU line</p>
</div> 
<p>In this example, average active sessions sit right at the Max vCPU line, so the instance is running near its CPU capacity. The load is dominated by CPU (green), followed by <code>wait/io/table/sql/handler</code> (table I/O). That points the investigation toward the SQL consuming CPU and driving a high volume of row reads.</p> 
<p>Selecting the dominant wait and opening the <strong>Top SQL</strong> tab surfaces the statements driving that load, ranked by their contribution.</p> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5871-2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5871-2.png" alt="CloudWatch Database Insights Top SQL panel ranked by load by waits, showing point-select statements against sbtest tables, each contributing a small and similar share of CPU load." width="800"></a>
 <p class="wp-caption-text">Figure 2: Top SQL ranked by load for the dominant wait, showing many near-identical point-select statements</p>
</div> 
<p>Here the load is spread fairly evenly across many near-identical point-select statements (<code>SELECT c FROM sbtest... WHERE id = ?</code>), each contributing roughly 0.03–0.04 average active sessions. That pattern points to an aggregate CPU-bound online transaction processing (OLTP) workload rather than one expensive query. The remedy is to lower per-query CPU usage across the board through indexing, simplifying hot queries, and caching repeated reads, rather than hunting for a single outlier. To decide which remedy applies, use the Top SQL view. Statements that examine far more rows than they return are good index candidates, repeated identical reads are good caching candidates, and statements doing redundant work are candidates for simplification. The calls, rows-examined, and rows-sent statistics in the Top SQL view help you tell these apart.</p> 
<p>Database Insights groups statements into digests with literal values replaced by <code>?</code>, but full SQL samples can still contain literals. If your queries embed sensitive data as literal values, prefer parameterized queries and review the captured SQL text in a non-production environment before relying on it.</p> 
<p>With that investigation method in mind, the next section covers what the common MySQL waits mean and what to do about each.</p> 
<h2>Wait-event playbook</h2> 
<p>The following are the wait events that most often dominate MySQL workloads. Treat the <em>what to do</em> items as starting points. Always confirm with the top SQL correlated to the wait before making changes.</p> 
<h3>CPU</h3> 
<p>The <code>cpu</code> wait event means a session is running on, or waiting for, a CPU. If <code>cpu</code> dominates and DB load sits above Max vCPU, the instance is CPU-bound.</p> 
<p>What to do: Find the top SQL by CPU and reduce it: add or improve indexes to avoid full scans, simplify expensive joins and sorts, and cache repeated read-heavy queries. If the workload is genuinely CPU-bound after tuning, scale the instance class or offload reads to Aurora replicas.</p> 
<h3>io/aurora_redo_log_flush (Aurora MySQL-specific)</h3> 
<p>This event is specific to Aurora MySQL. On RDS for MySQL, commit-time flushing appears as standard InnoDB redo log I/O waits.</p> 
<p>What to do: Batch small writes into larger transactions where the application allows, so you commit less frequently. Review whether every write path needs to be synchronous. For sustained high write I/O, evaluate Aurora I/O-Optimized cluster storage for cost predictability.</p> 
<h3>io/table/sql/handler</h3> 
<p>This event reflects table I/O through the storage engine handler, often the signature of large reads and full table scans.</p> 
<p>What to do: Inspect the correlated SQL and its execution plan with <code>EXPLAIN</code>. Add indexes that support the query’s filter and join columns, avoid <code>SELECT *</code> on wide tables, and partition very large tables so scans touch less data. Aim for a buffer pool cache hit ratio high enough that most reads are served from memory.</p> 
<h3>SYNCH events (synch/mutex, synch/sxlock)</h3> 
<p>SYNCH waits occur when multiple sessions contend for the same in-memory structure that a mutex or lock protects. They often rise sharply as concurrency increases and can drive CPU up with them.</p> 
<p>What to do: Reduce contention rather than only adding CPU. Look for hot rows or hot tables updated by many sessions at once, spread writes across more keys, and shorten transactions so locks are held briefly. See <a href="https://repost.aws/knowledge-center/aurora-mysql-synch-wait-events" target="_blank" rel="noopener">Troubleshoot SYNCH wait events</a> for specific patterns.</p> 
<h3>io/socket/sql/client_connection</h3> 
<p>This event means the database is busy creating threads to handle incoming new connections. It indicates connection churn, where clients open and close connections rapidly.</p> 
<p>What to do: Introduce or right-size application-side connection pooling so connections are reused, and consider <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html" target="_blank" rel="noopener">Amazon RDS Proxy</a> to pool and reuse connections. This is an application connectivity problem rather than a query problem, so scaling the instance will not fix it.</p> 
<h3>io/aurora_respond_to_client (Aurora MySQL-specific)</h3> 
<p>This event is specific to Aurora MySQL. On RDS for MySQL, time spent returning results appears under standard network and socket waits.</p> 
<p>What to do: Return only the columns and rows the application needs, paginate large result sets, and check for network or client-side processing delays that keep the session open while it waits to return data.</p> 
<h3>Locking waits</h3> 
<p>When sessions wait on row or table locks, DB load rises even when CPU isn’t saturated, and throughput stalls under concurrency.</p> 
<p>What to do: Keep transactions short, access rows in a consistent order to avoid deadlocks, and avoid long-running transactions that hold locks. Review isolation level and application retry logic for lock-wait timeouts and deadlocks.</p> 
<h2>Triage across a fleet with the Fleet Health Dashboard</h2> 
<p>When you run many databases, work top down: start at the fleet, rank instances by load, then drop into the per-instance wait and SQL views for the busiest ones. That triage starts with the Database Insights <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Database-Insights-Fleet-Health-Dashboard.html" target="_blank" rel="noopener">Fleet Health Dashboard</a>, which shows DB load and health across a group of databases, so you can rank instances by load and open the busiest ones first.</p> 
<p>The following figure shows the Fleet Health Dashboard ranking databases by DB load utilization, the ratio of DB load to the number of vCPUs. DB load shown here publishes to CloudWatch as the DBLoad metric, along with its DBLoadCPU and DBLoadNonCPU components. You can alarm on it and add it to CloudWatch dashboards next to the rest of your monitoring.</p> 
<div class="wp-caption alignnone">
 <a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5871-3.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5871-3.png" alt="CloudWatch Database Insights Fleet Health Dashboard showing two Aurora MySQL instances ranked by DB load utilization, with the writer under load at over 100% and the reader idle, alongside a panel listing the top instance’s top queries and top wait events." width="800"></a>
 <p class="wp-caption-text">Figure 3: Fleet Health Dashboard ranking Aurora MySQL instances by DB load utilization</p>
</div> 
<p>Here the writer instance tops the ranking at over 100 percent of its vCPU capacity, while the reader sits idle. Selecting the busiest resource surfaces its top queries and top wait events (in this case <code>CPU</code> at about 73 percent and <code>io/table/sql/handler</code> at about 20 percent). From there, you move from fleet triage straight into the per-instance playbook. Use fleet filters based on tags (for example, environment or application) to focus on the databases that matter.</p> 
<p>This flow, from fleet to instance to wait to SQL, keeps investigations fast and repeatable, and it scales as you add databases.</p> 
<h2>Conclusion</h2> 
<p>In this post, we showed how to diagnose Amazon Aurora MySQL-Compatible Edition and Amazon RDS for MySQL performance using Amazon CloudWatch Database Insights. We covered how Database Insights represents load as average active sessions against instance capacity, how to slice load by wait event and pivot to the top SQL, what the most common MySQL wait events mean, and how to triage across a fleet. Reading wait events this way helps you fix the actual bottleneck rather than defaulting to a larger instance. The cause might be a query, an index, a connection pattern, or contention.</p> 
<p>To learn more, see <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Managing.Tuning.wait-events.html" target="_blank" rel="noopener">Tuning Aurora MySQL with wait events</a>, <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Database-Insights-Get-Started.html" target="_blank" rel="noopener">Get started with CloudWatch Database Insights</a>, and <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Managing.Tuning.concepts.html" target="_blank" rel="noopener">Essential concepts for Aurora MySQL tuning</a>. For related guidance on the AWS Database Blog, see <a href="https://aws.amazon.com/blogs/database/amazon-cloudwatch-database-insights-applied-in-real-scenarios/" target="_blank" rel="noopener">Amazon CloudWatch Database Insights applied in real scenarios</a> and <a href="https://aws.amazon.com/blogs/database/analyze-amazon-aurora-mysql-workloads-with-performance-insights/" target="_blank" rel="noopener">Analyze Amazon Aurora MySQL workloads with Performance Insights</a>. If you have questions or suggestions, leave a comment, or visit <a href="https://repost.aws/" target="_blank" rel="noopener">AWS re:Post</a>.</p> 
<p> </p> 
<hr> 
<h2>About the author</h2> 
<footer> 
 <div class="blog-author-box"> 
  <div class="blog-author-image">
   <img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2026/11/10/DBBLOG-5871-4.jpg" alt="Aayushi Mittal" width="100" height="133">
  </div> 
  <h3 class="lb-h4">Aayushi Mittal</h3> 
  <p>Aayushi is a Technical Account Manager at AWS, where she helps enterprise customers keep Amazon Aurora and Amazon RDS databases healthy and performant. She specializes in memory and performance troubleshooting, and is driven by turning complex operational incidents into clear, actionable guidance her customers can act on with confidence.</p> 
 </div> 
</footer>]]></content:encoded>
    <pubDate>Wed, 11 Nov 2026 16:46:38 +0000</pubDate>
    <dc:creator>AWS Database Blog</dc:creator>
    <category>Advanced (300)</category>
    <category>Amazon Aurora</category>
    <category>Amazon CloudWatch</category>
    <category>Best Practices</category>
  </item>

  <item>
    <title>MySQL Community Spotlight - Omar Farhat</title>
    <guid isPermaLink="false">planet-manual-e9972edd38001f63290ceae1c8c54e3b</guid>
    <link>https://github.com/mysql/mysql-community/wiki/Omar-Farhat</link>
    <description>Omar shares how he got started with MySQL, his experience at the contributor summit, and his enthusiasm for replication performance, extensibility, and data interoperability.</description>
    <pubDate>Wed, 11 Nov 2026 12:55:35 +0000</pubDate>
    <dc:creator>MySQL Community Team</dc:creator>
  </item>

  <item>
    <title>DuckDB × MySQL Day: Eight Talks, One Question</title>
    <guid isPermaLink="false">https://oursqlfoundation.org/blog/duckdb-mysql-day-eight-talks/</guid>
    <link>https://oursqlfoundation.org/blog/duckdb-mysql-day-eight-talks/</link>
    <description>For about twenty years people have been asking the same thing about MySQL: can I please just run my reports on it? And for about twenty years the honest answer was “sort of, but you will regret it on a big table”.
On November 17 we are running DuckDB × MySQL Day. It’s a free online conference: one day, eight talks. The agenda was not random. If you read it top to bottom it is basically one long argument.</description>
    <content:encoded><![CDATA[<p>For about twenty years people have been asking the same thing about MySQL: <em>can I please just run my reports on it?</em> And for about twenty years the honest answer was “sort of, but you will regret it on a big table”.</p>
<p>On November 17 we are running <a href="https://oursqlfoundation.org/events/duckdb-mysql-day/">DuckDB × MySQL Day</a>. It’s a free online conference: one day, eight talks. The agenda was not random. If you read it top to bottom it is basically one long argument.</p>]]></content:encoded>
    <pubDate>Wed, 11 Nov 2026 00:00:00 +0000</pubDate>
    <dc:creator>OurSQL Foundation</dc:creator>
  </item>

  <item>
    <title>Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL</title>
    <guid isPermaLink="false">https://www.percona.com/?p=56003</guid>
    <link>https://www.percona.com/blog/heavy-mysql-reports-in-seconds-open-source-htap-with-dbtrail-and-percona-server-for-mysql/</link>
    <description>In Part 1 we ran analytical reports in DuckDB on a DBTrail copy instead of on MySQL. This time I want to look at a different question: what if the application does not need to know the copy exists?
Most MySQL servers have a few queries that make someone wait. The monthly report that aggregates a whole table. Page 5,000 of a sorted list. The screen that reads years of history. MySQL is built for many small reads and writes, and these queries are a different kind of work.
So what happens to them with DBTrail next to the server? The same 14 reports ran for a day on a Percona Server for MySQL replica and on the copy of the same database that DBTrail keeps.
Short answer: twelve were faster on the copy, and the slowest went from 225 seconds to 7.94 seconds.
Not everything got faster, though. The two lightest reports were faster on Percona Server: one takes 4.0 ms there and 103 ms on the copy. Both sides ran the same SQL.


Full disclosure: I did not run this benchmark myself. It was run and published by the DBTrail project (routing guide at dbtrail.com/docs): 14 reports, an average of 317 runs each, on a 200-warehouse order-entry database kept under constant writes by HammerDB, with 4 CPUs on each side. Each engine was timed answering the reports by itself. The copy ran one to five minutes behind the source; the replica had no lag. Over 4,436 paired rounds the copy was 6.5x faster (geometric mean).
The 225-second report
The slowest of the 14 is the kind of query every reporting screen ends up with sooner or later. It adds up every order line and every stock row and joins them by item:WITH demand AS (
  SELECT ol_i_id AS i_id, SUM(ol_quantity) AS units, COUNT(*) AS order_lines,
         SUM(CASE WHEN ol_delivery_d IS NULL THEN ol_quantity ELSE 0 END) AS pending_units
  FROM   order_line
  WHERE  ol_quantity &amp;gt;= 1
  GROUP  BY ol_i_id
), supply AS (
  SELECT s_i_id AS i_id, SUM(s_quantity) AS on_hand, SUM(s_ytd) AS shipped_ytd,
         SUM(CASE WHEN s_data LIKE '%ORIGINAL%' THEN 1 ELSE 0 END) AS brand_warehouses
  FROM   stock
  GROUP  BY s_i_id
)
SELECT i.i_id, i.i_name, i.i_price,
       d.units, d.order_lines, d.pending_units,
       s.on_hand, s.shipped_ytd, s.brand_warehouses,
       ROUND(1.0 * s.on_hand / d.units, 3) AS cover_ratio
FROM   demand d
JOIN   supply s ON s.i_id = d.i_id
JOIN   item i   ON i.i_id = d.i_id
ORDER  BY cover_ratio, d.units DESC, i.i_id
LIMIT  20This is Q14 of tplytics, the report set used in the benchmark. The minimum quantity (1 or 5) and the limit (20 or 50) change from run to run.
No index will fix this one, because the query has to read both tables from end to end. On the replica that takes 225 seconds. On the copy, where the data is stored by column and scanned in parallel, it takes 7.94 seconds.
HTAP with two open source pieces
HTAP, hybrid transactional/analytical processing, means the application’s small transactions and the heavy analytical queries are served side by side. The reports that read this database do not need a warehouse loaded for them. In practice the heavy ones are reports, deep pagination, and queries over years of history.
The basic idea is simple. It takes two open source pieces. Percona Server keeps answering the transactions. DBTrail, an independent project under Apache 2.0, follows the server the way a replica does and answers the heavy queries from its own columnar copy.
I show it here on Percona Server, but it works the same way with MySQL from Oracle, including RDS and Aurora, and with MariaDB.

What the application sees
To the application, DBTrail is just a MySQL server. It connects to DBTrail’s port over the MySQL protocol, with the driver it already uses and the SQL it already sends. No query is rewritten and no query is tagged.$ mysql -h  -P 3309 -u shop -p shopFor most SELECTs, DBTrail first asks the server for its plan, the same way EXPLAIN does, and the plan decides who answers:



Statement
Answered by
Why




SELECT * FROM orders WHERE id = 42
Percona Server
The plan finds a few rows through an index. The answer is live, to the last write.


SELECT status, COUNT(*) FROM orders GROUP BY status
DBTrail’s copy
The plan has a cost of 10,000 or more, or reads a whole table of 100,000 rows or more.


A heavy SELECT with GROUP_CONCAT, DATE_FORMAT or a JSON function
Percona Server
The copy would answer it differently, or cannot. A correct answer comes before a fast one.


Any SELECT inside a transaction
Percona Server
A read after your own write has to see that write.


A SELECT that returns more than 10,000 rows
Percona Server
The copy answers reports, not exports.



The full rules are in the routing guide. What I like here is that nobody maintains a list of queries. A query that is light today and heavy next year goes to the copy when its plan changes.
Column names in the result are the server’s, whoever answers, so code that reads a result by column name keeps working. And before you move an application, the sql-compare command runs its queries on both sides and lists the ones that would answer differently.
DBTrail’s web console counts who answered what, so you can see the split for your own workload:

Writes and reads on one connection
DBTrail’s port only reads. If your application uses one connection for everything, put ProxySQL in front. Writes, transactions and locking reads go to Percona Server; every other SELECT goes to DBTrail, which then decides between the server and the copy as above.
In ProxySQL that is two servers and two rules, with the application’s user defaulting to hostgroup 10:INSERT INTO mysql_servers (hostgroup_id, hostname, port) VALUES
  (10, 'percona-server-host', 3306),
  (20, 'dbtrail-host', 3309);

INSERT INTO mysql_query_rules (rule_id, active, match_digest, destination_hostgroup, apply) VALUES
  (1, 1, '^SELECT .* FOR (UPDATE|SHARE)', 10, 1),
  (2, 1, '^SELECT ', 20, 1);The application keeps its host, its port and its login.
But does the application actually feel the difference? This setup was measured with a MariaDB source, an application at a fixed 400 transactions per second, and three reports running in a loop. It is the only run that measures the application and the reports together, so for me it is the most interesting one:



Where the reports ran
Application latency, p95
Reports finished in 5 minutes




No reports running
13.7 ms
0


On the database
6,960 ms
12


Through DBTrail
13.2 ms
266



MariaDB 10.11, ProxySQL 3.1, sysbench. The copy was 0.9 to 3.0 seconds behind. Benchmark by the DBTrail project.
With the reports on the database, p95 latency went from 13.7 ms to 6,960 ms. Through DBTrail it stayed at 13.2 ms, and 22 times more reports finished.
One of those three reports was a page five million rows deep, ORDER BY k LIMIT 20 OFFSET 5000000. It took 92 seconds on the database and 0.7 seconds through DBTrail.
What it takes to run



Part
What it needs




Application
Point its reads at another host and port. Same driver, same SQL, no code to rewrite.


Database
Nothing is installed in it. It needs the binary log in ROW format with full row images, a primary key on every InnoDB table, and two users.


Hardware
One machine next to the database. In the 14-report benchmark it had 4 CPUs, the same as the replica. The copy is columnar files on local disk or in S3.


Cost
Apache 2.0, with no license fee. For these queries there is no warehouse to load and no pipeline to maintain.



The quick start is five steps on a machine with Docker Compose:curl -fsSL https://raw.githubusercontent.com/dbtrail/dbtrail/main/install.sh | sh

Install with the line above. It opens a web form.
Fill in the host and port of your server. The form shows the SQL that creates DBTrail’s user.
Take the first copy. It reads every table once, with a short lock, so pick a quiet time. Then turn on “Update the copy”.
Turn on the MySQL port.
Connect with your mysql client and run your slowest report.

Routing between the server and the copy is one more user and one switch, described in the routing guide.
Try it on your own report
These are DBTrail’s numbers on DBTrail’s workload. The only benchmark that really matters is your own. Put DBTrail on a separate machine, point it at a replica, and run your slowest report on both.
The documentation lists MySQL and Percona Server for MySQL 8.0 and 8.4, and MariaDB 10.11 and later. Install steps and the routing guide are at dbtrail.com/docs, and the source is at github.com/dbtrail/dbtrail.
If you try it, I am looking for your feedback: which of your queries went to the copy, and how did they do?
The post Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL appeared first on Percona.</description>
    <content:encoded><![CDATA[<p>In <a href="https://www.percona.com/blog/dbtrail-analytical-reports-on-your-mysql-data-without-running-them-on-mysql-part-1/">Part 1</a> we ran analytical reports in DuckDB on a DBTrail copy instead of on MySQL. This time I want to look at a different question: what if the application does not need to know the copy exists?</p>
<p>Most MySQL servers have a few queries that make someone wait. The monthly report that aggregates a whole table. Page 5,000 of a sorted list. The screen that reads years of history. MySQL is built for many small reads and writes, and these queries are a different kind of work.</p>
<p>So what happens to them with DBTrail next to the server? The same 14 reports ran for a day on a Percona Server for MySQL replica and on the copy of the same database that DBTrail keeps.</p>
<p><strong>Short answer: twelve were faster on the copy, and the slowest went from 225 seconds to 7.94 seconds.</strong></p>
<p>Not everything got faster, though. The two lightest reports were faster on Percona Server: one takes <strong>4.0 ms</strong> there and <strong>103 ms</strong> on the copy. Both sides ran the same SQL.</p>
<p><!-- IMAGE 1: upload chart-1-slowest-vs-lightest-report.png to the Media Library and replace the src below --><br>
<a href="https://www.percona.com/wp-content/uploads/2026/11/1-chart-14-reports.png"><img loading="lazy" decoding="async" class="aligncenter size-large wp-image-56005" src="https://www.percona.com/wp-content/uploads/2026/11/1-chart-14-reports-1024x805.png" alt="" width="1024" height="805" srcset="https://www.percona.com/wp-content/uploads/2026/11/1-chart-14-reports-1024x805.png 1024w, https://www.percona.com/wp-content/uploads/2026/11/1-chart-14-reports-300x236.png 300w, https://www.percona.com/wp-content/uploads/2026/11/1-chart-14-reports-768x603.png 768w, https://www.percona.com/wp-content/uploads/2026/11/1-chart-14-reports.png 1344w" sizes="auto, (max-width: 1024px) 100vw, 1024px"></a></p>
<p>Full disclosure: I did not run this benchmark myself. It was run and published by the DBTrail project (<a href="https://dbtrail.com/docs">routing guide at dbtrail.com/docs</a>): 14 reports, an average of 317 runs each, on a 200-warehouse order-entry database kept under constant writes by HammerDB, with 4 CPUs on each side. Each engine was timed answering the reports by itself. The copy ran one to five minutes behind the source; the replica had no lag. Over 4,436 paired rounds the copy was <strong>6.5x faster</strong> (geometric mean).</p>
<h2>The 225-second report</h2>
<p>The slowest of the 14 is the kind of query every reporting screen ends up with sooner or later. It adds up every order line and every stock row and joins them by item:</p><pre class="urvanov-syntax-highlighter-plain-tag">WITH demand AS (
  SELECT ol_i_id AS i_id, SUM(ol_quantity) AS units, COUNT(*) AS order_lines,
         SUM(CASE WHEN ol_delivery_d IS NULL THEN ol_quantity ELSE 0 END) AS pending_units
  FROM   order_line
  WHERE  ol_quantity &gt;= 1
  GROUP  BY ol_i_id
), supply AS (
  SELECT s_i_id AS i_id, SUM(s_quantity) AS on_hand, SUM(s_ytd) AS shipped_ytd,
         SUM(CASE WHEN s_data LIKE '%ORIGINAL%' THEN 1 ELSE 0 END) AS brand_warehouses
  FROM   stock
  GROUP  BY s_i_id
)
SELECT i.i_id, i.i_name, i.i_price,
       d.units, d.order_lines, d.pending_units,
       s.on_hand, s.shipped_ytd, s.brand_warehouses,
       ROUND(1.0 * s.on_hand / d.units, 3) AS cover_ratio
FROM   demand d
JOIN   supply s ON s.i_id = d.i_id
JOIN   item i   ON i.i_id = d.i_id
ORDER  BY cover_ratio, d.units DESC, i.i_id
LIMIT  20</pre><p>This is Q14 of <a href="https://github.com/Percona-Lab/tplytics">tplytics</a>, the report set used in the benchmark. The minimum quantity (1 or 5) and the limit (20 or 50) change from run to run.</p>
<p><strong>No index will fix this one</strong>, because the query has to read both tables from end to end. On the replica that takes <strong>225 seconds</strong>. On the copy, where the data is stored by column and scanned in parallel, it takes <strong>7.94 seconds</strong>.</p>
<h2>HTAP with two open source pieces</h2>
<p>HTAP, hybrid transactional/analytical processing, means the application’s small transactions and the heavy analytical queries are served side by side. The reports that read this database do not need a warehouse loaded for them. In practice the heavy ones are reports, deep pagination, and queries over years of history.</p>
<p><strong>The basic idea is simple.</strong> It takes two open source pieces. Percona Server keeps answering the transactions. DBTrail, an independent project under Apache 2.0, follows the server the way a replica does and answers the heavy queries from its own columnar copy.</p>
<p>I show it here on Percona Server, but it works the same way with MySQL from Oracle, including RDS and Aurora, and with MariaDB.</p>
<p><a href="https://www.percona.com/wp-content/uploads/2026/11/2-diagram-how-it-routes.png"><img loading="lazy" decoding="async" class="aligncenter size-large wp-image-56006" src="https://www.percona.com/wp-content/uploads/2026/11/2-diagram-how-it-routes-1024x428.png" alt="" width="1024" height="428" srcset="https://www.percona.com/wp-content/uploads/2026/11/2-diagram-how-it-routes-1024x428.png 1024w, https://www.percona.com/wp-content/uploads/2026/11/2-diagram-how-it-routes-300x125.png 300w, https://www.percona.com/wp-content/uploads/2026/11/2-diagram-how-it-routes-768x321.png 768w, https://www.percona.com/wp-content/uploads/2026/11/2-diagram-how-it-routes.png 1344w" sizes="auto, (max-width: 1024px) 100vw, 1024px"></a></p>
<h2>What the application sees</h2>
<p>To the application, DBTrail is just a MySQL server. It connects to DBTrail’s port over the MySQL protocol, with the driver it already uses and the SQL it already sends. No query is rewritten and no query is tagged.</p><pre class="urvanov-syntax-highlighter-plain-tag">$ mysql -h  -P 3309 -u shop -p shop</pre><p>For most SELECTs, DBTrail first asks the server for its plan, the same way EXPLAIN does, and the plan decides who answers:</p>
<table>
<thead>
<tr>
<th>Statement</th>
<th>Answered by</th>
<th>Why</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>SELECT * FROM orders WHERE id = 42</code></td>
<td>Percona Server</td>
<td>The plan finds a few rows through an index. The answer is live, to the last write.</td>
</tr>
<tr>
<td><code>SELECT status, COUNT(*) FROM orders GROUP BY status</code></td>
<td>DBTrail’s copy</td>
<td>The plan has a cost of 10,000 or more, or reads a whole table of 100,000 rows or more.</td>
</tr>
<tr>
<td>A heavy SELECT with GROUP_CONCAT, DATE_FORMAT or a JSON function</td>
<td>Percona Server</td>
<td>The copy would answer it differently, or cannot. A correct answer comes before a fast one.</td>
</tr>
<tr>
<td>Any SELECT inside a transaction</td>
<td>Percona Server</td>
<td>A read after your own write has to see that write.</td>
</tr>
<tr>
<td>A SELECT that returns more than 10,000 rows</td>
<td>Percona Server</td>
<td>The copy answers reports, not exports.</td>
</tr>
</tbody>
</table>
<p>The full rules are in the <a href="https://dbtrail.com/docs/guides/query-with-a-mysql-client/routing/how-it-decides">routing guide</a>. What I like here is that nobody maintains a list of queries. A query that is light today and heavy next year goes to the copy when its plan changes.</p>
<p>Column names in the result are the server’s, whoever answers, so code that reads a result by column name keeps working. And before you move an application, the <code>sql-compare</code> command runs its queries on both sides and lists the ones that would answer differently.</p>
<p>DBTrail’s web console counts who answered what, so you can see the split for your own workload:</p>
<p><a href="https://www.percona.com/wp-content/uploads/2026/11/3-console-engine-routing.png"><img loading="lazy" decoding="async" class="aligncenter size-large wp-image-56007" src="https://www.percona.com/wp-content/uploads/2026/11/3-console-engine-routing-856x1024.png" alt="" width="856" height="1024" srcset="https://www.percona.com/wp-content/uploads/2026/11/3-console-engine-routing-856x1024.png 856w, https://www.percona.com/wp-content/uploads/2026/11/3-console-engine-routing-251x300.png 251w, https://www.percona.com/wp-content/uploads/2026/11/3-console-engine-routing-768x919.png 768w, https://www.percona.com/wp-content/uploads/2026/11/3-console-engine-routing.png 1192w" sizes="auto, (max-width: 856px) 100vw, 856px"></a></p>
<h2>Writes and reads on one connection</h2>
<p>DBTrail’s port only reads. If your application uses one connection for everything, put ProxySQL in front. Writes, transactions and locking reads go to Percona Server; every other SELECT goes to DBTrail, which then decides between the server and the copy as above.</p>
<p>In ProxySQL that is two servers and two rules, with the application’s user defaulting to hostgroup 10:</p><pre class="urvanov-syntax-highlighter-plain-tag">INSERT INTO mysql_servers (hostgroup_id, hostname, port) VALUES
  (10, 'percona-server-host', 3306),
  (20, 'dbtrail-host', 3309);

INSERT INTO mysql_query_rules (rule_id, active, match_digest, destination_hostgroup, apply) VALUES
  (1, 1, '^SELECT .* FOR (UPDATE|SHARE)', 10, 1),
  (2, 1, '^SELECT ', 20, 1);</pre><p>The application keeps its host, its port and its login.</p>
<p>But does the application actually feel the difference? This setup was measured with a MariaDB source, an application at a fixed 400 transactions per second, and three reports running in a loop. It is the only run that measures the application and the reports together, so for me it is the most interesting one:</p>
<table>
<thead>
<tr>
<th>Where the reports ran</th>
<th>Application latency, p95</th>
<th>Reports finished in 5 minutes</th>
</tr>
</thead>
<tbody>
<tr>
<td>No reports running</td>
<td>13.7 ms</td>
<td>0</td>
</tr>
<tr>
<td>On the database</td>
<td>6,960 ms</td>
<td>12</td>
</tr>
<tr>
<td>Through DBTrail</td>
<td>13.2 ms</td>
<td>266</td>
</tr>
</tbody>
</table>
<p><em>MariaDB 10.11, ProxySQL 3.1, sysbench. The copy was 0.9 to 3.0 seconds behind. Benchmark by the DBTrail project.</em></p>
<p><strong>With the reports on the database, p95 latency went from 13.7 ms to 6,960 ms. Through DBTrail it stayed at 13.2 ms, and 22 times more reports finished.</strong></p>
<p>One of those three reports was a page five million rows deep, <code>ORDER BY k LIMIT 20 OFFSET 5000000</code>. It took 92 seconds on the database and 0.7 seconds through DBTrail.</p>
<h2>What it takes to run</h2>
<table>
<thead>
<tr>
<th>Part</th>
<th>What it needs</th>
</tr>
</thead>
<tbody>
<tr>
<td>Application</td>
<td>Point its reads at another host and port. Same driver, same SQL, no code to rewrite.</td>
</tr>
<tr>
<td>Database</td>
<td>Nothing is installed in it. It needs the binary log in ROW format with full row images, a primary key on every InnoDB table, and two users.</td>
</tr>
<tr>
<td>Hardware</td>
<td>One machine next to the database. In the 14-report benchmark it had 4 CPUs, the same as the replica. The copy is columnar files on local disk or in S3.</td>
</tr>
<tr>
<td>Cost</td>
<td>Apache 2.0, with no license fee. For these queries there is no warehouse to load and no pipeline to maintain.</td>
</tr>
</tbody>
</table>
<p>The <a href="https://dbtrail.com/docs/quickstart">quick start</a> is five steps on a machine with Docker Compose:</p><pre class="urvanov-syntax-highlighter-plain-tag">curl -fsSL https://raw.githubusercontent.com/dbtrail/dbtrail/main/install.sh | sh</pre><p></p>
<ol>
<li>Install with the line above. It opens a web form.</li>
<li>Fill in the host and port of your server. The form shows the SQL that creates DBTrail’s user.</li>
<li>Take the first copy. It reads every table once, with a short lock, so pick a quiet time. Then turn on “Update the copy”.</li>
<li>Turn on the MySQL port.</li>
<li>Connect with your mysql client and run your slowest report.</li>
</ol>
<p>Routing between the server and the copy is one more user and one switch, described in the <a href="https://dbtrail.com/docs/guides/query-with-a-mysql-client/routing">routing guide</a>.</p>
<h2>Try it on your own report</h2>
<p>These are DBTrail’s numbers on DBTrail’s workload. The only benchmark that really matters is your own. Put DBTrail on a separate machine, point it at a replica, and run your slowest report on both.</p>
<p>The documentation lists MySQL and Percona Server for MySQL 8.0 and 8.4, and MariaDB 10.11 and later. Install steps and the routing guide are at <a href="https://dbtrail.com/docs">dbtrail.com/docs</a>, and the source is at <a href="https://github.com/dbtrail/dbtrail">github.com/dbtrail/dbtrail</a>.</p>
<p>If you try it, I am looking for your feedback: which of your queries went to the copy, and how did they do?</p>
<p>The post <a href="https://www.percona.com/blog/heavy-mysql-reports-in-seconds-open-source-htap-with-dbtrail-and-percona-server-for-mysql/">Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL</a> appeared first on <a href="https://www.percona.com/">Percona</a>.</p>]]></content:encoded>
    <pubDate>Tue, 10 Nov 2026 21:04:03 +0000</pubDate>
    <dc:creator>MySQL Performance Blog</dc:creator>
    <category>Benchmarks</category>
    <category>Insight for Developers</category>
    <category>MySQL</category>
    <category>Open Source</category>
    <category>Uncategorized</category>
  </item>

  <item>
    <title>Diagnosing MySQL at Scale (Part 2): Storage Subsystems, Buffer Pool Latches, and Memory Spills</title>
    <guid isPermaLink="false">tag:blogger.com,1999:blog-5421473080502311460.post-89802551024168789</guid>
    <link>https://ushastry.blogspot.com/2026/11/diagnosing-mysql-at-scale-part-2.html</link>
    <description>In transactional database systems, the latency gap between memory access (tens to hundreds of nanoseconds) and persistent storage access (tens to hundreds of microseconds for local NVMe devices and milliseconds for remote cloud block storage such as Amazon EBS) spans multiple orders of magnitude. Once a workload exhausts available memory and begins interacting heavily with storage, query response times can deteriorate rapidly.

However, storage-related performance problems in MySQL are rarely solved by simply adding IOPS. In many production incidents, latency originates within the database engine itself:


  Redo-log checkpoint advancement falling behind incoming write velocity, forcing foreground transactions to participate in synchronous flushing activity (&quot;sync flush&quot;).
  Doublewrite buffering increasing write amplification, contributing to storage-pressure events and elevated flush queue depth.
  Binary log durability synchronization (fsync) serializing transaction commits, creating commit-pipeline bottlenecks under sustained write throughput.
  Buffer pool synchronization primitives and latch contention, causing worker-thread serialization during periods of intense page churn, eviction pressure, and cache turnover.
  Analytical workloads spilling large temporary datasets and filesort merge passes to disk, converting microsecond-scale in-memory operations into high-latency storage activity.
  Undo tablespace growth and MVCC purge lag, driven by long-running transactions, abandoned read views, and purge workers falling behind transactional churn.


This article explores the diagnosis and remediation of storage and memory contention patterns in MySQL 8.4 LTS, including wait-event analysis, checkpoint behavior, buffer-pool contention, temporary-table spills, purge worker lag, and transaction-commit bottlenecks.


  Operational Disclaimer: The diagnostic queries, Performance Schema instrumentation configurations, engine parameter recommendations, stored routines, and architectural remediations presented across this series are provided for investigative engineering analysis and technical reference. Contention characteristics, wait-event profiles, and optimal parameter thresholds vary significantly depending on transactional access patterns, data volume, schema design, storage subsystem throughput, replication topology, cloud virtualization constraints (e.g., AWS Nitro, EBS bandwidth limits, Aurora multi-tenant architectures), and specific MySQL engine release baselines. Administrative overrides, dynamic parameter modifications, session terminations (KILL), and schema modifications should never be executed blindly against production clusters. All queries, DDL statements, and configuration changes must be reviewed, benchmarked, and validated in representative staging environments through established change-management procedures prior to production implementation.




1. Storage &amp;amp; I/O Subsystem Bottlenecks: Taxonomy


  
    
      Wait Event
      Production Root Cause &amp;amp; Impact
      Recommended Engineering Remediation
    
  
  
    
      wait/io/file/innodb/innodb_redo_log
      Transaction commits stall during sustained write bursts because redo-log generation outpaces checkpoint advancement and flushing activity, forcing foreground transactions to participate in synchronous flushing.
      Batch write operations into larger transactions where appropriate; increase innodb_log_buffer_size (commonly 64M–128M for write-intensive workloads); in MySQL 8.4 LTS, size innodb_redo_log_capacity appropriately to absorb sustained write bursts while maintaining healthy checkpoint-age progression.
    
    
      wait/io/file/innodb/innodb_doublewrite_file
      Doublewrite buffering increases write amplification and may become a storage bottleneck during aggressive dirty-page flushing.
      Self-managed MySQL: tune innodb_doublewrite_files and evaluate storage throughput. AWS RDS for MySQL: consider RDS Optimized Writes where supported. Aurora MySQL: Aurora uses a different storage architecture and does not rely on the traditional InnoDB doublewrite mechanism.
    
    
      wait/io/file/innodb/innodb_data_file
      Synchronous page reads occur when required data is absent from the buffer pool, resulting in storage-access latency and throughput degradation.
      Increase innodb_buffer_pool_size where memory permits; target a sustained buffer-pool hit ratio near 99%+ for OLTP workloads; add appropriate composite and covering indexes to eliminate unnecessary table scans and random I/O.
    
    
      wait/io/file/sql/binlog
      Binary log durability synchronization (fsync) and commit-order preservation limit transaction throughput under heavy write concurrency.
      Do not disable sync_binlog solely for performance reasons in production. First evaluate binlog cache spillover, increase binlog_cache_size where warranted, and verify Binlog Group Commit efficiency before considering durability trade-offs.
    
  




2. Redo Log Checkpoint Saturation (innodb_redo_log)

Contention on wait/io/file/innodb/innodb_redo_log occurs when transactional workloads generate redo records faster than InnoDB can advance checkpoints and flush dirty pages. As redo generation outpaces checkpoint progression, foreground transactions increasingly experience delays associated with redo-log pressure and flushing activity.

Official Documentation: MySQL 8.4 Reference Manual: InnoDB Redo Log.

Production Failure Scenario: Checkpoint Pressure Under Sustained Write Load

Under sustained write concurrency, the gap between the current Log Sequence Number (LSN) and the most recent checkpoint grows continuously. This gap, commonly referred to as Checkpoint Age, represents the volume of uncheckpointed redo that must be retained for crash recovery.

As Checkpoint Age grows, InnoDB progressively increases background flushing pressure through adaptive flushing mechanisms. If dirty-page generation continues to outpace checkpoint advancement, foreground transactions may begin participating in flushing activity, increasing transaction latency and producing elevated waits on wait/io/file/innodb/innodb_redo_log.


  


Unlike traditional storage bottlenecks, the underlying issue is often not raw IOPS capacity alone, but the inability of checkpoint advancement and page flushing to keep pace with incoming write velocity.

Diagnostic Commands &amp;amp; Calculation

1. Inspect Current LSN and Checkpoint Position:

SHOW ENGINE INNODB STATUS\G

-- Locate the LOG section:
-- Log sequence number          214748364800
-- Log flushed up to            214748364750
-- Pages flushed up to          214648364800
-- Last checkpoint at           213248364800

2. Calculate Checkpoint Age:

-- Checkpoint Age = (Log sequence number - Last checkpoint at)
-- 214748364800 - 213248364800 = 1,500,000,000 bytes (~1.4 GB)

-- Compare this value against current redo capacity:
SHOW GLOBAL VARIABLES LIKE 'innodb_redo_log_capacity';

A steadily increasing Checkpoint Age, combined with elevated redo-log waits and aggressive flushing activity, is a strong indicator that checkpoint advancement is struggling to keep pace with redo generation.

3. Monitor Dirty Page Pressure:

-- Compute dirty-page and data-page metrics in the buffer pool
SELECT VARIABLE_NAME, VARIABLE_VALUE 
  FROM performance_schema.global_status 
 WHERE VARIABLE_NAME IN ('Innodb_buffer_pool_pages_dirty', 'Innodb_buffer_pool_pages_data');

A persistently high dirty-page ratio accompanied by growing redo-log waits indicates checkpoint pressure and insufficient flushing throughput.

Remediation &amp;amp; Engine Configuration


  Dynamically Expand Redo Capacity (MySQL 8.4 LTS):
  In MySQL 8.4 LTS, innodb_redo_log_capacity is dynamic and can be resized online without requiring a server restart:
-- Expand redo capacity to better absorb sustained write bursts
SET GLOBAL innodb_redo_log_capacity = 8589934592; -- 8 GB
SHOW GLOBAL VARIABLES LIKE 'innodb_redo_log_capacity';
  Increasing redo capacity provides additional recovery headroom and reduces the likelihood of checkpoint-age pressure during large write bursts.

  Expand the In-Memory Log Buffer:
  For write-intensive workloads, scale innodb_log_buffer_size (typical production ranges: 64 MB–128 MB). This reduces contention and allocation pressure associated with large transactions and bursty write workloads.

  Verify Storage Throughput and Latency:
  Redo writes ultimately depend on sequential storage durability operations (write() and fsync()). Validate that underlying storage can sustain expected throughput:
    
      Local NVMe SSDs, AWS EBS io2, or provisioned AWS EBS gp3.
      Monitor write latency, queue depth, flush latency spikes, and storage throughput saturation.
    
  

  Reduce Redo Generation Where Possible:
  Reducing redo generation is often more cost-effective than continuously scaling storage throughput:
    
      Batch DML operations into larger deterministic transactions.
      Eliminate unused or redundant secondary indexes that amplify log records.
      Reduce write amplification from frequent row updates.
      Optimize bulk-ingestion and ETL workflows using chunked staging.
    
  



  Key Operational Takeaway: wait/io/file/innodb/innodb_redo_log is rarely the result of a simple disk-performance problem. In most production incidents, the root cause is an imbalance between redo generation, checkpoint advancement, and page-flushing capacity. When write velocity persistently exceeds flushing throughput, InnoDB increases flushing pressure and foreground transaction latency rises. The most effective remedies are usually a combination of appropriately sized redo capacity, healthy checkpoint progression, efficient flushing, and controlled redo generation rates.




3. Doublewrite Buffer &amp;amp; Hardware Torn-Write Prevention

The wait event wait/io/file/innodb/innodb_doublewrite_file is associated with dirty-page flushing activity within InnoDB's doublewrite subsystem. Elevated waits typically indicate sustained flushing pressure, significant dirty-page accumulation, or storage throughput limitations during checkpoint advancement and buffer-pool page eviction.

Official Documentation: MySQL 8.4 Reference Manual: InnoDB Doublewrite Buffer.

Production Mechanics &amp;amp; Write Amplification

InnoDB stores data in 16 KiB pages, while underlying operating systems and storage devices typically perform physical writes at 4 KiB sector boundaries. If a power loss, kernel crash, or hardware failure occurs mid-write, only part of the page may reach persistent storage, producing a torn page and catastrophic tablespace corruption.

To protect against this failure mode, InnoDB first writes dirty pages sequentially into the doublewrite buffer, performs the required durability operations (fsync), and then writes those pages to their final tablespace locations. During crash recovery, InnoDB reads the doublewrite buffer to repair partially written pages before normal recovery proceeds.

Because every flushed dirty page is written twice, this mechanism introduces a 2× write amplification factor. Under heavy checkpoint pressure or aggressive dirty-page flushing, elevated activity within the doublewrite subsystem becomes a primary bottleneck, visible via wait/io/file/innodb/innodb_doublewrite_file.

Production Optimization Standards

1. AWS RDS for MySQL: RDS Optimized Writes

On supported AWS Nitro-based RDS instance classes (such as Graviton db.r6g, db.r7g and Intel/AMD db.r6i, db.m6i), AWS provides RDS Optimized Writes:

# Applied via AWS RDS Parameter Group (requires instance reboot)
rds.optimized_writes = 1


  Hardware Mechanism: Nitro SSD controllers feature hardware-enforced Torn-Write Prevention (TWP), guaranteeing atomic 16 KiB page writes at the hardware controller level.
  Operational Result: MySQL automatically sets innodb_doublewrite = 0, eliminating doublewrite overhead, halving write I/O amplification, and eliminating wait/io/file/innodb/innodb_doublewrite_file entirely without compromising ACID crash recovery.
  Note: Availability depends on engine version, storage type, and instance class. Always verify support in the AWS RDS documentation before enabling.


2. Aurora MySQL Architecture Note

Aurora MySQL uses a fundamentally different storage architecture than standard InnoDB deployments:


  Instead of relying on traditional InnoDB doublewrite buffers and local data files, Aurora persists redo records directly into a distributed, six-way replicated storage fleet (&quot;the log is the database&quot;) and reconstructs data pages in the storage layer.
  Consequently, Aurora does not rely on the traditional InnoDB doublewrite mechanism, RDS Optimized Writes does not apply to Aurora, and wait/io/file/innodb/innodb_doublewrite_file analysis is not relevant in Aurora environments.


3. Self-Managed MySQL on Dedicated NVMe Storage

For self-managed cloud or bare-metal servers running MySQL 8.4 LTS, scale doublewrite worker concurrency across dedicated parallel files:

[mysqld]
# Parallelize doublewrite flushing across multiple files
innodb_doublewrite_files = 4
innodb_doublewrite_pages = 128

Benchmark optimal values based on:

  NVMe storage throughput and provisioned IOPS
  Storage queue depth
  Buffer pool size and dirty-page generation velocity
  Checkpoint workload characteristics


Diagnostic Indicators

Investigate the doublewrite subsystem when you observe wait/io/file/innodb/innodb_doublewrite_file together with:


  Elevated dirty-page counts (Innodb_buffer_pool_pages_dirty)
  Aggressive checkpoint activity and rising Checkpoint Age
  Increased page-flush rates (Innodb_buffer_pool_pages_flushed)
  Storage write throughput approaching saturation or high queue depths
  Sustained buffer-pool eviction pressure



  Key Operational Takeaway: wait/io/file/innodb/innodb_doublewrite_file is rarely the primary bottleneck. In most production incidents, it is a downstream symptom reflecting excessive page-flushing pressure driven by checkpoint advancement, dirty-page accumulation, or storage-layer saturation. The correct remediation is usually to address the underlying flushing workload and checkpoint progression rather than focusing exclusively on the doublewrite subsystem itself.




4. Data File I/O &amp;amp; Working-Set Cache Misses

The wait event wait/io/file/innodb/innodb_data_file is commonly associated with synchronous page reads that occur when required data pages are not present in the InnoDB buffer pool. In these situations, foreground threads must wait while the storage subsystem retrieves the necessary 16 KiB pages from disk before query execution can proceed.

Official Documentation:

  MySQL 8.4 Reference Manual: InnoDB Buffer Pool
  MySQL 8.4 Reference Manual: Performance Schema File I/O Instrumentation


Production Failure Scenario: Working-Set Exceeds Available Memory


  


Under sustained pressure, the entire workload can become storage-bound and IOPS-constrained even though database CPU utilization remains deceptively low.

Diagnostic Commands &amp;amp; Hit-Ratio Calculation

1. Compute Global InnoDB Buffer Pool Hit Ratio:

SELECT ROUND(
         100 * (1 - (bp_reads.variable_value / bp_requests.variable_value)),
         4
       ) AS hit_ratio_pct
  FROM performance_schema.global_status bp_reads
  JOIN performance_schema.global_status bp_requests
    ON bp_reads.variable_name = 'Innodb_buffer_pool_reads'
   AND bp_requests.variable_name = 'Innodb_buffer_pool_read_requests';

Operational Hit Ratio Standards:

  &amp;gt; 99% — Standard target for high-concurrency OLTP workloads.
  95% – 99% — Well-cached, optimal production workload.
  &amp;lt; 95% — Investigate working-set sizing, cold cache sweeps, or unindexed queries.
  Contextual Interpretation: The hit ratio should always be evaluated alongside storage latency and access patterns. Lower hit ratios are typical and expected for reporting, DSS, analytical range scans, or batch ETL workloads where scanning full tables is required.


2. Identify Queries Driving Physical Disk Reads:

SELECT DIGEST_TEXT,
       COUNT_STAR AS exec_count,
       SUM_NO_INDEX_USED AS scans_without_index,
       SUM_ROWS_EXAMINED,
       ROUND(SUM_TIMER_WAIT / 1e12, 2) AS total_latency_sec
  FROM performance_schema.events_statements_summary_by_digest
 WHERE SUM_NO_INDEX_USED &amp;gt; 0
 ORDER BY SUM_ROWS_EXAMINED DESC
 LIMIT 5;

Look for:

  Full table scans cycling cold pages through the LRU list
  Wide range scans examining large fractions of large tables
  Missing composite indexes causing secondary index to primary table lookups
  Disproportionate rows examined versus rows returned


Remediation Standards


  Size the Buffer Pool to the Active Working Set:
  Configure innodb_buffer_pool_size so the primary transactional dataset remains resident in memory. On dedicated database hosts, a standard guideline is 70–80% of available physical RAM, balanced against:
    
      Thread stack and connection memory (max_connections × thread buffers)
      Internal temporary table memory limits (temptable_max_ram)
      Replication worker buffers
      Host operating system and kernel page cache requirements
      Local backup utilities and monitoring agents
    
  

  Eliminate Full Scans Using Composite Covering Indexes:
  Problem Query:
SELECT order_id, total 
  FROM customer_orders 
 WHERE customer_id = 918 AND status = 'SHIPPED';
  Without an optimal index, MySQL performs table or range scans touching millions of disk pages.
  Remediation:
ALTER TABLE customer_orders 
  ADD INDEX idx_cust_status_covered (customer_id, status, order_id, total);
  A composite covering index satisfies both filtering and projection directly out of the index B-tree, eliminating physical clustered index page reads.

  Validate Buffer Pool Eviction Pressure:
  Inspect page turnover and eviction velocity in InnoDB status:
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads%';
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read_requests%';
SHOW ENGINE INNODB STATUS\G
  Under the BUFFER POOL AND MEMORY section, inspect:
    
      Pages read per second vs. Pages created per second
      Evicted without access rate (high values indicate LRU thrashing)
      Read-ahead efficiency (low hit rates indicate unhelpful prefetching)
    
  



  Key Operational Takeaway: wait/io/file/innodb/innodb_data_file is usually not a storage problem first; it is primarily a memory-efficiency and indexing problem. In most production incidents, elevated data-file waits indicate that the active working set no longer fits within the buffer pool, forcing foreground threads to perform synchronous disk reads. The most effective remediation is a combination of proper buffer-pool sizing, covering indexes, and query refactoring rather than simply provisioning additional IOPS.




5. Binlog Wait Diagnostic Workflow

When wait/io/file/sql/binlog dominates the wait profile, setting sync_binlog = 0 is often treated as a quick operational shortcut. In reality, this introduces severe durability and consistency risks by delegating fsync() timing entirely to the operating system page cache.

If an operating system crash, kernel panic, or power failure occurs before pending binary log writes are flushed to disk, transactions acknowledged as committed can be permanently lost. Furthermore, replicas may never receive those transactions—or may have already received transactions that are subsequently lost on the source after reboot—producing severe replication divergence (split-brain) and complicating disaster recovery.

Before modifying durability settings, engineers should execute this three-step diagnostic triage workflow.

Official Documentation: MySQL 8.4 Reference Manual: Binary Log Group Commit.

Step 1: Check Binlog Cache Spillage

SHOW GLOBAL STATUS LIKE 'Binlog_cache%';


  
    
      Status Variable
      Production Meaning
    
  
  
    
      Binlog_cache_use
      Total number of transactions that utilized the thread-local binlog cache.
    
    
      Binlog_cache_disk_use
      Transactions that exceeded binlog_cache_size and spilled into temporary disk files.
    
  


Calculate the Disk Spillage Ratio:


  Disk Spillage Ratio (%) = [Binlog_cache_disk_use / (Binlog_cache_use + Binlog_cache_disk_use)] × 100


A persistently elevated spillage ratio indicates that large transactions are exceeding available thread memory, generating temporary-file disk I/O before the binary-log synchronization phase begins.

Operational Guideline: Investigate spillage ratios exceeding 1–5%, particularly when accompanied by elevated transactional commit latency or high waits on wait/io/file/sql/binlog.

Step 2: Evaluate Cache Sizing vs. Concurrency

SHOW GLOBAL VARIABLES LIKE 'binlog_cache_size';
SHOW GLOBAL VARIABLES LIKE 'max_binlog_cache_size';

If Binlog_cache_disk_use is increasing rapidly while binlog_cache_size remains at the 32 KB default, consider scaling:

# Scale thread-local binlog cache to reduce disk spillage
binlog_cache_size = 256K .. 1M

Always validate available host memory first, as this buffer is allocated per active transactional thread:


  Max Potential Cache Memory = max_connections × binlog_cache_size


Simultaneously audit application workloads for large, unbatched operations:

  Multi-million-row UPDATE or DELETE statements
  Bulk ETL and data ingestion jobs
  Unbounded transactional loops


Chunking large transactions into deterministic batches (e.g., 1,000–5,000 rows) is far more effective and memory-safe than continuously inflating thread cache allocations.

Step 3: Optimize Binlog Group Commit (Maintain Full Durability)

SHOW GLOBAL VARIABLES LIKE 'binlog_group_commit_sync_delay';
SHOW GLOBAL VARIABLES LIKE 'binlog_group_commit_sync_no_delay_count';

Rather than compromising crash safety, leverage Binary Log Group Commit to batch concurrent transactional fsync operations into a single disk write:

[mysqld]
# Full ACID durability baseline
sync_binlog                             = 1
innodb_flush_log_at_trx_commit          = 1

# Binary Log Group Commit batching
binlog_group_commit_sync_delay          = 1000   # Pause up to 1000 microseconds (1 ms)
binlog_group_commit_sync_no_delay_count = 100    # Flush immediately if 100 transactions accumulate

Production Behavior:

  binlog_group_commit_sync_delay = 1000 (1 ms): The commit-group leader thread pauses for up to 1 millisecond before issuing fsync(), allowing concurrent transactions to join the same commit group.
  binlog_group_commit_sync_no_delay_count = 100: If 100 transactions accumulate before the 1 ms timer expires, the group flushes immediately to prevent latency degradation.


This configuration achieves write throughput comparable to sync_binlog = 0 under high-concurrency OLTP workloads while preserving 100% crash durability and preventing replication drift.


  Key Operational Takeaway: wait/io/file/sql/binlog does not justify disabling durability settings. In production, the correct remediation sequence is to eliminate binlog-cache disk spillage, right-size binlog_cache_size, enforce deterministic DML chunking, and tune Binary Log Group Commit. Durability baselines (sync_binlog = 1 and innodb_flush_log_at_trx_commit = 1) should remain non-negotiable standards wherever transactional integrity and replication consistency are business requirements.




6. Memory Contention &amp;amp; Engine Overhead


  
    
      Wait Event
      Production Root Cause &amp;amp; Impact
      Recommended Engineering Remediation
    
  
  
    
      wait/synch/mutex/innodb/buf_pool_mutex
      Concurrent sessions contend for synchronization protecting InnoDB buffer-pool structures while loading, flushing, evicting, or modifying cached data pages. Elevated waits are commonly associated with buffer-pool pressure, excessive page churn, storage misses, or cache-efficiency problems rather than mutex overhead alone.
      Verify innodb_buffer_pool_size is sufficient for the active working set; investigate page-read, dirty-page, and eviction rates; eliminate unnecessary scans; validate storage latency; and reduce buffer-pool churn. MySQL 8.4 automatically manages buffer-pool parallelism and internal partitioning, significantly reducing historical buffer-pool mutex contention.
    
    
      wait/io/file/sql/FILE_TMPFILE
      Internal temporary tables, filesort merge passes, hash aggregation structures, or intermediate result sets exceed memory thresholds and spill to disk, converting memory operations into storage I/O.
      Tune temptable_max_ram; review tmp_table_size and max_heap_table_size where applicable; optimize GROUP BY, ORDER BY, and large joins; increase sort_buffer_size cautiously after validating per-session memory impact; and add covering indexes that allow filtering, grouping, and ordering directly from index structures.
    
    
      wait/synch/cond/innodb/purge_worker_cond
      Purge workers are coordinating, waiting for work, or processing undo cleanup tasks. Elevated History List Length (HLL), growing undo tablespaces, and long-running transactions together indicate that purge activity is falling behind transactional churn.
      Monitor HLL through INNODB_METRICS; identify long-running transactions and stale read views using information_schema.innodb_trx; investigate replication, backup, or analytical workloads holding snapshots open; and consider increasing innodb_purge_threads when purge consistently lags behind workload demand.
    
  



  Operational Analysis Standard: Wait events should never be analyzed in isolation. For memory-related waits, always correlate Performance Schema telemetry with buffer-pool statistics, temporary-table metrics, undo history growth, storage latency, and workload characteristics before identifying a root cause. A wait event often reflects downstream symptoms rather than the originating bottleneck.




7. Deep-Dive: Buffer Pool Contention, Page Churn &amp;amp; Cache Pressure (buf_pool_*)

The wait family wait/synch/mutex/innodb/buf_pool_* represents contention within InnoDB buffer-pool management structures responsible for tracking, locating, loading, evicting, and flushing data pages. In modern MySQL 8.4 LTS environments, these waits are typically downstream symptoms of excessive page churn, cache inefficiency, or storage pressure rather than a standalone mutex bottleneck.

Official Documentation:

  MySQL 8.4 Reference Manual: InnoDB Buffer Pool
  MySQL 8.4 Reference Manual: Buffer Pool Sizing &amp;amp; Resize


Production Failure Scenario: Working Set Exceeds Memory

When the active transactional working set becomes larger than available buffer pool capacity, pages are evicted faster than they can be reused. Simultaneously, unindexed analytical scans, poorly selective range filters, or elevated storage latency create a continuous cycle of cold-page loading and hot-page eviction.


  


Common Production Symptoms:

  Elevated waits on wait/synch/mutex/innodb/buf_pool_*
  Steep increase in physical synchronous disk reads (Innodb_buffer_pool_reads)
  Plummeting buffer pool cache efficiency (&amp;lt; 95%)
  Increasing disk queue depth and storage latency
  Aggressive page-flush velocity and free list exhaustion
  Near-100% CPU utilization driven by mutex spin-locks despite falling transactional throughput


Diagnostic Commands &amp;amp; Investigation

1. Identify Buffer Pool Related Waits:

SELECT EVENT_NAME,
       COUNT_STAR AS total_waits,
       ROUND(SUM_TIMER_WAIT / 1e12, 2) AS total_wait_sec,
       ROUND(AVG_TIMER_WAIT / 1e6, 2) AS avg_wait_us
  FROM performance_schema.events_waits_summary_global_by_event_name
 WHERE EVENT_NAME LIKE 'wait/synch/%/innodb/buf_pool%'
 ORDER BY SUM_TIMER_WAIT DESC;

Look for: Rapidly increasing wait counts, high cumulative wait duration, and lock thrashing across Page Hash Table latches and LRU list mutexes.

2. Inspect Buffer Pool Churn in InnoDB Status:

SHOW ENGINE INNODB STATUS\G

Under the BUFFER POOL AND MEMORY section, inspect:

  Pages read/sec, created/sec, written/sec: Gauges active page turnover rate.
  Evicted without access per second: A high rate is a definitive indicator of cold cache sweeps thrashing the LRU list.
  Free buffers: Persistently low or zero values force foreground worker threads to perform synchronous on-demand page evictions.
  Pending reads / Pending normal aio writes: Indicates whether storage queue depths are saturated.


3. Calculate Buffer Pool Hit Ratio:

SELECT ROUND(
         100 * (1 - (bp_reads.variable_value / bp_requests.variable_value)),
         4
       ) AS hit_ratio_pct
  FROM performance_schema.global_status bp_reads
  JOIN performance_schema.global_status bp_requests
    ON bp_reads.variable_name = 'Innodb_buffer_pool_reads'
   AND bp_requests.variable_name = 'Innodb_buffer_pool_read_requests';


  &amp;gt; 99% — Optimal OLTP cache efficiency.
  95% – 99% — Acceptable production baseline.
  &amp;lt; 95% — Investigate working-set sizing, cold table scans, and memory allocation.


Remediation Standards


  Size the Buffer Pool to the Active Working Set:
  Configure innodb_buffer_pool_size so the primary hot dataset remains 100% memory-resident. For dedicated MySQL instances, allocate 70–80% of physical RAM, after deducting:
    
      Thread memory allocations (max_connections × session buffers)
      Internal temporary table buffers (temptable_max_ram)
      Replication worker queues
      Host operating system and filesystem page cache requirements
      Local backup utilities and telemetry daemons
    
  

  Eliminate Unnecessary Page Churn:
  Audit and optimize queries driving cold-page sweeps through the buffer pool:
    
      Full table scans on multi-gigabyte historical tables
      Inefficient join plans cycling outer rows
      Missing composite indexes causing unindexed range scans
      Low-selectivity index predicates forcing widespread index-to-clustered-table lookups
    
  

  Buffer Pool Instances in MySQL 8.4 LTS:
  In MySQL 8.4 LTS, innodb_buffer_pool_instances is automatically calculated by InnoDB based on logical CPU count (up to 64 instances when innodb_buffer_pool_size &amp;gt;= 1GB). Manual instance tuning is obsolete; focus engineering efforts on working-set sizing, index optimization, and storage latency reduction.



  Key Operational Takeaway: Buffer-pool waits in MySQL 8.4 are usually downstream symptoms rather than root causes. When wait/synch/mutex/innodb/buf_pool_* dominates Performance Schema profiles, the underlying pathology is typically excessive page churn, insufficient memory residency, unindexed access patterns, or storage bottlenecks. The most effective remediation is improving cache efficiency and eliminating unnecessary physical I/O rather than attempting to tune mutex synchronization directly.




8. Deep-Dive: On-Disk Temp Table Spills (FILE_TMPFILE)

The wait event wait/io/file/sql/FILE_TMPFILE occurs when MySQL materializes internal temporary tables, filesort merge runs, hash aggregation structures, window-function state, or intermediate result sets onto storage because in-memory thresholds have been exceeded.

Official Documentation: MySQL 8.4 Reference Manual: Internal Temporary Tables.

Production Failure Scenario: Large Grouping, Sorting &amp;amp; Intermediate Result Materialization

Complex relational queries involving:

  GROUP BY and DISTINCT aggregations
  ORDER BY sorting operations
  Window Functions (OVER (...))
  Large multi-table hash joins
  Derived tables and Common Table Expressions (CTEs)


frequently require MySQL to construct intermediate working sets. When those working sets exceed available in-memory thresholds, MySQL spills them to disk:


  


A workload that normally completes using microsecond-scale memory operations can rapidly become storage-bound and IOPS-constrained once repeated temp-table and filesort spills begin occurring.

Representative Execution Plan Pattern

EXPLAIN ANALYZE
SELECT customer_id,
       memo,
       SUM(amount)
  FROM transaction_ledger
 GROUP BY customer_id, memo
 ORDER BY SUM(amount) DESC;

Execution plans displaying Using temporary; Using filesort do not automatically indicate a production emergency; however, they frequently correlate with severe FILE_TMPFILE waits when the intermediate row count or projected byte width exceeds in-memory buffers.

Diagnostic Commands &amp;amp; Spill Ratio Analysis

1. Monitor Temporary-Table Spill Rates:

SHOW GLOBAL STATUS LIKE 'Created_tmp%';


  
    
      Status Variable
      Production Meaning
    
  
  
    
      Created_tmp_tables
      Total internal temporary tables created by MySQL sessions.
    
    
      Created_tmp_disk_tables
      Internal temporary tables that exceeded in-memory thresholds and spilled to disk.
    
  


Calculate the Disk Spill Ratio:


  Disk Spill Ratio (%) = (Created_tmp_disk_tables / Created_tmp_tables) × 100


Operational Guidance:

  &amp;lt; 5% — Excellent in-memory caching efficiency.
  5% – 20% — Acceptable range; worth periodic monitoring during peak hours.
  &amp;gt; 20% — Investigate query design, result-set row widths, and memory allocations.
  Contextual Reality: A high spill ratio is not inherently a configuration bug for dedicated analytics, reporting, batch ETL, or data warehouse replicas, where scanning multi-gigabyte datasets legitimately requires storage-backed materialization.


2. Identify Filesort and Spill Contributors:

SELECT DIGEST_TEXT,
       COUNT_STAR AS exec_count,
       SUM_SORT_MERGE_PASSES,
       SUM_SORT_ROWS,
       ROUND(SUM_TIMER_WAIT / 1e12, 2) AS total_wait_sec
  FROM performance_schema.events_statements_summary_by_digest
 WHERE SUM_SORT_MERGE_PASSES &amp;gt; 0
 ORDER BY SUM_SORT_MERGE_PASSES DESC
 LIMIT 5;

Look for: Repeated multi-pass sort merges (SUM_SORT_MERGE_PASSES &amp;gt; 0), massive row sort volumes, and expensive analytical statements. Multiple merge passes indicate that intermediate sort chunks could not fit within sort_buffer_size, requiring external merge sort runs on disk.

Remediation &amp;amp; Engine Configuration


  Tune the TempTable Engine Memory Budget (MySQL 8.4 LTS):
  In MySQL 8.4 LTS, temptable_max_ram governs how much memory the TempTable engine can consume before materializing temporary tables to disk:
-- Dynamically expand TempTable in-memory limit
SET GLOBAL temptable_max_ram = 1073741824; -- 1 GiB
SHOW GLOBAL VARIABLES LIKE 'temptable_max_ram';
  Caution: Increasing temptable_max_ram does not prevent spills entirely; it merely raises the ceiling. Validate available physical host memory, concurrent active connection volume, and replication buffer requirements before increasing globally.

  Review Sort Memory Allocation Carefully:
  The variable sort_buffer_size influences whether sorting completes in a single memory pass or requires multi-pass disk merges:
# Moderate increase for reporting-heavy workloads
sort_buffer_size = 2M .. 4M
  Critical Memory Warning: sort_buffer_size is allocated per session per sort operation. Inflating this setting globally on servers with hundreds of concurrent connections can rapidly trigger out-of-memory (OOM) kernel terminations. Validate changes via session-level overrides or controlled staging benchmarks rather than setting large global values.

  Reduce Intermediate Row Width Through Query Design:
    
      Eliminate SELECT *: Only project columns required by downstream application consumers to minimize the memory footprint per row in the intermediate TempTable buffer.
      Push Predicates Early: Apply WHERE filters before joins, derived tables, or window functions to reduce the volume of rows entering temporary buffers.
      Stage Large Aggregations: Split massive multi-table analytical transformations into bounded, deterministic stages.
    
  

  Covering and Ordering Indexes:
  Creating composite indexes that match query grouping or ordering columns (e.g., (customer_id, transaction_date)) allows the B-tree to return records in pre-sorted order, avoiding both filesorts and temporary tables.
  Important Constraint: Covering indexes cannot prevent temporary table materialization for expressions such as ORDER BY SUM(amount) or complex window aggregations, which inherently require computing intermediate aggregate values.



  Key Operational Takeaway: wait/io/file/sql/FILE_TMPFILE is rarely a storage capacity problem. It is typically evidence that query execution requires more intermediate memory than is currently available or that unoptimized execution plans are materializing excessively wide working sets. The most effective remediation is improving query design, trimming projected row width, tuning TempTable memory thresholds, and adding composite ordering indexes rather than simply provisioning additional IOPS.




9. Deep-Dive: Purge Lag, Undo Growth &amp;amp; History List Length (HLL)

The wait event wait/synch/cond/innodb/purge_worker_cond represents synchronization activity among InnoDB purge worker threads responsible for removing obsolete Multi-Version Concurrency Control (MVCC) row versions from undo logs. By itself, the wait event is not evidence of an engine malfunction. However, when accompanied by a rapidly growing History List Length (HLL), expanding undo tablespaces, and long-running transactions, it indicates that purge processing is falling behind transactional churn.

Official Documentation: MySQL 8.4 Reference Manual: InnoDB Multi-Versioning.

Production Failure Scenario: The Abandoned Read-View Trap

Under MySQL's default REPEATABLE READ isolation level, InnoDB must preserve older row versions for any active transaction that still requires access to its original snapshot view.

A single abandoned or uncommitted transaction will prevent purge processing from reclaiming obsolete undo records across the entire database, regardless of which tables that transaction originally touched:


  


As the History List Length balloons, InnoDB must maintain increasingly large pointer chains of historical row versions. This produces:

  Substantial CPU overhead as queries traverse lengthy undo chains looking for visible versions
  Uncontrolled growth of undo tablespaces on disk
  Significantly prolonged crash-recovery times during restart
  Severe systemic query latency degradation


Diagnostic Commands &amp;amp; Investigation

1. Check Current History List Length (HLL):

SELECT NAME,
       COUNT AS history_list_length
  FROM information_schema.innodb_metrics
 WHERE NAME = 'trx_rseg_history_len';

Operational Guidance:

  &amp;lt; 10,000 — Typically healthy baseline.
  10,000 – 100,000 — Monitor trend, growth rate, and write velocity.
  100,000 – 1,000,000 — Investigate long-running transactions and abandoned read views.
  &amp;gt; 1,000,000 — Critical purge backlog; immediate intervention required.
  Trend Over Absolute Value: There is no universal &quot;safe&quot; absolute threshold. In high-throughput OLTP systems generating tens of thousands of DML statements per second, HLL may briefly rise during batch peaks. A steadily rising slope over hours is far more dangerous than a temporary spike.


2. Identify the Oldest Active Transaction:

SELECT trx_id,
       trx_mysql_thread_id AS connection_id,
       trx_state,
       trx_started,
       TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_seconds,
       trx_query
  FROM information_schema.innodb_trx
 ORDER BY trx_started ASC
 LIMIT 1;

Focus on:

  Idle or sleeping connections (trx_state = 'RUNNING' with trx_query IS NULL in information_schema.processlist marked as Sleep)
  Leaked connection-pool sessions
  Long-running analytical reports
  Abandoned database backup locks


3. Correlate Purge Progress in InnoDB Status:

SHOW ENGINE INNODB STATUS\G

Under the TRANSACTIONS section, inspect:

  History list length &amp;lt;value&amp;gt;
  Purge done for trx's n:o &amp;lt;trx_id&amp;gt; undo n:o &amp;lt;undo_id&amp;gt;
  Number of active read views held open across all connected sessions


Remediation Standards


  Eliminate the Root Blocking Transaction:
  Identify the connection holding the oldest read view. Only after confirming that termination is operationally safe and will not violate application data integrity:
KILL &amp;lt;connection_id&amp;gt;;
  Caution: Never terminate transactions blindly in production. Always validate the application identity, business context, and transaction duration before executing KILL.

  Review Application Transaction Hygiene:
    
      Audit connection pool frameworks (HikariCP, DBCP, c3p0) to ensure connections with uncommitted transactions are rolled back before returning to the pool.
      Verify ORM configurations (e.g., Hibernate, SQLAlchemy, Prisma) to prevent autocommit-disabled connections from lingering in Sleep state.
      Move long-running analytical queries to dedicated read replicas or use READ COMMITTED isolation if historical snapshot consistency is not required.
    
  

  Scale Purge Threads When Appropriate:
  For systems sustaining massive continuous DML write velocity, scale purge concurrency in my.cnf:
[mysqld]
# Scale purge concurrency on multi-core database servers
innodb_purge_threads = 4   # Typical 4-8 on dedicated multi-core hosts
  Crucial Operational Prerequisite: Adjust innodb_purge_threads only after verifying that long-running transactions are not the root cause, and after verifying sufficient host CPU headroom exists.

  Chunk Large DML Operations:
  Avoid unconstrained multi-million-row DELETE or UPDATE operations:
-- DANGEROUS: Floods undo tablespaces and overwhelms purge workers
DELETE FROM audit_log WHERE created_at &amp;lt; NOW() - INTERVAL 2 YEAR;
  Instead, enforce deterministic, bounded chunking executed iteratively:
-- RECOMMENDED: Bounded batch deletion allowing purge to pace smoothly
DELETE FROM audit_log 
 WHERE created_at &amp;lt; NOW() - INTERVAL 2 YEAR 
 LIMIT 10000;
  Introduce brief sleep intervals between chunks if necessary to allow purge workers to keep pace with undo generation.



  Key Operational Takeaway: wait/synch/cond/innodb/purge_worker_cond is not, by itself, evidence of a performance bottleneck. The true indicators of purge lag are a continuously growing History List Length, expanding undo tablespaces, and long-running transactions that prevent the removal of obsolete MVCC row versions. In most production incidents, the root cause is not insufficient purge-worker capacity, but an application session holding an old read view open far longer than intended.




Summary &amp;amp; What's Next

Storage and memory bottlenecks represent the physical layer of database performance. While sizing buffers properly is crucial, improper query design and uncommitted snapshots will overwhelm even the most capable NVMe storage arrays.

In Part 3: Network Transit, Connection Storms, and Replication Pipelines, the series moves to connection management and replication dynamics:


  Thread cache mutex contention during microservice scaling events
  Client socket I/O waits and slow consumer remediation via Keyset Pagination
  Telemetry scrape storms (LOCK_status) and Prometheus exporter optimization
  Multi-Threaded Replication (MTR) worker bottlenecks and missing Primary Keys
  Distinguishing benign idle waits from genuine network delivery lag in replication receivers


Continue to Part 3: Network Transit, Connection Storms, and Replication Pipelines.7TKU66CE8V5W</description>
    <content:encoded><![CDATA[<p>In transactional database systems, the latency gap between memory access (tens to hundreds of nanoseconds) and persistent storage access (tens to hundreds of microseconds for local NVMe devices and milliseconds for remote cloud block storage such as Amazon EBS) spans multiple orders of magnitude. Once a workload exhausts available memory and begins interacting heavily with storage, query response times can deteriorate rapidly.</p>

<p>However, storage-related performance problems in MySQL are rarely solved by simply adding IOPS. In many production incidents, latency originates within the database engine itself:</p>

<ul>
  <li><strong>Redo-log checkpoint advancement falling behind incoming write velocity</strong>, forcing foreground transactions to participate in synchronous flushing activity ("sync flush").</li>
  <li><strong>Doublewrite buffering increasing write amplification</strong>, contributing to storage-pressure events and elevated flush queue depth.</li>
  <li><strong>Binary log durability synchronization (fsync) serializing transaction commits</strong>, creating commit-pipeline bottlenecks under sustained write throughput.</li>
  <li><strong>Buffer pool synchronization primitives and latch contention</strong>, causing worker-thread serialization during periods of intense page churn, eviction pressure, and cache turnover.</li>
  <li><strong>Analytical workloads spilling large temporary datasets and filesort merge passes to disk</strong>, converting microsecond-scale in-memory operations into high-latency storage activity.</li>
  <li><strong>Undo tablespace growth and MVCC purge lag</strong>, driven by long-running transactions, abandoned read views, and purge workers falling behind transactional churn.</li>
</ul>

<p>This article explores the diagnosis and remediation of storage and memory contention patterns in <strong>MySQL 8.4 LTS</strong>, including wait-event analysis, checkpoint behavior, buffer-pool contention, temporary-table spills, purge worker lag, and transaction-commit bottlenecks.</p>

<div>
  <strong>Operational Disclaimer:</strong> <em>The diagnostic queries, Performance Schema instrumentation configurations, engine parameter recommendations, stored routines, and architectural remediations presented across this series are provided for investigative engineering analysis and technical reference. Contention characteristics, wait-event profiles, and optimal parameter thresholds vary significantly depending on transactional access patterns, data volume, schema design, storage subsystem throughput, replication topology, cloud virtualization constraints (e.g., AWS Nitro, EBS bandwidth limits, Aurora multi-tenant architectures), and specific MySQL engine release baselines. Administrative overrides, dynamic parameter modifications, session terminations (KILL), and schema modifications should never be executed blindly against production clusters. All queries, DDL statements, and configuration changes must be reviewed, benchmarked, and validated in representative staging environments through established change-management procedures prior to production implementation.</em>
</div>

<hr>

<h3>1. Storage &amp; I/O Subsystem Bottlenecks: Taxonomy</h3>

<table>
  <thead>
    <tr>
      <th>Wait Event</th>
      <th>Production Root Cause &amp; Impact</th>
      <th>Recommended Engineering Remediation</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>wait/io/file/innodb/innodb_redo_log</code></td>
      <td>Transaction commits stall during sustained write bursts because redo-log generation outpaces checkpoint advancement and flushing activity, forcing foreground transactions to participate in synchronous flushing.</td>
      <td>Batch write operations into larger transactions where appropriate; increase <code>innodb_log_buffer_size</code> (commonly 64M–128M for write-intensive workloads); in MySQL 8.4 LTS, size <code>innodb_redo_log_capacity</code> appropriately to absorb sustained write bursts while maintaining healthy checkpoint-age progression.</td>
    </tr>
    <tr>
      <td><code>wait/io/file/innodb/innodb_doublewrite_file</code></td>
      <td>Doublewrite buffering increases write amplification and may become a storage bottleneck during aggressive dirty-page flushing.</td>
      <td>Self-managed MySQL: tune <code>innodb_doublewrite_files</code> and evaluate storage throughput. AWS RDS for MySQL: consider RDS Optimized Writes where supported. Aurora MySQL: Aurora uses a different storage architecture and does not rely on the traditional InnoDB doublewrite mechanism.</td>
    </tr>
    <tr>
      <td><code>wait/io/file/innodb/innodb_data_file</code></td>
      <td>Synchronous page reads occur when required data is absent from the buffer pool, resulting in storage-access latency and throughput degradation.</td>
      <td>Increase <code>innodb_buffer_pool_size</code> where memory permits; target a sustained buffer-pool hit ratio near 99%+ for OLTP workloads; add appropriate composite and covering indexes to eliminate unnecessary table scans and random I/O.</td>
    </tr>
    <tr>
      <td><code>wait/io/file/sql/binlog</code></td>
      <td>Binary log durability synchronization (fsync) and commit-order preservation limit transaction throughput under heavy write concurrency.</td>
      <td>Do not disable <code>sync_binlog</code> solely for performance reasons in production. First evaluate binlog cache spillover, increase <code>binlog_cache_size</code> where warranted, and verify Binlog Group Commit efficiency before considering durability trade-offs.</td>
    </tr>
  </tbody>
</table>

<hr>

<h3>2. Redo Log Checkpoint Saturation (innodb_redo_log)</h3>

<p>Contention on <code>wait/io/file/innodb/innodb_redo_log</code> occurs when transactional workloads generate redo records faster than InnoDB can advance checkpoints and flush dirty pages. As redo generation outpaces checkpoint progression, foreground transactions increasingly experience delays associated with redo-log pressure and flushing activity.</p>

<p>Official Documentation: <a href="https://dev.mysql.com/doc/refman/8.4/en/innodb-redo-log.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: InnoDB Redo Log</a>.</p>

<h4>Production Failure Scenario: Checkpoint Pressure Under Sustained Write Load</h4>

<p>Under sustained write concurrency, the gap between the current Log Sequence Number (LSN) and the most recent checkpoint grows continuously. This gap, commonly referred to as <strong>Checkpoint Age</strong>, represents the volume of uncheckpointed redo that must be retained for crash recovery.</p>

<p>As Checkpoint Age grows, InnoDB progressively increases background flushing pressure through adaptive flushing mechanisms. If dirty-page generation continues to outpace checkpoint advancement, foreground transactions may begin participating in flushing activity, increasing transaction latency and producing elevated waits on <code>wait/io/file/innodb/innodb_redo_log</code>.</p>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAqqk6szeqMGufusXtxmti4vYxtgS1jAbXsjPxtROzh3OLE6hejF3Ow08OhyHGOEo0rbq7TWwD3yxI15J_AsZ0JJXG6qKyPiA-ZzGjrIegs6nls9AInONLiSZkSpS1r1uIZ5IilpiOwlCeFaZqL2ZT20o4n67PN3z1rHwGdHaaKP5k89e4LCGv8AfrBIY/s1600/image.jpeg" alt="Redo Log Checkpoint Saturation">
</div>

<p>Unlike traditional storage bottlenecks, the underlying issue is often not raw IOPS capacity alone, but the inability of checkpoint advancement and page flushing to keep pace with incoming write velocity.</p>

<h4>Diagnostic Commands &amp; Calculation</h4>

<p><strong>1. Inspect Current LSN and Checkpoint Position:</strong></p>

<pre class="language-sql"><code class="language-sql">SHOW ENGINE INNODB STATUS\G

-- Locate the LOG section:
-- Log sequence number          214748364800
-- Log flushed up to            214748364750
-- Pages flushed up to          214648364800
-- Last checkpoint at           213248364800</code></pre>

<p><strong>2. Calculate Checkpoint Age:</strong></p>

<pre class="language-sql"><code class="language-sql">-- Checkpoint Age = (Log sequence number - Last checkpoint at)
-- 214748364800 - 213248364800 = 1,500,000,000 bytes (~1.4 GB)

-- Compare this value against current redo capacity:
SHOW GLOBAL VARIABLES LIKE 'innodb_redo_log_capacity';</code></pre>

<p>A steadily increasing Checkpoint Age, combined with elevated redo-log waits and aggressive flushing activity, is a strong indicator that checkpoint advancement is struggling to keep pace with redo generation.</p>

<p><strong>3. Monitor Dirty Page Pressure:</strong></p>

<pre class="language-sql"><code class="language-sql">-- Compute dirty-page and data-page metrics in the buffer pool
SELECT VARIABLE_NAME, VARIABLE_VALUE 
  FROM performance_schema.global_status 
 WHERE VARIABLE_NAME IN ('Innodb_buffer_pool_pages_dirty', 'Innodb_buffer_pool_pages_data');</code></pre>

<p>A persistently high dirty-page ratio accompanied by growing redo-log waits indicates checkpoint pressure and insufficient flushing throughput.</p>

<h4>Remediation &amp; Engine Configuration</h4>

<ol>
  <li><strong>Dynamically Expand Redo Capacity (MySQL 8.4 LTS):</strong><br>
  In MySQL 8.4 LTS, <code>innodb_redo_log_capacity</code> is dynamic and can be resized online without requiring a server restart:
<pre class="language-sql"><code class="language-sql">-- Expand redo capacity to better absorb sustained write bursts
SET GLOBAL innodb_redo_log_capacity = 8589934592; -- 8 GB
SHOW GLOBAL VARIABLES LIKE 'innodb_redo_log_capacity';</code></pre>
  Increasing redo capacity provides additional recovery headroom and reduces the likelihood of checkpoint-age pressure during large write bursts.</li>

  <li><strong>Expand the In-Memory Log Buffer:</strong><br>
  For write-intensive workloads, scale <code>innodb_log_buffer_size</code> (typical production ranges: <strong>64 MB–128 MB</strong>). This reduces contention and allocation pressure associated with large transactions and bursty write workloads.</li>

  <li><strong>Verify Storage Throughput and Latency:</strong><br>
  Redo writes ultimately depend on sequential storage durability operations (<code>write()</code> and <code>fsync()</code>). Validate that underlying storage can sustain expected throughput:
    <ul>
      <li>Local NVMe SSDs, AWS EBS <code>io2</code>, or provisioned AWS EBS <code>gp3</code>.</li>
      <li>Monitor write latency, queue depth, flush latency spikes, and storage throughput saturation.</li>
    </ul>
  </li>

  <li><strong>Reduce Redo Generation Where Possible:</strong><br>
  Reducing redo generation is often more cost-effective than continuously scaling storage throughput:
    <ul>
      <li>Batch DML operations into larger deterministic transactions.</li>
      <li>Eliminate unused or redundant secondary indexes that amplify log records.</li>
      <li>Reduce write amplification from frequent row updates.</li>
      <li>Optimize bulk-ingestion and ETL workflows using chunked staging.</li>
    </ul>
  </li>
</ol>

<div>
  <strong>Key Operational Takeaway:</strong> <code>wait/io/file/innodb/innodb_redo_log</code> is rarely the result of a simple disk-performance problem. In most production incidents, the root cause is an imbalance between redo generation, checkpoint advancement, and page-flushing capacity. When write velocity persistently exceeds flushing throughput, InnoDB increases flushing pressure and foreground transaction latency rises. The most effective remedies are usually a combination of appropriately sized redo capacity, healthy checkpoint progression, efficient flushing, and controlled redo generation rates.
</div>

<hr>

<h3>3. Doublewrite Buffer &amp; Hardware Torn-Write Prevention</h3>

<p>The wait event <code>wait/io/file/innodb/innodb_doublewrite_file</code> is associated with dirty-page flushing activity within InnoDB's doublewrite subsystem. Elevated waits typically indicate sustained flushing pressure, significant dirty-page accumulation, or storage throughput limitations during checkpoint advancement and buffer-pool page eviction.</p>

<p>Official Documentation: <a href="https://dev.mysql.com/doc/refman/8.4/en/innodb-doublewrite-buffer.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: InnoDB Doublewrite Buffer</a>.</p>

<h4>Production Mechanics &amp; Write Amplification</h4>

<p>InnoDB stores data in 16 KiB pages, while underlying operating systems and storage devices typically perform physical writes at 4 KiB sector boundaries. If a power loss, kernel crash, or hardware failure occurs mid-write, only part of the page may reach persistent storage, producing a <strong>torn page</strong> and catastrophic tablespace corruption.</p>

<p>To protect against this failure mode, InnoDB first writes dirty pages sequentially into the doublewrite buffer, performs the required durability operations (<code>fsync</code>), and then writes those pages to their final tablespace locations. During crash recovery, InnoDB reads the doublewrite buffer to repair partially written pages before normal recovery proceeds.</p>

<p>Because every flushed dirty page is written twice, this mechanism introduces a <strong>2× write amplification factor</strong>. Under heavy checkpoint pressure or aggressive dirty-page flushing, elevated activity within the doublewrite subsystem becomes a primary bottleneck, visible via <code>wait/io/file/innodb/innodb_doublewrite_file</code>.</p>

<h4>Production Optimization Standards</h4>

<p><strong>1. AWS RDS for MySQL: RDS Optimized Writes</strong></p>

<p>On supported AWS Nitro-based RDS instance classes (such as Graviton <code>db.r6g</code>, <code>db.r7g</code> and Intel/AMD <code>db.r6i</code>, <code>db.m6i</code>), AWS provides <strong>RDS Optimized Writes</strong>:</p>

<pre class="language-bash"><code class="language-bash"># Applied via AWS RDS Parameter Group (requires instance reboot)
rds.optimized_writes = 1</code></pre>

<ul>
  <li><strong>Hardware Mechanism:</strong> Nitro SSD controllers feature hardware-enforced Torn-Write Prevention (TWP), guaranteeing atomic 16 KiB page writes at the hardware controller level.</li>
  <li><strong>Operational Result:</strong> MySQL automatically sets <code>innodb_doublewrite = 0</code>, eliminating doublewrite overhead, halving write I/O amplification, and eliminating <code>wait/io/file/innodb/innodb_doublewrite_file</code> entirely without compromising ACID crash recovery.</li>
  <li><em>Note:</em> Availability depends on engine version, storage type, and instance class. Always verify support in the AWS RDS documentation before enabling.</li>
</ul>

<p><strong>2. Aurora MySQL Architecture Note</strong></p>

<p>Aurora MySQL uses a fundamentally different storage architecture than standard InnoDB deployments:</p>

<ul>
  <li>Instead of relying on traditional InnoDB doublewrite buffers and local data files, Aurora persists redo records directly into a distributed, six-way replicated storage fleet ("the log is the database") and reconstructs data pages in the storage layer.</li>
  <li>Consequently, Aurora does not rely on the traditional InnoDB doublewrite mechanism, RDS Optimized Writes does not apply to Aurora, and <code>wait/io/file/innodb/innodb_doublewrite_file</code> analysis is not relevant in Aurora environments.</li>
</ul>

<p><strong>3. Self-Managed MySQL on Dedicated NVMe Storage</strong></p>

<p>For self-managed cloud or bare-metal servers running MySQL 8.4 LTS, scale doublewrite worker concurrency across dedicated parallel files:</p>

<pre class="language-bash"><code class="language-bash">[mysqld]
# Parallelize doublewrite flushing across multiple files
innodb_doublewrite_files = 4
innodb_doublewrite_pages = 128</code></pre>

<p>Benchmark optimal values based on:</p>
<ul>
  <li>NVMe storage throughput and provisioned IOPS</li>
  <li>Storage queue depth</li>
  <li>Buffer pool size and dirty-page generation velocity</li>
  <li>Checkpoint workload characteristics</li>
</ul>

<h4>Diagnostic Indicators</h4>

<p>Investigate the doublewrite subsystem when you observe <code>wait/io/file/innodb/innodb_doublewrite_file</code> together with:</p>

<ul>
  <li>Elevated dirty-page counts (<code>Innodb_buffer_pool_pages_dirty</code>)</li>
  <li>Aggressive checkpoint activity and rising Checkpoint Age</li>
  <li>Increased page-flush rates (<code>Innodb_buffer_pool_pages_flushed</code>)</li>
  <li>Storage write throughput approaching saturation or high queue depths</li>
  <li>Sustained buffer-pool eviction pressure</li>
</ul>

<div>
  <strong>Key Operational Takeaway:</strong> <code>wait/io/file/innodb/innodb_doublewrite_file</code> is rarely the primary bottleneck. In most production incidents, it is a downstream symptom reflecting excessive page-flushing pressure driven by checkpoint advancement, dirty-page accumulation, or storage-layer saturation. The correct remediation is usually to address the underlying flushing workload and checkpoint progression rather than focusing exclusively on the doublewrite subsystem itself.
</div>

<hr>

<h3>4. Data File I/O &amp; Working-Set Cache Misses</h3>

<p>The wait event <code>wait/io/file/innodb/innodb_data_file</code> is commonly associated with synchronous page reads that occur when required data pages are not present in the InnoDB buffer pool. In these situations, foreground threads must wait while the storage subsystem retrieves the necessary 16 KiB pages from disk before query execution can proceed.</p>

<p>Official Documentation:</p>
<ul>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/innodb-buffer-pool.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: InnoDB Buffer Pool</a></li>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/performance-schema-file-summary-tables.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Performance Schema File I/O Instrumentation</a></li>
</ul>

<h4>Production Failure Scenario: Working-Set Exceeds Available Memory</h4>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbh7AgEXrBawn-cciqVWEe8_qv5-TAy6l7a3csvVZp7xQKu987i0zN9n0bFTrqai1JLetyhE9qM_3tUrQPgiyqNotvEj5A87_jQ7Ruh-WkoVzh2EJDi_cRKzwrSSxn5HWL0N0zt4RJroGOiF25i86cySz__DyPWMliRLTyRAtli83_GCTIlFRzs4ne1Dg/s1600/image.jpeg" alt="Working Set Exceeds Memory">
</div>

<p>Under sustained pressure, the entire workload can become storage-bound and IOPS-constrained even though database CPU utilization remains deceptively low.</p>

<h4>Diagnostic Commands &amp; Hit-Ratio Calculation</h4>

<p><strong>1. Compute Global InnoDB Buffer Pool Hit Ratio:</strong></p>

<pre class="language-sql"><code class="language-sql">SELECT ROUND(
         100 * (1 - (bp_reads.variable_value / bp_requests.variable_value)),
         4
       ) AS hit_ratio_pct
  FROM performance_schema.global_status bp_reads
  JOIN performance_schema.global_status bp_requests
    ON bp_reads.variable_name = 'Innodb_buffer_pool_reads'
   AND bp_requests.variable_name = 'Innodb_buffer_pool_read_requests';</code></pre>

<p><strong>Operational Hit Ratio Standards:</strong></p>
<ul>
  <li><span>&gt; 99%</span> — Standard target for high-concurrency OLTP workloads.</li>
  <li><span>95% – 99%</span> — Well-cached, optimal production workload.</li>
  <li><span>&lt; 95%</span> — Investigate working-set sizing, cold cache sweeps, or unindexed queries.</li>
  <li><em>Contextual Interpretation:</em> The hit ratio should always be evaluated alongside storage latency and access patterns. Lower hit ratios are typical and expected for reporting, DSS, analytical range scans, or batch ETL workloads where scanning full tables is required.</li>
</ul>

<p><strong>2. Identify Queries Driving Physical Disk Reads:</strong></p>

<pre class="language-sql"><code class="language-sql">SELECT DIGEST_TEXT,
       COUNT_STAR AS exec_count,
       SUM_NO_INDEX_USED AS scans_without_index,
       SUM_ROWS_EXAMINED,
       ROUND(SUM_TIMER_WAIT / 1e12, 2) AS total_latency_sec
  FROM performance_schema.events_statements_summary_by_digest
 WHERE SUM_NO_INDEX_USED &gt; 0
 ORDER BY SUM_ROWS_EXAMINED DESC
 LIMIT 5;</code></pre>

<p>Look for:</p>
<ul>
  <li>Full table scans cycling cold pages through the LRU list</li>
  <li>Wide range scans examining large fractions of large tables</li>
  <li>Missing composite indexes causing secondary index to primary table lookups</li>
  <li>Disproportionate rows examined versus rows returned</li>
</ul>

<h4>Remediation Standards</h4>

<ol>
  <li><strong>Size the Buffer Pool to the Active Working Set:</strong><br>
  Configure <code>innodb_buffer_pool_size</code> so the primary transactional dataset remains resident in memory. On dedicated database hosts, a standard guideline is <strong>70–80% of available physical RAM</strong>, balanced against:
    <ul>
      <li>Thread stack and connection memory (<code>max_connections × thread buffers</code>)</li>
      <li>Internal temporary table memory limits (<code>temptable_max_ram</code>)</li>
      <li>Replication worker buffers</li>
      <li>Host operating system and kernel page cache requirements</li>
      <li>Local backup utilities and monitoring agents</li>
    </ul>
  </li>

  <li><strong>Eliminate Full Scans Using Composite Covering Indexes:</strong><br>
  <em>Problem Query:</em>
<pre class="language-sql"><code class="language-sql">SELECT order_id, total 
  FROM customer_orders 
 WHERE customer_id = 918 AND status = 'SHIPPED';</code></pre>
  Without an optimal index, MySQL performs table or range scans touching millions of disk pages.<br><br>
  <em>Remediation:</em>
<pre class="language-sql"><code class="language-sql">ALTER TABLE customer_orders 
  ADD INDEX idx_cust_status_covered (customer_id, status, order_id, total);</code></pre>
  A composite covering index satisfies both filtering and projection directly out of the index B-tree, eliminating physical clustered index page reads.</li>

  <li><strong>Validate Buffer Pool Eviction Pressure:</strong><br>
  Inspect page turnover and eviction velocity in InnoDB status:
<pre class="language-sql"><code class="language-sql">SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads%';
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read_requests%';
SHOW ENGINE INNODB STATUS\G</code></pre>
  Under the <strong>BUFFER POOL AND MEMORY</strong> section, inspect:
    <ul>
      <li><strong>Pages read per second vs. Pages created per second</strong></li>
      <li><strong>Evicted without access rate</strong> (high values indicate LRU thrashing)</li>
      <li><strong>Read-ahead efficiency</strong> (low hit rates indicate unhelpful prefetching)</li>
    </ul>
  </li>
</ol>

<div>
  <strong>Key Operational Takeaway:</strong> <code>wait/io/file/innodb/innodb_data_file</code> is usually not a storage problem first; it is primarily a memory-efficiency and indexing problem. In most production incidents, elevated data-file waits indicate that the active working set no longer fits within the buffer pool, forcing foreground threads to perform synchronous disk reads. The most effective remediation is a combination of proper buffer-pool sizing, covering indexes, and query refactoring rather than simply provisioning additional IOPS.
</div>

<hr>

<h3>5. Binlog Wait Diagnostic Workflow</h3>

<p>When <code>wait/io/file/sql/binlog</code> dominates the wait profile, setting <code>sync_binlog = 0</code> is often treated as a quick operational shortcut. In reality, this introduces severe durability and consistency risks by delegating <code>fsync()</code> timing entirely to the operating system page cache.</p>

<p>If an operating system crash, kernel panic, or power failure occurs before pending binary log writes are flushed to disk, transactions acknowledged as committed can be permanently lost. Furthermore, replicas may never receive those transactions—or may have already received transactions that are subsequently lost on the source after reboot—producing severe <strong>replication divergence</strong> (split-brain) and complicating disaster recovery.</p>

<p>Before modifying durability settings, engineers should execute this three-step diagnostic triage workflow.</p>

<p>Official Documentation: <a href="https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html#sysvar_binlog_group_commit_sync_delay" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Binary Log Group Commit</a>.</p>

<h4>Step 1: Check Binlog Cache Spillage</h4>

<pre class="language-sql"><code class="language-sql">SHOW GLOBAL STATUS LIKE 'Binlog_cache%';</code></pre>

<table>
  <thead>
    <tr>
      <th>Status Variable</th>
      <th>Production Meaning</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>Binlog_cache_use</code></td>
      <td>Total number of transactions that utilized the thread-local binlog cache.</td>
    </tr>
    <tr>
      <td><code>Binlog_cache_disk_use</code></td>
      <td>Transactions that exceeded <code>binlog_cache_size</code> and spilled into temporary disk files.</td>
    </tr>
  </tbody>
</table>

<p>Calculate the <strong>Disk Spillage Ratio</strong>:</p>

<div>
  Disk Spillage Ratio (%) = [Binlog_cache_disk_use / (Binlog_cache_use + Binlog_cache_disk_use)] × 100
</div>

<p>A persistently elevated spillage ratio indicates that large transactions are exceeding available thread memory, generating temporary-file disk I/O <em>before</em> the binary-log synchronization phase begins.</p>

<p><strong>Operational Guideline:</strong> Investigate spillage ratios exceeding <strong>1–5%</strong>, particularly when accompanied by elevated transactional commit latency or high waits on <code>wait/io/file/sql/binlog</code>.</p>

<h4>Step 2: Evaluate Cache Sizing vs. Concurrency</h4>

<pre class="language-sql"><code class="language-sql">SHOW GLOBAL VARIABLES LIKE 'binlog_cache_size';
SHOW GLOBAL VARIABLES LIKE 'max_binlog_cache_size';</code></pre>

<p>If <code>Binlog_cache_disk_use</code> is increasing rapidly while <code>binlog_cache_size</code> remains at the 32 KB default, consider scaling:</p>

<pre class="language-bash"><code class="language-bash"># Scale thread-local binlog cache to reduce disk spillage
binlog_cache_size = 256K .. 1M</code></pre>

<p>Always validate available host memory first, as this buffer is allocated <strong>per active transactional thread</strong>:</p>

<div>
  Max Potential Cache Memory = max_connections × binlog_cache_size
</div>

<p>Simultaneously audit application workloads for large, unbatched operations:</p>
<ul>
  <li>Multi-million-row <code>UPDATE</code> or <code>DELETE</code> statements</li>
  <li>Bulk ETL and data ingestion jobs</li>
  <li>Unbounded transactional loops</li>
</ul>

<p>Chunking large transactions into deterministic batches (e.g., 1,000–5,000 rows) is far more effective and memory-safe than continuously inflating thread cache allocations.</p>

<h4>Step 3: Optimize Binlog Group Commit (Maintain Full Durability)</h4>

<pre class="language-sql"><code class="language-sql">SHOW GLOBAL VARIABLES LIKE 'binlog_group_commit_sync_delay';
SHOW GLOBAL VARIABLES LIKE 'binlog_group_commit_sync_no_delay_count';</code></pre>

<p>Rather than compromising crash safety, leverage <strong>Binary Log Group Commit</strong> to batch concurrent transactional <code>fsync</code> operations into a single disk write:</p>

<pre class="language-bash"><code class="language-bash">[mysqld]
# Full ACID durability baseline
sync_binlog                             = 1
innodb_flush_log_at_trx_commit          = 1

# Binary Log Group Commit batching
binlog_group_commit_sync_delay          = 1000   # Pause up to 1000 microseconds (1 ms)
binlog_group_commit_sync_no_delay_count = 100    # Flush immediately if 100 transactions accumulate</code></pre>

<p><strong>Production Behavior:</strong></p>
<ul>
  <li><strong><code>binlog_group_commit_sync_delay = 1000</code> (1 ms):</strong> The commit-group leader thread pauses for up to 1 millisecond before issuing <code>fsync()</code>, allowing concurrent transactions to join the same commit group.</li>
  <li><strong><code>binlog_group_commit_sync_no_delay_count = 100</code>:</strong> If 100 transactions accumulate before the 1 ms timer expires, the group flushes immediately to prevent latency degradation.</li>
</ul>

<p>This configuration achieves write throughput comparable to <code>sync_binlog = 0</code> under high-concurrency OLTP workloads while preserving 100% crash durability and preventing replication drift.</p>

<div>
  <strong>Key Operational Takeaway:</strong> <code>wait/io/file/sql/binlog</code> does not justify disabling durability settings. In production, the correct remediation sequence is to eliminate binlog-cache disk spillage, right-size <code>binlog_cache_size</code>, enforce deterministic DML chunking, and tune Binary Log Group Commit. Durability baselines (<code>sync_binlog = 1</code> and <code>innodb_flush_log_at_trx_commit = 1</code>) should remain non-negotiable standards wherever transactional integrity and replication consistency are business requirements.
</div>

<hr>

<h3>6. Memory Contention &amp; Engine Overhead</h3>

<table>
  <thead>
    <tr>
      <th>Wait Event</th>
      <th>Production Root Cause &amp; Impact</th>
      <th>Recommended Engineering Remediation</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>wait/synch/mutex/innodb/buf_pool_mutex</code></td>
      <td>Concurrent sessions contend for synchronization protecting InnoDB buffer-pool structures while loading, flushing, evicting, or modifying cached data pages. Elevated waits are commonly associated with buffer-pool pressure, excessive page churn, storage misses, or cache-efficiency problems rather than mutex overhead alone.</td>
      <td>Verify <code>innodb_buffer_pool_size</code> is sufficient for the active working set; investigate page-read, dirty-page, and eviction rates; eliminate unnecessary scans; validate storage latency; and reduce buffer-pool churn. MySQL 8.4 automatically manages buffer-pool parallelism and internal partitioning, significantly reducing historical buffer-pool mutex contention.</td>
    </tr>
    <tr>
      <td><code>wait/io/file/sql/FILE_TMPFILE</code></td>
      <td>Internal temporary tables, filesort merge passes, hash aggregation structures, or intermediate result sets exceed memory thresholds and spill to disk, converting memory operations into storage I/O.</td>
      <td>Tune <code>temptable_max_ram</code>; review <code>tmp_table_size</code> and <code>max_heap_table_size</code> where applicable; optimize <code>GROUP BY</code>, <code>ORDER BY</code>, and large joins; increase <code>sort_buffer_size</code> cautiously after validating per-session memory impact; and add covering indexes that allow filtering, grouping, and ordering directly from index structures.</td>
    </tr>
    <tr>
      <td><code>wait/synch/cond/innodb/purge_worker_cond</code></td>
      <td>Purge workers are coordinating, waiting for work, or processing undo cleanup tasks. Elevated History List Length (HLL), growing undo tablespaces, and long-running transactions together indicate that purge activity is falling behind transactional churn.</td>
      <td>Monitor HLL through <code>INNODB_METRICS</code>; identify long-running transactions and stale read views using <code>information_schema.innodb_trx</code>; investigate replication, backup, or analytical workloads holding snapshots open; and consider increasing <code>innodb_purge_threads</code> when purge consistently lags behind workload demand.</td>
    </tr>
  </tbody>
</table>

<div>
  <strong>Operational Analysis Standard:</strong> Wait events should never be analyzed in isolation. For memory-related waits, always correlate Performance Schema telemetry with buffer-pool statistics, temporary-table metrics, undo history growth, storage latency, and workload characteristics before identifying a root cause. A wait event often reflects downstream symptoms rather than the originating bottleneck.
</div>

<hr>

<h3>7. Deep-Dive: Buffer Pool Contention, Page Churn &amp; Cache Pressure (buf_pool_*)</h3>

<p>The wait family <code>wait/synch/mutex/innodb/buf_pool_*</code> represents contention within InnoDB buffer-pool management structures responsible for tracking, locating, loading, evicting, and flushing data pages. In modern MySQL 8.4 LTS environments, these waits are typically downstream symptoms of excessive page churn, cache inefficiency, or storage pressure rather than a standalone mutex bottleneck.</p>

<p>Official Documentation:</p>
<ul>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/innodb-buffer-pool.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: InnoDB Buffer Pool</a></li>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/innodb-buffer-pool-resize.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Buffer Pool Sizing &amp; Resize</a></li>
</ul>

<h4>Production Failure Scenario: Working Set Exceeds Memory</h4>

<p>When the active transactional working set becomes larger than available buffer pool capacity, pages are evicted faster than they can be reused. Simultaneously, unindexed analytical scans, poorly selective range filters, or elevated storage latency create a continuous cycle of cold-page loading and hot-page eviction.</p>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWZZoLFsEoza0PnVReS474pzTaNofuoOV12bi4Hi2PP9JoOroT_I18h_1eVz7v7rTDxOGZf0j7uBkIVBB5BOxd8FIx-IXT1CjDbfwnnrVQGwQ_ttwlFrt01AeEMQU5aFDoFouyXih69Llc6UhT54xDQVcL4yiYcFKxY4u7ujtAEhkuOAr2i4MXOGMIf0M/s1600/image.jpeg" alt="Buffer Pool Contention and LRU Thrashing">
</div>

<p><strong>Common Production Symptoms:</strong></p>
<ul>
  <li>Elevated waits on <code>wait/synch/mutex/innodb/buf_pool_*</code></li>
  <li>Steep increase in physical synchronous disk reads (<code>Innodb_buffer_pool_reads</code>)</li>
  <li>Plummeting buffer pool cache efficiency (&lt; 95%)</li>
  <li>Increasing disk queue depth and storage latency</li>
  <li>Aggressive page-flush velocity and free list exhaustion</li>
  <li>Near-100% CPU utilization driven by mutex spin-locks despite falling transactional throughput</li>
</ul>

<h4>Diagnostic Commands &amp; Investigation</h4>

<p><strong>1. Identify Buffer Pool Related Waits:</strong></p>

<pre class="language-sql"><code class="language-sql">SELECT EVENT_NAME,
       COUNT_STAR AS total_waits,
       ROUND(SUM_TIMER_WAIT / 1e12, 2) AS total_wait_sec,
       ROUND(AVG_TIMER_WAIT / 1e6, 2) AS avg_wait_us
  FROM performance_schema.events_waits_summary_global_by_event_name
 WHERE EVENT_NAME LIKE 'wait/synch/%/innodb/buf_pool%'
 ORDER BY SUM_TIMER_WAIT DESC;</code></pre>

<p><em>Look for:</em> Rapidly increasing wait counts, high cumulative wait duration, and lock thrashing across Page Hash Table latches and LRU list mutexes.</p>

<p><strong>2. Inspect Buffer Pool Churn in InnoDB Status:</strong></p>

<pre class="language-sql"><code class="language-sql">SHOW ENGINE INNODB STATUS\G</code></pre>

<p>Under the <strong>BUFFER POOL AND MEMORY</strong> section, inspect:</p>
<ul>
  <li><strong>Pages read/sec, created/sec, written/sec:</strong> Gauges active page turnover rate.</li>
  <li><strong>Evicted without access per second:</strong> A high rate is a definitive indicator of cold cache sweeps thrashing the LRU list.</li>
  <li><strong>Free buffers:</strong> Persistently low or zero values force foreground worker threads to perform synchronous on-demand page evictions.</li>
  <li><strong>Pending reads / Pending normal aio writes:</strong> Indicates whether storage queue depths are saturated.</li>
</ul>

<p><strong>3. Calculate Buffer Pool Hit Ratio:</strong></p>

<pre class="language-sql"><code class="language-sql">SELECT ROUND(
         100 * (1 - (bp_reads.variable_value / bp_requests.variable_value)),
         4
       ) AS hit_ratio_pct
  FROM performance_schema.global_status bp_reads
  JOIN performance_schema.global_status bp_requests
    ON bp_reads.variable_name = 'Innodb_buffer_pool_reads'
   AND bp_requests.variable_name = 'Innodb_buffer_pool_read_requests';</code></pre>

<ul>
  <li><span>&gt; 99%</span> — Optimal OLTP cache efficiency.</li>
  <li><span>95% – 99%</span> — Acceptable production baseline.</li>
  <li><span>&lt; 95%</span> — Investigate working-set sizing, cold table scans, and memory allocation.</li>
</ul>

<h4>Remediation Standards</h4>

<ol>
  <li><strong>Size the Buffer Pool to the Active Working Set:</strong><br>
  Configure <code>innodb_buffer_pool_size</code> so the primary hot dataset remains 100% memory-resident. For dedicated MySQL instances, allocate <strong>70–80% of physical RAM</strong>, after deducting:
    <ul>
      <li>Thread memory allocations (<code>max_connections × session buffers</code>)</li>
      <li>Internal temporary table buffers (<code>temptable_max_ram</code>)</li>
      <li>Replication worker queues</li>
      <li>Host operating system and filesystem page cache requirements</li>
      <li>Local backup utilities and telemetry daemons</li>
    </ul>
  </li>

  <li><strong>Eliminate Unnecessary Page Churn:</strong><br>
  Audit and optimize queries driving cold-page sweeps through the buffer pool:
    <ul>
      <li>Full table scans on multi-gigabyte historical tables</li>
      <li>Inefficient join plans cycling outer rows</li>
      <li>Missing composite indexes causing unindexed range scans</li>
      <li>Low-selectivity index predicates forcing widespread index-to-clustered-table lookups</li>
    </ul>
  </li>

  <li><strong>Buffer Pool Instances in MySQL 8.4 LTS:</strong><br>
  In MySQL 8.4 LTS, <code>innodb_buffer_pool_instances</code> is automatically calculated by InnoDB based on logical CPU count (up to 64 instances when <code>innodb_buffer_pool_size &gt;= 1GB</code>). Manual instance tuning is obsolete; focus engineering efforts on working-set sizing, index optimization, and storage latency reduction.</li>
</ol>

<div>
  <strong>Key Operational Takeaway:</strong> Buffer-pool waits in MySQL 8.4 are usually downstream symptoms rather than root causes. When <code>wait/synch/mutex/innodb/buf_pool_*</code> dominates Performance Schema profiles, the underlying pathology is typically excessive page churn, insufficient memory residency, unindexed access patterns, or storage bottlenecks. The most effective remediation is improving cache efficiency and eliminating unnecessary physical I/O rather than attempting to tune mutex synchronization directly.
</div>

<hr>

<h3>8. Deep-Dive: On-Disk Temp Table Spills (FILE_TMPFILE)</h3>

<p>The wait event <code>wait/io/file/sql/FILE_TMPFILE</code> occurs when MySQL materializes internal temporary tables, filesort merge runs, hash aggregation structures, window-function state, or intermediate result sets onto storage because in-memory thresholds have been exceeded.</p>

<p>Official Documentation: <a href="https://dev.mysql.com/doc/refman/8.4/en/internal-temporary-tables.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Internal Temporary Tables</a>.</p>

<h4>Production Failure Scenario: Large Grouping, Sorting &amp; Intermediate Result Materialization</h4>

<p>Complex relational queries involving:</p>
<ul>
  <li><code>GROUP BY</code> and <code>DISTINCT</code> aggregations</li>
  <li><code>ORDER BY</code> sorting operations</li>
  <li>Window Functions (<code>OVER (...)</code>)</li>
  <li>Large multi-table hash joins</li>
  <li>Derived tables and Common Table Expressions (CTEs)</li>
</ul>

<p>frequently require MySQL to construct intermediate working sets. When those working sets exceed available in-memory thresholds, MySQL spills them to disk:</p>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjTD4FF44yI-0JCyj_yXKDUSm9U0Vd7Kyzi2c80P1jZjAuoxorB5aZM2h4UaYlN0VjIrL5DEytei1_W4VMeWg6vAjSJzw_H8xu39ELiSIUnZuKJxzUh5ksZOdMkT2lKwBR-E29IZ9_43YcofRY6tZziVdiZmyorZOyPt-CbQFGxGB7Yuu8MefI7RCW_4s/s1600/image.jpeg" alt="Internal Temporary Table Spills to Disk">
</div>

<p>A workload that normally completes using microsecond-scale memory operations can rapidly become storage-bound and IOPS-constrained once repeated temp-table and filesort spills begin occurring.</p>

<h4>Representative Execution Plan Pattern</h4>

<pre class="language-sql"><code class="language-sql">EXPLAIN ANALYZE
SELECT customer_id,
       memo,
       SUM(amount)
  FROM transaction_ledger
 GROUP BY customer_id, memo
 ORDER BY SUM(amount) DESC;</code></pre>

<p>Execution plans displaying <code>Using temporary; Using filesort</code> do not automatically indicate a production emergency; however, they frequently correlate with severe <code>FILE_TMPFILE</code> waits when the intermediate row count or projected byte width exceeds in-memory buffers.</p>

<h4>Diagnostic Commands &amp; Spill Ratio Analysis</h4>

<p><strong>1. Monitor Temporary-Table Spill Rates:</strong></p>

<pre class="language-sql"><code class="language-sql">SHOW GLOBAL STATUS LIKE 'Created_tmp%';</code></pre>

<table>
  <thead>
    <tr>
      <th>Status Variable</th>
      <th>Production Meaning</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>Created_tmp_tables</code></td>
      <td>Total internal temporary tables created by MySQL sessions.</td>
    </tr>
    <tr>
      <td><code>Created_tmp_disk_tables</code></td>
      <td>Internal temporary tables that exceeded in-memory thresholds and spilled to disk.</td>
    </tr>
  </tbody>
</table>

<p>Calculate the <strong>Disk Spill Ratio</strong>:</p>

<div>
  Disk Spill Ratio (%) = (Created_tmp_disk_tables / Created_tmp_tables) × 100
</div>

<p><strong>Operational Guidance:</strong></p>
<ul>
  <li><span>&lt; 5%</span> — Excellent in-memory caching efficiency.</li>
  <li><span>5% – 20%</span> — Acceptable range; worth periodic monitoring during peak hours.</li>
  <li><span>&gt; 20%</span> — Investigate query design, result-set row widths, and memory allocations.</li>
  <li><em>Contextual Reality:</em> A high spill ratio is not inherently a configuration bug for dedicated analytics, reporting, batch ETL, or data warehouse replicas, where scanning multi-gigabyte datasets legitimately requires storage-backed materialization.</li>
</ul>

<p><strong>2. Identify Filesort and Spill Contributors:</strong></p>

<pre class="language-sql"><code class="language-sql">SELECT DIGEST_TEXT,
       COUNT_STAR AS exec_count,
       SUM_SORT_MERGE_PASSES,
       SUM_SORT_ROWS,
       ROUND(SUM_TIMER_WAIT / 1e12, 2) AS total_wait_sec
  FROM performance_schema.events_statements_summary_by_digest
 WHERE SUM_SORT_MERGE_PASSES &gt; 0
 ORDER BY SUM_SORT_MERGE_PASSES DESC
 LIMIT 5;</code></pre>

<p><em>Look for:</em> Repeated multi-pass sort merges (<code>SUM_SORT_MERGE_PASSES &gt; 0</code>), massive row sort volumes, and expensive analytical statements. Multiple merge passes indicate that intermediate sort chunks could not fit within <code>sort_buffer_size</code>, requiring external merge sort runs on disk.</p>

<h4>Remediation &amp; Engine Configuration</h4>

<ol>
  <li><strong>Tune the TempTable Engine Memory Budget (MySQL 8.4 LTS):</strong><br>
  In MySQL 8.4 LTS, <code>temptable_max_ram</code> governs how much memory the TempTable engine can consume before materializing temporary tables to disk:
<pre class="language-sql"><code class="language-sql">-- Dynamically expand TempTable in-memory limit
SET GLOBAL temptable_max_ram = 1073741824; -- 1 GiB
SHOW GLOBAL VARIABLES LIKE 'temptable_max_ram';</code></pre>
  <em>Caution:</em> Increasing <code>temptable_max_ram</code> does not prevent spills entirely; it merely raises the ceiling. Validate available physical host memory, concurrent active connection volume, and replication buffer requirements before increasing globally.</li>

  <li><strong>Review Sort Memory Allocation Carefully:</strong><br>
  The variable <code>sort_buffer_size</code> influences whether sorting completes in a single memory pass or requires multi-pass disk merges:
<pre class="language-bash"><code class="language-bash"># Moderate increase for reporting-heavy workloads
sort_buffer_size = 2M .. 4M</code></pre>
  <em>Critical Memory Warning:</em> <code>sort_buffer_size</code> is allocated <strong>per session per sort operation</strong>. Inflating this setting globally on servers with hundreds of concurrent connections can rapidly trigger out-of-memory (OOM) kernel terminations. Validate changes via session-level overrides or controlled staging benchmarks rather than setting large global values.</li>

  <li><strong>Reduce Intermediate Row Width Through Query Design:</strong>
    <ul>
      <li><strong>Eliminate <code>SELECT *</code>:</strong> Only project columns required by downstream application consumers to minimize the memory footprint per row in the intermediate TempTable buffer.</li>
      <li><strong>Push Predicates Early:</strong> Apply <code>WHERE</code> filters before joins, derived tables, or window functions to reduce the volume of rows entering temporary buffers.</li>
      <li><strong>Stage Large Aggregations:</strong> Split massive multi-table analytical transformations into bounded, deterministic stages.</li>
    </ul>
  </li>

  <li><strong>Covering and Ordering Indexes:</strong><br>
  Creating composite indexes that match query grouping or ordering columns (e.g., <code>(customer_id, transaction_date)</code>) allows the B-tree to return records in pre-sorted order, avoiding both filesorts and temporary tables.<br>
  <em>Important Constraint:</em> Covering indexes cannot prevent temporary table materialization for expressions such as <code>ORDER BY SUM(amount)</code> or complex window aggregations, which inherently require computing intermediate aggregate values.</li>
</ol>

<div>
  <strong>Key Operational Takeaway:</strong> <code>wait/io/file/sql/FILE_TMPFILE</code> is rarely a storage capacity problem. It is typically evidence that query execution requires more intermediate memory than is currently available or that unoptimized execution plans are materializing excessively wide working sets. The most effective remediation is improving query design, trimming projected row width, tuning TempTable memory thresholds, and adding composite ordering indexes rather than simply provisioning additional IOPS.
</div>

<hr>

<h3>9. Deep-Dive: Purge Lag, Undo Growth &amp; History List Length (HLL)</h3>

<p>The wait event <code>wait/synch/cond/innodb/purge_worker_cond</code> represents synchronization activity among InnoDB purge worker threads responsible for removing obsolete Multi-Version Concurrency Control (MVCC) row versions from undo logs. By itself, the wait event is not evidence of an engine malfunction. However, when accompanied by a rapidly growing History List Length (HLL), expanding undo tablespaces, and long-running transactions, it indicates that purge processing is falling behind transactional churn.</p>

<p>Official Documentation: <a href="https://dev.mysql.com/doc/refman/8.4/en/innodb-multi-versioning.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: InnoDB Multi-Versioning</a>.</p>

<h4>Production Failure Scenario: The Abandoned Read-View Trap</h4>

<p>Under MySQL's default <code>REPEATABLE READ</code> isolation level, InnoDB must preserve older row versions for any active transaction that still requires access to its original snapshot view.</p>

<p>A single abandoned or uncommitted transaction will prevent purge processing from reclaiming obsolete undo records across the <strong>entire database</strong>, regardless of which tables that transaction originally touched:</p>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeFINaKOUdSFiujhzbvSuHNl9hTkjnHRKH_SnMIPn1EmhD_Yh-1lFwXcwhDH5-G86Qab1Fd5RRA57t61C4iXnSRJvw_nhAIOZMljVmguYz-IQ7p9j90vEs6piHxMqPAvYqsSGxRZ7BMXLCTrEANPPTvSl-RYGcp3_aKzQYtskdXIub8NcgOA-yzv84VnY/s1600/image.jpeg" alt="Abandoned Read View and History List Length Growth">
</div>

<p>As the History List Length balloons, InnoDB must maintain increasingly large pointer chains of historical row versions. This produces:</p>
<ul>
  <li>Substantial CPU overhead as queries traverse lengthy undo chains looking for visible versions</li>
  <li>Uncontrolled growth of undo tablespaces on disk</li>
  <li>Significantly prolonged crash-recovery times during restart</li>
  <li>Severe systemic query latency degradation</li>
</ul>

<h4>Diagnostic Commands &amp; Investigation</h4>

<p><strong>1. Check Current History List Length (HLL):</strong></p>

<pre class="language-sql"><code class="language-sql">SELECT NAME,
       COUNT AS history_list_length
  FROM information_schema.innodb_metrics
 WHERE NAME = 'trx_rseg_history_len';</code></pre>

<p><strong>Operational Guidance:</strong></p>
<ul>
  <li><span>&lt; 10,000</span> — Typically healthy baseline.</li>
  <li><span>10,000 – 100,000</span> — Monitor trend, growth rate, and write velocity.</li>
  <li><span>100,000 – 1,000,000</span> — Investigate long-running transactions and abandoned read views.</li>
  <li><span>&gt; 1,000,000</span> — Critical purge backlog; immediate intervention required.</li>
  <li><em>Trend Over Absolute Value:</em> There is no universal "safe" absolute threshold. In high-throughput OLTP systems generating tens of thousands of DML statements per second, HLL may briefly rise during batch peaks. A steadily rising slope over hours is far more dangerous than a temporary spike.</li>
</ul>

<p><strong>2. Identify the Oldest Active Transaction:</strong></p>

<pre class="language-sql"><code class="language-sql">SELECT trx_id,
       trx_mysql_thread_id AS connection_id,
       trx_state,
       trx_started,
       TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_seconds,
       trx_query
  FROM information_schema.innodb_trx
 ORDER BY trx_started ASC
 LIMIT 1;</code></pre>

<p><em>Focus on:</em></p>
<ul>
  <li>Idle or sleeping connections (<code>trx_state = 'RUNNING'</code> with <code>trx_query IS NULL</code> in <code>information_schema.processlist</code> marked as <code>Sleep</code>)</li>
  <li>Leaked connection-pool sessions</li>
  <li>Long-running analytical reports</li>
  <li>Abandoned database backup locks</li>
</ul>

<p><strong>3. Correlate Purge Progress in InnoDB Status:</strong></p>

<pre class="language-sql"><code class="language-sql">SHOW ENGINE INNODB STATUS\G</code></pre>

<p>Under the <strong>TRANSACTIONS</strong> section, inspect:</p>
<ul>
  <li><code>History list length &lt;value&gt;</code></li>
  <li><code>Purge done for trx's n:o &lt;trx_id&gt; undo n:o &lt;undo_id&gt;</code></li>
  <li>Number of active read views held open across all connected sessions</li>
</ul>

<h4>Remediation Standards</h4>

<ol>
  <li><strong>Eliminate the Root Blocking Transaction:</strong><br>
  Identify the connection holding the oldest read view. Only after confirming that termination is operationally safe and will not violate application data integrity:
<pre class="language-sql"><code class="language-sql">KILL &lt;connection_id&gt;;</code></pre>
  <em>Caution:</em> Never terminate transactions blindly in production. Always validate the application identity, business context, and transaction duration before executing <code>KILL</code>.</li>

  <li><strong>Review Application Transaction Hygiene:</strong>
    <ul>
      <li>Audit connection pool frameworks (HikariCP, DBCP, c3p0) to ensure connections with uncommitted transactions are rolled back before returning to the pool.</li>
      <li>Verify ORM configurations (e.g., Hibernate, SQLAlchemy, Prisma) to prevent autocommit-disabled connections from lingering in <code>Sleep</code> state.</li>
      <li>Move long-running analytical queries to dedicated read replicas or use <code>READ COMMITTED</code> isolation if historical snapshot consistency is not required.</li>
    </ul>
  </li>

  <li><strong>Scale Purge Threads When Appropriate:</strong><br>
  For systems sustaining massive continuous DML write velocity, scale purge concurrency in <code>my.cnf</code>:
<pre class="language-bash"><code class="language-bash">[mysqld]
# Scale purge concurrency on multi-core database servers
innodb_purge_threads = 4   # Typical 4-8 on dedicated multi-core hosts</code></pre>
  <em>Crucial Operational Prerequisite:</em> Adjust <code>innodb_purge_threads</code> <strong>only</strong> after verifying that long-running transactions are not the root cause, and after verifying sufficient host CPU headroom exists.</li>

  <li><strong>Chunk Large DML Operations:</strong><br>
  Avoid unconstrained multi-million-row <code>DELETE</code> or <code>UPDATE</code> operations:
<pre class="language-sql"><code class="language-sql">-- DANGEROUS: Floods undo tablespaces and overwhelms purge workers
DELETE FROM audit_log WHERE created_at &lt; NOW() - INTERVAL 2 YEAR;</code></pre>
  Instead, enforce deterministic, bounded chunking executed iteratively:
<pre class="language-sql"><code class="language-sql">-- RECOMMENDED: Bounded batch deletion allowing purge to pace smoothly
DELETE FROM audit_log 
 WHERE created_at &lt; NOW() - INTERVAL 2 YEAR 
 LIMIT 10000;</code></pre>
  Introduce brief sleep intervals between chunks if necessary to allow purge workers to keep pace with undo generation.</li>
</ol>

<div>
  <strong>Key Operational Takeaway:</strong> <code>wait/synch/cond/innodb/purge_worker_cond</code> is not, by itself, evidence of a performance bottleneck. The true indicators of purge lag are a continuously growing History List Length, expanding undo tablespaces, and long-running transactions that prevent the removal of obsolete MVCC row versions. In most production incidents, the root cause is not insufficient purge-worker capacity, but an application session holding an old read view open far longer than intended.
</div>

<hr>

<h3>Summary &amp; What's Next</h3>

<p>Storage and memory bottlenecks represent the physical layer of database performance. While sizing buffers properly is crucial, improper query design and uncommitted snapshots will overwhelm even the most capable NVMe storage arrays.</p>

<p>In <strong>Part 3: Network Transit, Connection Storms, and Replication Pipelines</strong>, the series moves to connection management and replication dynamics:</p>

<ul>
  <li>Thread cache mutex contention during microservice scaling events</li>
  <li>Client socket I/O waits and slow consumer remediation via Keyset Pagination</li>
  <li>Telemetry scrape storms (<code>LOCK_status</code>) and Prometheus exporter optimization</li>
  <li>Multi-Threaded Replication (MTR) worker bottlenecks and missing Primary Keys</li>
  <li>Distinguishing benign idle waits from genuine network delivery lag in replication receivers</li>
</ul>

<p>Continue to <strong>Part 3: Network Transit, Connection Storms, and Replication Pipelines</strong>.</p><div class="blogger-post-footer">7TKU66CE8V5W</div>]]></content:encoded>
    <pubDate>Tue, 10 Nov 2026 10:03:37 +0000</pubDate>
    <dc:creator>Umesh Shastry</dc:creator>
    <category>innodb-internals</category>
    <category>MySQL 8.4</category>
    <category>storage-and-memory</category>
    <category>wait-events</category>
  </item>

  <item>
    <title>Releasing iClaustron API, a new Rust NDB API with a C and a Rust API towards RonDB 26.10.0</title>
    <guid isPermaLink="false">tag:blogger.com,1999:blog-14455177.post-9088582667792024211</guid>
    <link>https://mikaelronstrom.blogspot.com/2026/11/releasing-iclaustron-api-new-rust-ndb.html</link>
    <description>We have released version 1.0.0 of the iClaustron API.
The github tree is at https://github.com/logicalclocks/iclaustron

Training material for Rust is available here

Training material for C is available here


Extensive training material on RonDB is available here

Some takeaways:

1. MIT licensed

2. Receiver thread only routing messages =&amp;gt; Scales to 3x more CPUs

3. Capable of doing 9M key lookups per second on a Mac M5 Pro laptop running cluster and API on the laptop

4. Cost of key lookup can be as low as 500ns per row in iClaustron API

5. 0 memory allocation for a normal key lookup

6. Currently supports Key lookups and Scan operations
</description>
    <content:encoded><![CDATA[We have released version 1.0.0 of the iClaustron API.
The github tree is at https://github.com/logicalclocks/iclaustron
<br>
<a href="https://slides.hops.io/2026-10/iclaustron-training/rust/01-overview/#modules">Training material for Rust is available here</a>
<br>
<a href="https://slides.hops.io/2026-10/iclaustron-training/c/01-overview/#modules">Training material for C is available here</a>
<br><br>

<a href="https://slides.hops.io/2026-10/rondb-training/01-why-rondb/#modules">Extensive training material on RonDB is available here</a>
<br><br>
Some takeaways:
<br><br>
1. MIT licensed
<br>
2. Receiver thread only routing messages =&gt; Scales to 3x more CPUs
<br>
3. Capable of doing 9M key lookups per second on a Mac M5 Pro laptop running cluster and API on the laptop
<br>
4. Cost of key lookup can be as low as 500ns per row in iClaustron API
<br>
5. 0 memory allocation for a normal key lookup
<br>
6. Currently supports Key lookups and Scan operations
<br>]]></content:encoded>
    <pubDate>Mon, 09 Nov 2026 16:08:00 +0000</pubDate>
    <dc:creator>Mikael Ronström</dc:creator>
  </item>

  <item>
    <title>Kiểm Thử PHP Kết Nối MySQL Trước Khi Phát Hành</title>
    <guid isPermaLink="false">https://dbahire.com/?p=3276</guid>
    <link>https://dbahire.com/kiem-thu-php-ket-noi-mysql-truoc-khi-phat-hanh</link>
    <description>Đã bao giờ bạn đẩy một tính năng lên máy chủ thật, rồi đến khi khách hàng đầu tiên bấm thanh toán thì hệ thống lại báo lỗi? Đây là tình huống khá quen thuộc với nhiều dự án nhỏ. Trong bài viết này, chúng tôi chia sẻ kiến thức lập trình web về cách kiểm thử ứng dụng PHP và MySQL bằng các công cụ phát triển phù hợp, trước khi người dùng thật bắt đầu sử dụng.
Vì sao lỗi kết nối PHP và MySQL thường chỉ lộ ra sau khi phát hành
Cách chạy thử theo cảm tính chỉ kiểm tra được những trường hợp thuận lợi: dữ liệu đúng chuẩn, số lượng bản ghi ít, không có ký tự lạ. Dữ liệu thật thì khác. Người dùng có thể bỏ trống một ô ngày tháng, nhập chuỗi rỗng, gõ tiếng Việt theo kiểu Unicode tổ hợp, hoặc nhập số âm vào ô số lượng.
Những trường hợp biên này ít khi được nghĩ tới khi viết code, nhưng lại thường gây lỗi truy vấn hoặc lỗi logic khi chạy trên môi trường thật. Một đoạn code có thể chạy ổn trên máy phát triển. Chỉ cần một đơn hàng có tên sản phẩm chứa dấu nháy đơn là chức năng đã gặp sự cố.
Sự khác biệt giữa local, staging và production cũng là nguyên nhân quan trọng. Trên máy cá nhân, dữ liệu thường được nhập tay với số lượng nhỏ. Khi chuyển sang máy chủ thật, phiên bản PHP có thể khác. Múi giờ của MySQL có thể lệch và dung lượng dữ liệu lớn hơn nhiều. Vì vậy, một lỗi tưởng đơn giản đôi khi tốn cả buổi để tái hiện.
Chi phí sửa lỗi cũng cần được cân nhắc. Lỗi phát hiện ngay khi viết code thường chỉ mất vài phút để sửa. Nếu lỗi lọt tới production, việc xử lý thường kéo theo nhiều bước: đọc log, dựng lại kịch bản gây lỗi, kiểm tra dữ liệu đã lưu có bị sai không, rồi mới viết bản vá. Với hệ thống đang phục vụ khách thật, một lỗi kết nối bị bỏ sót có thể làm đơn hàng thất lạc hoặc số liệu thanh toán sai lệch.
Các lớp kiểm thử cần có trong kiến thức lập trình web với PHP và MySQL
Một ứng dụng PHP kết nối MySQL nên được kiểm thử theo nhiều lớp thay vì chỉ chạy thử thủ công. Lớp đầu tiên là unit test cho logic nghiệp vụ, thường viết bằng PHPUnit. Điểm quan trọng là tách logic xử lý ra khỏi tầng truy vấn.
Hàm tính giá, hàm áp dụng khuyến mãi hay hàm kiểm tra điều kiện đơn hàng nên được test độc lập, không cần kết nối MySQL thật. Nhờ vậy test chạy nhanh và không bị ảnh hưởng bởi trạng thái dữ liệu. Bạn có thể bắt đầu bằng vài test cho những hàm quan trọng nhất của dự án.
Lớp thứ hai là test tích hợp, nơi ứng dụng làm việc trực tiếp với MySQL. Ở lớp này, dữ liệu mẫu (seed) nên đủ để mô phỏng cả tình huống thường gặp lẫn tình huống biên. Sau mỗi lần test, mọi thay đổi trong database cần được hoàn tác bằng transaction để không làm bẩn dữ liệu cho lần chạy sau.
Nếu bạn chưa quen thao tác với cấu trúc bảng, một công cụ trực quan hóa cơ sở dữ liệu sẽ giúp kiểm tra nhanh dữ liệu seed có đúng như kỳ vọng hay không, thay vì đọc từng dòng kết quả trên dòng lệnh.
Lớp thứ ba là kiểm tra hiệu năng truy vấn với dữ liệu lớn, và thường bị bỏ quên. Một câu truy vấn chạy nhanh với vài chục bản ghi có thể chậm đi rõ rệt khi bảng lên tới hàng trăm nghìn dòng, nhất là khi thiếu index phù hợp. Lỗi N+1 query cũng khá phổ biến: gọi lặp lại truy vấn cho từng phần tử trong vòng lặp thay vì gộp thành một câu duy nhất. Mô phỏng dữ liệu lớn trong môi trường test giúp phát hiện các điểm nghẽn này sớm.



Lớp kiểm thử
Trọng tâm kiểm tra
Thời điểm phù hợp để chạy




Unit test
Logic nghiệp vụ thuần túy, tách biệt khỏi database
Ngay khi vừa viết xong một hàm hoặc một lớp


Test tích hợp
Sự phối hợp giữa mã PHP và dữ liệu trong MySQL, có seed và rollback
Sau khi hoàn thiện một tính năng, trước khi gộp nhánh


Test hiệu năng
Tốc độ truy vấn, cách dùng index, nguy cơ N+1 khi dữ liệu tăng
Định kỳ hoặc trước các đợt phát hành lớn



Thiết lập pipeline kiểm thử tự động trước khi gộp code
Viết được test mới là bước đầu. Để test phát huy tác dụng, chúng cần được chạy đều đặn, không phụ thuộc vào việc ai đó nhớ ra mà bấm chạy tay. Nên tách riêng một database dành cho test, độc lập với database phát triển và database thật. Mỗi khi pipeline được kích hoạt, schema của database test nên được khởi tạo lại từ đầu, để mọi lần chạy đều bắt đầu từ trạng thái sạch.
Bước tiếp theo là gắn việc chạy test vào quy trình CI/CD. Nếu có test nào thất bại, hệ thống tự động chặn việc gộp code vào nhánh chính. Cách này bảo đảm mọi thay đổi đều qua một lớp kiểm tra, thay vì chỉ dựa vào cảm giác rằng code có vẻ ổn. Với nhóm nhỏ, phần thiết lập có thể chỉ gồm vài dòng cấu hình trên các nền tảng CI phổ biến, và lợi ích sẽ rõ hơn khi số người tham gia dự án tăng lên.
Nếu bạn mới làm quen với PHP và MySQL, hãy tìm hiểu thêm về các công cụ lập trình chuyên nghiệp để xây nền tảng đúng ngay từ đầu. Hiểu cách MySQL xử lý transaction, cách PHP quản lý kết nối cơ sở dữ liệu và cách công cụ debug hoạt động sẽ giúp việc viết test tự nhiên hơn. Bạn cũng nên nắm kiến thức nền tảng về hệ quản trị cơ sở dữ liệu MySQL để thiết kế schema test hợp lý, tránh phải sửa lại cấu trúc bảng khi dự án đã đi xa.
Để tìm hiểu chi tiết cách kết nối và xử lý lỗi với MySQL, bạn có thể tham khảo tài liệu PDO trong PHP trên trang chính thức.
Kết luận
Kiểm thử không phải bước làm cho có, mà là khoản đầu tư giúp giảm rủi ro khi hệ thống mở rộng. Một ứng dụng PHP kết nối MySQL được kiểm thử kỹ sẽ ít gặp sự cố bất ngờ hơn trên môi trường thật. Khi có sự cố, việc khoanh vùng nguyên nhân cũng nhanh hơn nhờ đã có sẵn các lớp test.
Không cần một hệ thống kiểm thử hoàn hảo ngay từ đầu, vì điều đó dễ khiến người mới quá tải và bỏ cuộc giữa chừng. Cách thực tế hơn là bắt đầu từ vài unit test cho các hàm quan trọng nhất, rồi dần bổ sung test tích hợp cho những luồng nghiệp vụ hay gây lỗi. Khi thói quen đã hình thành, việc gắn test vào pipeline sẽ trở thành bước tự nhiên tiếp theo.
The post Kiểm Thử PHP Kết Nối MySQL Trước Khi Phát Hành appeared first on DBAhire.</description>
    <content:encoded><![CDATA[<p>Đã bao giờ bạn đẩy một tính năng lên máy chủ thật, rồi đến khi khách hàng đầu tiên bấm thanh toán thì hệ thống lại báo lỗi? Đây là tình huống khá quen thuộc với nhiều dự án nhỏ. Trong bài viết này, chúng tôi chia sẻ kiến thức lập trình web về cách kiểm thử ứng dụng <strong>PHP</strong> và <strong>MySQL</strong> bằng các công cụ phát triển phù hợp, trước khi người dùng thật bắt đầu sử dụng.</p>
<h2>Vì sao lỗi kết nối PHP và MySQL thường chỉ lộ ra sau khi phát hành</h2>
<p>Cách chạy thử theo cảm tính chỉ kiểm tra được những trường hợp thuận lợi: dữ liệu đúng chuẩn, số lượng bản ghi ít, không có ký tự lạ. Dữ liệu thật thì khác. Người dùng có thể bỏ trống một ô ngày tháng, nhập chuỗi rỗng, gõ tiếng Việt theo kiểu Unicode tổ hợp, hoặc nhập số âm vào ô số lượng.</p>
<p>Những trường hợp biên này ít khi được nghĩ tới khi viết code, nhưng lại thường gây lỗi truy vấn hoặc lỗi logic khi chạy trên môi trường thật. Một đoạn code có thể chạy ổn trên máy phát triển. Chỉ cần một đơn hàng có tên sản phẩm chứa dấu nháy đơn là chức năng đã gặp sự cố.</p>
<p>Sự khác biệt giữa local, staging và production cũng là nguyên nhân quan trọng. Trên máy cá nhân, dữ liệu thường được nhập tay với số lượng nhỏ. Khi chuyển sang máy chủ thật, phiên bản PHP có thể khác. Múi giờ của MySQL có thể lệch và dung lượng dữ liệu lớn hơn nhiều. Vì vậy, một lỗi tưởng đơn giản đôi khi tốn cả buổi để tái hiện.</p>
<p>Chi phí sửa lỗi cũng cần được cân nhắc. Lỗi phát hiện ngay khi viết code thường chỉ mất vài phút để sửa. Nếu lỗi lọt tới production, việc xử lý thường kéo theo nhiều bước: đọc log, dựng lại kịch bản gây lỗi, kiểm tra dữ liệu đã lưu có bị sai không, rồi mới viết bản vá. Với hệ thống đang phục vụ khách thật, một lỗi kết nối bị bỏ sót có thể làm đơn hàng thất lạc hoặc số liệu thanh toán sai lệch.</p>
<h2>Các lớp kiểm thử cần có trong kiến thức lập trình web với PHP và MySQL</h2>
<p>Một ứng dụng PHP kết nối MySQL nên được kiểm thử theo nhiều lớp thay vì chỉ chạy thử thủ công. Lớp đầu tiên là <em>unit test</em> cho logic nghiệp vụ, thường viết bằng PHPUnit. Điểm quan trọng là tách logic xử lý ra khỏi tầng truy vấn.</p>
<p>Hàm tính giá, hàm áp dụng khuyến mãi hay hàm kiểm tra điều kiện đơn hàng nên được test độc lập, không cần kết nối MySQL thật. Nhờ vậy test chạy nhanh và không bị ảnh hưởng bởi trạng thái dữ liệu. Bạn có thể bắt đầu bằng vài test cho những hàm quan trọng nhất của dự án.</p>
<p>Lớp thứ hai là test tích hợp, nơi ứng dụng làm việc trực tiếp với MySQL. Ở lớp này, dữ liệu mẫu (seed) nên đủ để mô phỏng cả tình huống thường gặp lẫn tình huống biên. Sau mỗi lần test, mọi thay đổi trong database cần được hoàn tác bằng transaction để không làm bẩn dữ liệu cho lần chạy sau.</p>
<p>Nếu bạn chưa quen thao tác với cấu trúc bảng, một <a href="https://dbahire.com/mysql-workbench-la-gi-tai-sao-phai-cai-dat-mysql-workbench">công cụ trực quan hóa cơ sở dữ liệu</a> sẽ giúp kiểm tra nhanh dữ liệu seed có đúng như kỳ vọng hay không, thay vì đọc từng dòng kết quả trên dòng lệnh.</p>
<p>Lớp thứ ba là kiểm tra hiệu năng truy vấn với dữ liệu lớn, và thường bị bỏ quên. Một câu truy vấn chạy nhanh với vài chục bản ghi có thể chậm đi rõ rệt khi bảng lên tới hàng trăm nghìn dòng, nhất là khi thiếu index phù hợp. Lỗi N+1 query cũng khá phổ biến: gọi lặp lại truy vấn cho từng phần tử trong vòng lặp thay vì gộp thành một câu duy nhất. Mô phỏng dữ liệu lớn trong môi trường test giúp phát hiện các điểm nghẽn này sớm.</p>
<table>
<thead>
<tr>
<th>Lớp kiểm thử</th>
<th>Trọng tâm kiểm tra</th>
<th>Thời điểm phù hợp để chạy</th>
</tr>
</thead>
<tbody>
<tr>
<td>Unit test</td>
<td>Logic nghiệp vụ thuần túy, tách biệt khỏi database</td>
<td>Ngay khi vừa viết xong một hàm hoặc một lớp</td>
</tr>
<tr>
<td>Test tích hợp</td>
<td>Sự phối hợp giữa mã PHP và dữ liệu trong MySQL, có seed và rollback</td>
<td>Sau khi hoàn thiện một tính năng, trước khi gộp nhánh</td>
</tr>
<tr>
<td>Test hiệu năng</td>
<td>Tốc độ truy vấn, cách dùng index, nguy cơ N+1 khi dữ liệu tăng</td>
<td>Định kỳ hoặc trước các đợt phát hành lớn</td>
</tr>
</tbody>
</table>
<h2>Thiết lập pipeline kiểm thử tự động trước khi gộp code</h2>
<p>Viết được test mới là bước đầu. Để test phát huy tác dụng, chúng cần được chạy đều đặn, không phụ thuộc vào việc ai đó nhớ ra mà bấm chạy tay. Nên tách riêng một database dành cho test, độc lập với database phát triển và database thật. Mỗi khi pipeline được kích hoạt, schema của database test nên được khởi tạo lại từ đầu, để mọi lần chạy đều bắt đầu từ trạng thái sạch.</p>
<p>Bước tiếp theo là gắn việc chạy test vào quy trình CI/CD. Nếu có test nào thất bại, hệ thống tự động chặn việc gộp code vào nhánh chính. Cách này bảo đảm mọi thay đổi đều qua một lớp kiểm tra, thay vì chỉ dựa vào cảm giác rằng code có vẻ ổn. Với nhóm nhỏ, phần thiết lập có thể chỉ gồm vài dòng cấu hình trên các nền tảng CI phổ biến, và lợi ích sẽ rõ hơn khi số người tham gia dự án tăng lên.</p>
<p>Nếu bạn mới làm quen với PHP và MySQL, hãy tìm hiểu thêm về <a href="https://dbahire.com/top-10-web-noi-ve-cac-tools-lap-trinh-chuyen-nghiep">các công cụ lập trình chuyên nghiệp</a> để xây nền tảng đúng ngay từ đầu. Hiểu cách MySQL xử lý transaction, cách PHP quản lý kết nối cơ sở dữ liệu và cách công cụ debug hoạt động sẽ giúp việc viết test tự nhiên hơn. Bạn cũng nên nắm <a href="https://dbahire.com/tong-quan-ve-he-quan-tri-co-so-du-lieu-mysql">kiến thức nền tảng về hệ quản trị cơ sở dữ liệu MySQL</a> để thiết kế schema test hợp lý, tránh phải sửa lại cấu trúc bảng khi dự án đã đi xa.</p>
<p>Để tìm hiểu chi tiết cách kết nối và xử lý lỗi với MySQL, bạn có thể tham khảo <a href="https://www.php.net/manual/en/book.pdo.php">tài liệu PDO trong PHP trên trang chính thức</a>.</p>
<h2>Kết luận</h2>
<p>Kiểm thử không phải bước làm cho có, mà là khoản đầu tư giúp giảm rủi ro khi hệ thống mở rộng. Một ứng dụng PHP kết nối MySQL được kiểm thử kỹ sẽ ít gặp sự cố bất ngờ hơn trên môi trường thật. Khi có sự cố, việc khoanh vùng nguyên nhân cũng nhanh hơn nhờ đã có sẵn các lớp test.</p>
<p>Không cần một hệ thống kiểm thử hoàn hảo ngay từ đầu, vì điều đó dễ khiến người mới quá tải và bỏ cuộc giữa chừng. Cách thực tế hơn là bắt đầu từ vài unit test cho các hàm quan trọng nhất, rồi dần bổ sung test tích hợp cho những luồng nghiệp vụ hay gây lỗi. Khi thói quen đã hình thành, việc gắn test vào pipeline sẽ trở thành bước tự nhiên tiếp theo.</p>
<p>The post <a rel="nofollow" href="https://dbahire.com/kiem-thu-php-ket-noi-mysql-truoc-khi-phat-hanh">Kiểm Thử PHP Kết Nối MySQL Trước Khi Phát Hành</a> appeared first on <a rel="nofollow" href="https://dbahire.com/">DBAhire</a>.</p>]]></content:encoded>
    <pubDate>Sun, 08 Nov 2026 01:39:00 +0000</pubDate>
    <dc:creator>Jaime Crespo</dc:creator>
  </item>

  <item>
    <title>Modernize with Confidence: Best Practices for Migrating to MySQL HeatWave on OCI</title>
    <guid isPermaLink="false">cf22adc450d11d01b44d28b9198991d4</guid>
    <link>https://blogs.oracle.com/mysql/modernize-with-confidence-best-practices-for-migrating-to-mysql-heatwave-on-oci</link>
    <description>For organizations running business-critical MySQL applications, database modernization must deliver value while protecting day-to-day operations. Retailers need reliable checkout experiences. Financial services firms need accurate transactions and reporting. Manufacturers need timely visibility across production and supply chains. Migrating to MySQL HeatWave on Oracle Cloud Infrastructure (OCI) should therefore be treated as a business transformation initiative—with […]</description>
    <pubDate>Sat, 07 Nov 2026 03:58:55 +0000</pubDate>
    <dc:creator>Oracle MySQL Group</dc:creator>
    <category>MySQL</category>
    <category>MySQL Basics/How-To</category>
    <category>MySQL HeatWave</category>
    <category>OCI</category>
  </item>

  <item>
    <title>Vector search in MySQL: from a plugin to the top of the Contributor Summit agenda</title>
    <guid isPermaLink="false">https://askdba.net/?p=3145</guid>
    <link>https://askdba.net/vector-search-in-mysql-from-a-plugin-to-the-top-of-the-contributor-summit-agenda/</link>
    <description>
A plugin started the fire, the M-ecosystem picked it up, the hyperscalers went native, and now the benchmark wars have begun.



Not long ago, if you asked about vector search in MySQL, you got a polite smile and a link to pgvector. Even after MySQL 9 added a VECTOR column type, the obvious next questions had no answer. Where is the index? Where is the distance function? For a while the honest answer was “use something else.”



This week the MySQL Contributor Summit (Nov 4–5, online) opened its technical program with vector search. The first discussion block, AI &amp;amp; Cloud, had four vector and AI proposals back to back from Oracle, Google, Percona and Intel. It was the clearest sign yet of how far vector search in MySQL has come.



Looking back, vector search in MySQL has moved through four phases:




The spark: MyVector


Adoption: the M-ecosystem picks up the idea


Going native: the hyperscalers arrive


The benchmark wars: where we are now





Phase 1: The spark, MyVector



We didn’t want to wait for a server release, so I co-founded MyVector, which I now own. It gives MySQL HNSW indexing, distance functions and approximate nearest neighbor search, all open source.



MyVector started as a plugin on MySQL 8.0. It has since moved to MySQL’s component architecture, which is far more robust on recent MySQL releases, and it now supports MySQL Community Edition 8.4, 9.7 and 26.10.



We took it to FOSDEM and SCALE, and the conversation changed from “MySQL can’t do this” to “wait, MySQL can do this?” You can try it yourself at demo.myvector.online.



Phase 2: Adoption, the M-ecosystem picks it up



Once MyVector showed that vector search fits naturally inside a MySQL-compatible engine, the idea spread across the M-ecosystem. MariaDB, TiDB, VillageSQL and Percona all started adopting the idea in their own products and distributions.



Each took its own path, but they all headed the same way: embeddings belong next to the data, inside the database people already run.



Phase 3: Going native, the hyperscalers arrive



The third phase is native implementations, with contributions now coming from the hyperscalers and the biggest names in the ecosystem. You could see it at the summit, where the four AI &amp;amp; Cloud talks came from the upstream owner, a hyperscaler, a fork vendor and a chip maker:



SessionFromGitHubEvolving MySQL VECTOR Support: Native Indexing and Similarity SearchMayank Prasad, Kuba LopuszanskiOracleIssue #3Invoking AI models from MySQLJulia OffermanGoogleIssue #158DISTANCE() Functions for the VECTOR Data TypeCatalin BesleagaPerconaPR #860Bringing SVS’s Graph-Based Vector Search to MySQLKelly L. McKeighanIntelIssue #116AI &amp;amp; Cloud discussion block, MySQL Contributor Summit Fall 2026, followed by a design review with all attendees



All four brought proposals, bug reports and pull requests for the same feature.



Phase 4: The benchmark wars



With several implementations in play, the next step was inevitable: people started measuring in public and answering each other’s numbers.



Percona, Aug 27: Benchmarking vector indexes



Evgeniy Patlan set the ground rules with vector-bench, an open harness that runs each engine in its own pinned container on the same cores with the same data. Every measurement is a pair of throughput and recall, and recall is a setting, not a property of the engine.



ef_search 10 → 800 · recall 0.959 → 0.999 · 3,678 → 409 QPS



askdba, Oct 14: How fast is vector search inside MySQL?



I ran MyVector on MySQL against MariaDB Vector on 50,000 vectors. MariaDB’s first run came in at only 29 QPS. Instead of publishing that, I traced it to the 16 MB default HNSW cache, which could not hold the 27 MB index, and reran it.



MyVector ≈ 548 QPS @ recall 0.82 · MariaDB ≈ 257 QPS @ recall 0.95



lefred (MariaDB), Oct 16: Vector Search in MySQL vs MariaDB: Look Beyond QPS



lefred answered: two QPS figures at different recall are not apples to apples. He estimated MariaDB at matched recall, showed that a bigger cache cut index build from about 583 to 92 seconds, and argued the 16 MB default deserves a rethink. Fair points, made in good faith.



MariaDB est. ≈ 1,196 QPS @ recall 0.82 · build 583 s → 92 s



VillageSQL, Oct 22: Benchmarking vector indexes in MySQL with vector-dev-bench



Adam Levin, working with Percona, released vector-dev-bench, a laptop-scale companion that answers in minutes. It sweeps ef_search and measures what writers cost readers.



20k × 32-dim · recall 0.787 → 0.988 · 8,829 → 2,044 QPS · reads −40% with writers



Every round raised the bar. The community converged on pairing throughput with recall, exposing defaults and testing writes alongside reads. MyVector is being added to vector-bench as an engine, so anyone will be able to rerun the numbers. Results for MyVector and other engines are published side by side at bench.myvector.online. That is how an open-source ecosystem decides something matters.




The spark is now a fire.




Why this matters



We have seen this pattern before. JSON in MySQL went through the same steps: store it as text, then a native type, then functions, then indexes, and finally the default choice.



Vectors are moving through those steps much faster, because a plugin proved it was possible, the ecosystem adopted it, the hyperscalers are building it in, and the benchmarks keep everyone honest.



For teams running MySQL in production, the question is moving away from “which separate vector database?” and toward “can my embeddings live next to my transactional data, in the database my team already knows how to run?”



Join the conversation



The Contributor Summit is open and online. Proposals are filed as issues in the mysql/mysql-community GitHub repo, and the hallway track continues in the MySQL Community Slack in #contributor-summit-hallway-track. If you work on vector search in MySQL, bring your ideas, your pull requests and your benchmarks to the next one.



What’s next




Native indexing and similarity search proposals moving through upstream design review


Standard DISTANCE() functions replacing vendor-specific workarounds


MyVector continuing as a way to run vector search on MySQL Community Edition today, with open, reproducible benchmarks





We’re not done. Thanks to Heather VanCura and the MySQL Community team for running an open, design-review-style summit.



Links




myvector.online


bench.myvector.online


demo.myvector.online


github.com/askdba/myvector

</description>
    <content:encoded><![CDATA[<p><img src="https://askdba.net/wp-content/uploads/2026/11/vector-search-mysql-contributor-summit-2026.png" width="1200" height="630" alt="MySQL Contributor Summit, Nov 4–5, 2026: vector search in MySQL in four phases, from the MyVector spark to the benchmark wars"></p>
<p class="wp-block-paragraph"><em>A plugin started the fire, the M-ecosystem picked it up, the hyperscalers went native, and now the benchmark wars have begun.</em></p>



<p class="wp-block-paragraph">Not long ago, if you asked about vector search in MySQL, you got a polite smile and a link to pgvector. Even after MySQL 9 added a <code>VECTOR</code> column type, the obvious next questions had no answer. Where is the index? Where is the distance function? For a while the honest answer was “use something else.”</p>



<p class="wp-block-paragraph">This week the <a href="https://github.com/mysql/mysql-community/wiki/MySQL-Contributor-Summit-Fall,-Nov-4%E2%80%905,-2026-%E2%80%90-Proposed-Agenda">MySQL Contributor Summit</a> (Nov 4–5, online) opened its technical program with vector search. The first discussion block, AI &amp; Cloud, had four vector and AI proposals back to back from Oracle, Google, Percona and Intel. It was the clearest sign yet of how far vector search in MySQL has come.</p>



<p class="wp-block-paragraph">Looking back, vector search in MySQL has moved through four phases:</p>



<ol class="wp-block-list">
<li><strong>The spark:</strong> MyVector</li>


<li><strong>Adoption:</strong> the M-ecosystem picks up the idea</li>


<li><strong>Going native:</strong> the hyperscalers arrive</li>


<li><strong>The benchmark wars:</strong> where we are now</li>

</ol>



<h2 class="wp-block-heading">Phase 1: The spark, MyVector</h2>



<p class="wp-block-paragraph">We didn’t want to wait for a server release, so I co-founded <strong>MyVector</strong>, which I now own. It gives MySQL HNSW indexing, distance functions and approximate nearest neighbor search, all open source.</p>



<p class="wp-block-paragraph">MyVector started as a plugin on MySQL 8.0. It has since moved to MySQL’s component architecture, which is far more robust on recent MySQL releases, and it now supports MySQL Community Edition 8.4, 9.7 and 26.10.</p>



<p class="wp-block-paragraph">We took it to FOSDEM and SCALE, and the conversation changed from “MySQL can’t do this” to “wait, MySQL can do this?” You can try it yourself at <a href="https://demo.myvector.online/">demo.myvector.online</a>.</p>



<h2 class="wp-block-heading">Phase 2: Adoption, the M-ecosystem picks it up</h2>



<p class="wp-block-paragraph">Once MyVector showed that vector search fits naturally inside a MySQL-compatible engine, the idea spread across the M-ecosystem. <strong>MariaDB</strong>, <strong>TiDB</strong>, <strong>VillageSQL</strong> and <strong>Percona</strong> all started adopting the idea in their own products and distributions.</p>



<p class="wp-block-paragraph">Each took its own path, but they all headed the same way: embeddings belong next to the data, inside the database people already run.</p>



<h2 class="wp-block-heading">Phase 3: Going native, the hyperscalers arrive</h2>



<p class="wp-block-paragraph">The third phase is native implementations, with contributions now coming from the hyperscalers and the biggest names in the ecosystem. You could see it at the summit, where the four AI &amp; Cloud talks came from the upstream owner, a hyperscaler, a fork vendor and a chip maker:</p>



<figure class="wp-block-table"><table class="has-fixed-layout"><thead><tr><th>Session</th><th>From</th><th>GitHub</th></tr></thead><tbody><tr><td>Evolving MySQL VECTOR Support: Native Indexing and Similarity Search<br><em>Mayank Prasad, Kuba Lopuszanski</em></td><td><strong>Oracle</strong></td><td><a href="https://github.com/mysql/mysql-community/issues/3">Issue #3</a></td></tr><tr><td>Invoking AI models from MySQL<br><em>Julia Offerman</em></td><td><strong>Google</strong></td><td><a href="https://github.com/mysql/mysql-community/issues/158">Issue #158</a></td></tr><tr><td>DISTANCE() Functions for the VECTOR Data Type<br><em>Catalin Besleaga</em></td><td><strong>Percona</strong></td><td><a href="https://github.com/mysql/mysql-server/pull/860">PR #860</a></td></tr><tr><td>Bringing SVS’s Graph-Based Vector Search to MySQL<br><em>Kelly L. McKeighan</em></td><td><strong>Intel</strong></td><td><a href="https://github.com/mysql/mysql-community/issues/116">Issue #116</a></td></tr></tbody></table><figcaption class="wp-element-caption">AI &amp; Cloud discussion block, MySQL Contributor Summit Fall 2026, followed by a design review with all attendees</figcaption></figure>



<p class="wp-block-paragraph">All four brought proposals, bug reports and pull requests for the same feature.</p>



<h2 class="wp-block-heading">Phase 4: The benchmark wars</h2>



<p class="wp-block-paragraph">With several implementations in play, the next step was inevitable: people started measuring in public and answering each other’s numbers.</p>



<h3 class="wp-block-heading">Percona, Aug 27: <a href="https://www.percona.com/blog/benchmarking-vector-indexes/">Benchmarking vector indexes</a></h3>



<p class="wp-block-paragraph">Evgeniy Patlan set the ground rules with <a href="https://github.com/Percona-Lab/vector-bench">vector-bench</a>, an open harness that runs each engine in its own pinned container on the same cores with the same data. Every measurement is a pair of throughput and recall, and recall is a setting, not a property of the engine.</p>



<p class="wp-block-paragraph"><code>ef_search 10 → 800 · recall 0.959 → 0.999 · 3,678 → 409 QPS</code></p>



<h3 class="wp-block-heading">askdba, Oct 14: <a href="https://askdba.net/how-fast-is-vector-search-inside-mysql/">How fast is vector search inside MySQL?</a></h3>



<p class="wp-block-paragraph">I ran MyVector on MySQL against MariaDB Vector on 50,000 vectors. MariaDB’s first run came in at only 29 QPS. Instead of publishing that, I traced it to the 16 MB default HNSW cache, which could not hold the 27 MB index, and reran it.</p>



<p class="wp-block-paragraph"><code>MyVector ≈ 548 QPS @ recall 0.82 · MariaDB ≈ 257 QPS @ recall 0.95</code></p>



<h3 class="wp-block-heading">lefred (MariaDB), Oct 16: <a href="https://lefred.be/content/vector-search-in-mysql-vs-mariadb-look-beyond-qps/">Vector Search in MySQL vs MariaDB: Look Beyond QPS</a></h3>



<p class="wp-block-paragraph">lefred answered: two QPS figures at different recall are not apples to apples. He estimated MariaDB at matched recall, showed that a bigger cache cut index build from about 583 to 92 seconds, and argued the 16 MB default deserves a rethink. Fair points, made in good faith.</p>



<p class="wp-block-paragraph"><code>MariaDB est. ≈ 1,196 QPS @ recall 0.82 · build 583 s → 92 s</code></p>



<h3 class="wp-block-heading">VillageSQL, Oct 22: <a href="https://villagesql.com/blog/vector-dev-bench/">Benchmarking vector indexes in MySQL with vector-dev-bench</a></h3>



<p class="wp-block-paragraph">Adam Levin, working with Percona, released <a href="https://github.com/villagesql/vector-dev-bench">vector-dev-bench</a>, a laptop-scale companion that answers in minutes. It sweeps ef_search and measures what writers cost readers.</p>



<p class="wp-block-paragraph"><code>20k × 32-dim · recall 0.787 → 0.988 · 8,829 → 2,044 QPS · reads −40% with writers</code></p>



<p class="wp-block-paragraph">Every round raised the bar. The community converged on pairing throughput with recall, exposing defaults and testing writes alongside reads. MyVector is being <a href="https://github.com/EvgeniyPatlan/vector-bench/pull/73">added to vector-bench as an engine</a>, so anyone will be able to rerun the numbers. Results for MyVector and other engines are published side by side at <a href="https://bench.myvector.online/">bench.myvector.online</a>. That is how an open-source ecosystem decides something matters.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">The spark is now a fire.</p>
</blockquote>



<h2 class="wp-block-heading">Why this matters</h2>



<p class="wp-block-paragraph">We have seen this pattern before. JSON in MySQL went through the same steps: store it as text, then a native type, then functions, then indexes, and finally the default choice.</p>



<p class="wp-block-paragraph">Vectors are moving through those steps much faster, because a plugin proved it was possible, the ecosystem adopted it, the hyperscalers are building it in, and the benchmarks keep everyone honest.</p>



<p class="wp-block-paragraph">For teams running MySQL in production, the question is moving away from “which separate vector database?” and toward “can my embeddings live next to my transactional data, in the database my team already knows how to run?”</p>



<h2 class="wp-block-heading">Join the conversation</h2>



<p class="wp-block-paragraph">The Contributor Summit is open and online. Proposals are filed as issues in the <a href="https://github.com/mysql/mysql-community">mysql/mysql-community</a> GitHub repo, and the hallway track continues in the MySQL Community Slack in #contributor-summit-hallway-track. If you work on vector search in MySQL, bring your ideas, your pull requests and your benchmarks to the next one.</p>



<h2 class="wp-block-heading">What’s next</h2>



<ul class="wp-block-list">
<li>Native indexing and similarity search proposals moving through upstream design review</li>


<li>Standard <code>DISTANCE()</code> functions replacing vendor-specific workarounds</li>


<li>MyVector continuing as a way to run vector search on MySQL Community Edition today, with open, reproducible benchmarks</li>

</ul>



<p class="wp-block-paragraph">We’re not done. Thanks to Heather VanCura and the MySQL Community team for running an open, design-review-style summit.</p>



<h2 class="wp-block-heading">Links</h2>



<ul class="wp-block-list">
<li><a href="https://myvector.online/">myvector.online</a></li>


<li><a href="https://bench.myvector.online/">bench.myvector.online</a></li>


<li><a href="https://demo.myvector.online/">demo.myvector.online</a></li>


<li><a href="https://github.com/askdba/myvector">github.com/askdba/myvector</a></li>

</ul>]]></content:encoded>
    <pubDate>Fri, 06 Nov 2026 21:01:56 +0000</pubDate>
    <dc:creator>Alkin Tezuysal</dc:creator>
    <category>mysql</category>
    <category>Technical</category>
    <category>benchmark</category>
    <category>contributor-summit</category>
    <category>mariadb</category>
    <category>myvector</category>
    <category>search</category>
    <category>vector</category>
  </item>

  <item>
    <title>MySQL Ecosystem Project Spotlight of the Week - ProxySQL</title>
    <guid isPermaLink="false">planet-manual-02a24a897c074c014da8616e0d68fd0b</guid>
    <link>https://www.linkedin.com/feed/update/urn:li:activity:7524499184880844800/?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAAnXxcBZjEDWwCw5j05YNBM72lQlGWdNio</link>
    <description>This week, we're highlighting ProxySQL. As applications grow, so do the demands on MySQL. ProxySQL helps developers and DBAs manage database traffic through connection pooling, query routing, and load balancing, providing greater control over how applications connect to MySQL.</description>
    <pubDate>Fri, 06 Nov 2026 16:18:28 +0000</pubDate>
    <dc:creator>MySQL Community Team</dc:creator>
  </item>

  <item>
    <title>Upcoming MySQL Events: November 2026–February 2027</title>
    <guid isPermaLink="false">3a4159c19ff341e1aaebfd3812db95c1</guid>
    <link>https://blogs.oracle.com/mysql/upcoming-mysql-events-november-2026-february-2027</link>
    <description>Join MySQL engineers, developers, DBAs, and community members at conferences and local gatherings across Europe, Asia, and North America. This roundup includes confirmed MySQL team appearances, community-led sessions, and a preview of activities being planned. Explore the events below to find opportunities to learn, share practical experience, and connect with the community. Conferences KubeCon + […]</description>
    <pubDate>Fri, 06 Nov 2026 15:53:24 +0000</pubDate>
    <dc:creator>Oracle MySQL Group</dc:creator>
    <category>MySQL</category>
    <category>MySQL Community</category>
  </item>

  <item>
    <title>OurSQL Digest: What Matters in MySQL Ecosystem in October 2026</title>
    <guid isPermaLink="false">https://oursqlfoundation.org/blog/oursql-digest-october-2026/</guid>
    <link>https://oursqlfoundation.org/blog/oursql-digest-october-2026/</link>
    <description>So, we decided to try something new: a monthly digest. Curated MySQL ecosystem news of the month in one place, so you can understand what is going on in MySQL from one email, without reading twenty blogs and three Slack channels.
In October our team, with our AI friend, went through something like 150 posts, release notes, announcements, and social media posts. Below is what we think mattered most, and why it matters. Of course this is our choice, so it is opinionated. If you think we missed something or got something wrong, tell us in OurSQL Foundation Slack, we really like when people do this.</description>
    <content:encoded><![CDATA[<p>So, we decided to try something new: a monthly digest. Curated MySQL ecosystem news of the month in one place, so you can understand what is going on in MySQL from one email, without reading twenty blogs and three Slack channels.</p>
<p>In October our team, with our AI friend, went through something like 150 posts, release notes, announcements, and social media posts. Below is what we think mattered most, and why it matters. Of course this is our choice, so it is opinionated. If you think we missed something or got something wrong, tell us in OurSQL Foundation <a href="https://join.slack.com/t/oursqlfoundation/shared_invite/zt-480k6h62a-t0HalCPwvXUC_lpd4aNylA">Slack</a>, we really like when people do this.</p>]]></content:encoded>
    <pubDate>Fri, 06 Nov 2026 00:00:00 +0000</pubDate>
    <dc:creator>OurSQL Foundation</dc:creator>
  </item>

  <item>
    <title>Diagnosing MySQL at Scale (Part 1): Critical System Blocks, Metadata Locks, and Cluster Convoys</title>
    <guid isPermaLink="false">tag:blogger.com,1999:blog-5421473080502311460.post-8653731598327108272</guid>
    <link>https://ushastry.blogspot.com/2026/11/diagnosing-mysql-at-scale-part-1.html</link>
    <description>High-concurrency database workloads rarely fail because of simple CPU or disk capacity limits. Instead, performance collapses occur when transactions serialize on shared internal database resources: metadata locks, data dictionary latches, group communication certification queues, and network buffers.

When contention occurs, external monitoring dashboards often show deceptive metrics such as near-zero CPU and disk activity alongside a massive surge in active connections and application timeouts. MySQL exposes these internal contention points through Performance Schema wait events.

This article—the first in a four-part deep dive—covers the foundational mechanics of Performance Schema wait telemetry, the governance of instrumentation overhead in high-throughput environments, and the diagnosis of Critical System Blocks: Metadata Locks (MDL), Data Dictionary latch serialization, and Group Replication Flow-Control stalls.


  Operational Disclaimer: The diagnostic queries, Performance Schema instrumentation configurations, engine parameter recommendations, stored routines, and architectural remediations presented across this series are provided for investigative engineering analysis and technical reference. Contention characteristics, wait-event profiles, and optimal parameter thresholds vary significantly depending on transactional access patterns, data volume, schema design, storage subsystem throughput, replication topology, cloud virtualization constraints (e.g., AWS Nitro, EBS bandwidth limits, Aurora multi-tenant architectures), and specific MySQL engine release baselines. Administrative overrides, dynamic parameter modifications, session terminations (KILL), and schema modifications should never be executed blindly against production clusters. All queries, DDL statements, and configuration changes must be reviewed, benchmarked, and validated in representative staging environments through established change-management procedures prior to production implementation.


1. Why Wait Events Matter in High Concurrency

In MySQL, every client query competes for finite shared resources. Rather than guessing whether a latency spike is driven by storage latency, table schema locks, or network transport, wait events quantify the exact time threads spend waiting:


  


Core Principle: Analyzing wait-event distributions and latency profiles transforms database troubleshooting from speculative trial-and-error into a deterministic diagnostic process.

Official documentation: MySQL 8.4 Reference Manual: Performance Schema Wait Event Tables.



2. Performance Schema Prerequisites &amp;amp; Overhead Governance

By default, MySQL 8.4 LTS enables performance_schema, but detailed wait-event visibility is intentionally limited. Although numerous Performance Schema instruments are active, many wait-related instruments and the wait-event consumers (events_waits_%) remain disabled out of the box to reduce instrumentation overhead. Consequently, Performance Schema may initially expose statement-level and transaction-level diagnostics while providing little or no wait-event data until the relevant instruments and consumers are explicitly enabled.

Instrumentation Enablement Matrix


  
    
      Component
      Default State
      Operational Function
    
  
  
    
      performance_schema
      ON
      Master telemetry engine toggle.
    
    
      setup_instruments (wait/%)
      SELECTIVELY ENABLED
      Low-level probes embedded in server source code (mutexes, rwlocks, file I/O, socket I/O, metadata locking, and other synchronization primitives). Some wait instruments are enabled by default, others are disabled.
    
    
      setup_consumers (events_waits_*)
      DISABLED
      In-memory aggregation tables and history buffers capturing wait-event telemetry. Wait-event consumers are disabled by default.
    
  


Measured Overhead: What Telemetry Costs

Not all Performance Schema instruments impose the same overhead. High-frequency synchronization instruments such as mutexes and condition variables may execute millions of times per second on heavily concurrent OLTP systems. When instrumentation is enabled with TIMED=YES, additional high-resolution timing operations are performed for each instrumented event, increasing CPU consumption relative to untimed collection. The overhead estimates below represent illustrative field-observed ranges and should be interpreted as directional guidance rather than deterministic performance guarantees.


  
    
      Instrumentation Scope
      Typical CPU Overhead*
      Throughput Impact
      Workload Context
    
  
  
    
      Default Instruments (out-of-the-box)
      1–3%
      Negligible
      Standard production baseline
    
    
      Targeted (wait/io/%, wait/lock/%)
      3–8%
      Moderate
      Recommended for production triage
    
    
      All wait/synch/mutex with TIMED=YES
      10–25%
      High
      Heavy concurrent OLTP workloads
    
    
      Blanket wait/% with TIMED=YES
      15–40%
      Severe
      High-concurrency throughput regression
    
    
      Instruments ENABLED=YES, TIMED=NO
      2–5%
      Minimal
      Frequency and contention counts only
    
  


* Performance Impact Disclaimer: The overhead percentages shown above are illustrative field-observed ranges derived from benchmarking exercises, vendor guidance, and production support experience across diverse environments. Actual impact may vary substantially depending on workload characteristics, transaction concurrency, CPU architecture, storage latency, enabled consumers, instrumentation scope, MySQL version, and cloud platform implementation details. These figures should be interpreted as directional guidance rather than deterministic performance guarantees.

Operational Guideline: The TIMED=YES attribute on high-frequency synchronization instruments (particularly mutexes and condition variables) is often the largest contributor to instrumentation overhead. When investigating contention frequency, hotspot identification, or lock distribution rather than precise latency measurements, consider using TIMED='NO' to reduce timing-related overhead while still collecting event counts and contention statistics.

Dynamic Runtime Activation (Diagnostic Session)

To enable wait instrumentation dynamically during an incident window without restarting the server:

-- Dynamic Runtime Activation (Diagnostic Session)
-- Enable targeted wait instrumentation during an incident window without requiring a MySQL restart.

-- 1. Enable targeted wait instruments
UPDATE performance_schema.setup_instruments
   SET ENABLED = 'YES',
       TIMED   = 'YES'
 WHERE NAME LIKE 'wait/io/%'
    OR NAME LIKE 'wait/lock/%'
    OR NAME LIKE 'wait/synch/mutex/innodb/%';

-- 2. Enable wait-event consumers
UPDATE performance_schema.setup_consumers
   SET ENABLED = 'YES'
 WHERE NAME IN (
       'events_waits_current',
       'events_waits_history',
       'events_waits_history_long'
 );

-- 3. Verify enabled instrumentation
SELECT NAME,
       ENABLED,
       TIMED
  FROM performance_schema.setup_instruments
 WHERE NAME LIKE 'wait/%'
   AND ENABLED = 'YES'
 ORDER BY NAME
 LIMIT 20;

-- 4. Verify enabled consumers
SELECT NAME,
       ENABLED
  FROM performance_schema.setup_consumers
 WHERE NAME LIKE 'events_waits%';

Operational Note: Instrumentation changes affect only future wait events and do not retroactively populate historical data. Enable the required instruments and consumers, allow representative workload activity to execute, and then analyze wait-event telemetry.

Production Guidance: When investigating contention frequency rather than precise latency measurements, consider enabling instruments with TIMED='NO' first. This significantly reduces instrumentation overhead while still exposing contention hotspots and event frequency distributions.

Post-Investigation Cleanup Command

Always return the instance to safe telemetry baselines following an investigation:

-- Post-Investigation Cleanup
-- Revert any instrumentation that was enabled specifically for the investigation.
-- Ideally, restore the pre-investigation configuration captured before enabling additional instruments and consumers.

-- Example: Disable instrumentation enabled for the investigation
UPDATE performance_schema.setup_instruments
   SET ENABLED = 'NO',
       TIMED   = 'NO'
 WHERE NAME LIKE 'wait/io/%'
    OR NAME LIKE 'wait/lock/%'
    OR NAME LIKE 'wait/synch/mutex/innodb/%';

-- Disable wait-event consumers
UPDATE performance_schema.setup_consumers
   SET ENABLED = 'NO'
 WHERE NAME IN (
       'events_waits_current',
       'events_waits_history',
       'events_waits_history_long'
 );

Operational Guideline: Before enabling additional Performance Schema instrumentation, capture the current contents of performance_schema.setup_instruments and performance_schema.setup_consumers. Following the investigation, restore the original configuration rather than assuming all wait instrumentation should be disabled. This avoids inadvertently disabling pre-existing monitoring required for operational observability.



3. High-Concurrency Taxonomy: Critical System Blocks


  


Critical System Blocks Summary


  
    
      Wait Event
      Production Root Cause &amp;amp; Impact
      Recommended Engineering Remediation
    
  
  
    
      wait/lock/metadata/sql/mdl
      An administrative DDL operation requests an exclusive Metadata Lock (MDL) but is blocked by a pre-existing transaction holding a shared metadata lock. Once the exclusive request is queued, subsequent shared lock requests may also queue behind it, creating an MDL convoy that can rapidly escalate into application-wide latency, connection-pool exhaustion, and request timeouts.
      Follow the MDL Cascade Anatomy workflow below. Inspect sys.schema_table_lock_waits; identify and remediate the root blocking transaction; enforce SET SESSION lock_wait_timeout = 5 for administrative sessions; and use online DDL options such as ALGORITHM=INPLACE and LOCK=NONE where supported by the operation and MySQL version.
    
    
      wait/synch/sxlock/innodb/dict_operation_lock
      Concurrent metadata-changing operations contend for InnoDB's internal Data Dictionary Operation Lock (dict_operation_lock), serializing DDL, tablespace operations, partition maintenance, and dictionary updates. Under sustained contention, dictionary access becomes a bottleneck, causing schema operations and dependent application threads to stall.
      Serialize schema deployments; stagger automated partition-maintenance jobs using advisory locks (GET_LOCK()); avoid high-frequency table creation and drop patterns; and eliminate runtime schema churn by pre-provisioning objects whenever possible.
    
    
      wait/synch/cond/sql/Group_replication_*
      Distributed synchronization stalls within Group Replication, typically involving transaction certification, flow control, applier backlogs, or group-communication delays. Common causes include asymmetric node capacity, network latency, storage-performance disparities, or overloaded replica members. These conditions can throttle commit throughput across the entire cluster.
      Validate inter-node network health; inspect performance_schema.replication_group_member_stats; review certification queues and flow-control activity; maintain uniform hardware sizing and storage performance across all cluster members; and investigate lagging members before increasing flow-control thresholds.
    
  




4. Deep-Dive: Metadata Lock (MDL) &amp;amp; Cascade Anatomy

The wait event wait/lock/metadata/sql/mdl is among the most disruptive contention patterns in high-concurrency MySQL environments because a single blocked DDL statement can trigger a metadata-lock convoy that stalls large portions of the application tier while the database appears otherwise healthy. Database CPU, memory, and storage utilization may remain relatively normal even as application response times, connection counts, and timeout rates escalate.

Official Documentation: MySQL 8.4 Reference Manual: Metadata Locking.

The Convoy Problem: Catalyst vs. Culprit

In major production incidents, the operational culprit is rarely the DDL statement itself. More commonly, an idle transaction acquires a shared metadata lock, and a subsequent DDL request introduces an exclusive lock request that blocks all future access.


  


Critical Operational Insights

1. The Sleeping Blocker Trap

When troubleshooting via SHOW PROCESSLIST, engineers frequently focus on connections executing long-running queries (Command: Query). In reality, the blocking session is often a connection in Command: Sleep with an open transaction that has not yet committed or rolled back. Common causes include:


  Connection pool leaks
  Disabled autocommit
  Application error paths
  Unclosed ORM transactions
  Long-running logical backups


The blocking SQL may have completed hours ago.

2. The 1-Year Default Timeout Trap (lock_wait_timeout)

Many engineers confuse innodb_lock_wait_timeout with lock_wait_timeout. The distinction is critical:


  
    
      Parameter
      Governs
    
  
  
    
      innodb_lock_wait_timeout
      InnoDB row-lock waits only (Default: 50 seconds)
    
    
      lock_wait_timeout
      Metadata lock (MDL) waits (Default: 31,536,000 seconds / 1 year)
    
  


As a result, an unattended migration can remain blocked for an extremely long period unless an explicit session timeout is configured. A blocked DDL statement can therefore hold a convoy open long enough to exhaust application connection pools and trigger widespread outages.

3. Foreign-Key MDL Cascades


  


4. Connection Pool Amplification

Modern application frameworks rely heavily on connection pools. A single sleeping transaction holding a metadata lock can eventually block hundreds of worker threads as requests accumulate behind a pending exclusive lock request. Consequently, MDL incidents often surface as:


  Connection pool exhaustion
  API timeouts
  Elevated active session counts
  Request queue growth
  Application-wide degradation


well before any database resource saturation becomes visible.

Rapid Diagnostic SQL

Direct Triage via sys.schema_table_lock_waits (Recommended):

-- Identify blocking and waiting sessions
SELECT object_schema,
       object_name,
       waiting_account,
       waiting_query,
       waiting_thread_id,
       waiting_pid,
       blocking_account,
       blocking_lock_type,
       blocking_lock_duration,
       blocking_thread_id,
       blocking_pid,
       sql_kill_blocking_connection
  FROM sys.schema_table_lock_waits;

Performance Schema Correlation:

-- Correlate metadata lock requests with transactions and threads
SELECT ml.OBJECT_TYPE,
       ml.OBJECT_SCHEMA,
       ml.OBJECT_NAME,
       ml.LOCK_TYPE,
       ml.LOCK_STATUS,
       t.PROCESSLIST_ID AS connection_id,
       t.PROCESSLIST_USER AS user,
       t.PROCESSLIST_HOST AS host,
       t.PROCESSLIST_COMMAND AS command,
       t.PROCESSLIST_STATE AS state,
       trx.trx_started,
       TIMESTAMPDIFF(SECOND, trx.trx_started, NOW()) AS trx_age_seconds,
       esc.SQL_TEXT AS last_executed_sql
  FROM performance_schema.metadata_locks ml
  JOIN performance_schema.threads t
    ON ml.OWNER_THREAD_ID = t.THREAD_ID
  LEFT JOIN information_schema.innodb_trx trx
    ON t.PROCESSLIST_ID = trx.trx_mysql_thread_id
  LEFT JOIN performance_schema.events_statements_current esc
    ON t.THREAD_ID = esc.THREAD_ID
 WHERE ml.OBJECT_NAME = 'orders'
 ORDER BY ml.LOCK_STATUS ASC,
          trx.trx_started ASC;

Production Deployment Guardrails

1. Enforce Fail-Fast DDL Sessions

SET SESSION lock_wait_timeout = 5;

ALTER TABLE orders
  ADD COLUMN promo_code VARCHAR(20),
  ALGORITHM = INPLACE,
  LOCK = NONE;

Use online DDL clauses such as ALGORITHM=INPLACE and LOCK=NONE only where supported by the operation and MySQL version.

2. Perform Pre-Flight Validation

Before executing schema changes, check for:

  Long-running transactions in information_schema.innodb_trx
  Existing metadata lock contention in sys.schema_table_lock_waits
  Sleeping sessions with open transactions
  Backup or maintenance jobs that may prolong lock lifetimes


3. Avoid Major DDL During Logical Backups

mysqldump --single-transaction generally avoids table locking for InnoDB tables, but it still creates a long-running consistent-read transaction. Large logical backup operations can:

  Extend transaction lifetimes
  Retain metadata dependencies longer than expected
  Increase DDL scheduling conflicts
  Amplify metadata-lock contention windows


Avoid scheduling large schema changes during extensive logical backup activity.


  Key Operational Takeaway: The session causing an MDL outage is rarely the ALTER TABLE statement itself. In most production incidents, the root cause is an overlooked sleeping transaction holding a shared metadata lock. The DDL statement merely becomes the catalyst that transforms an isolated transaction into a system-wide metadata-lock convoy.




5. Deep-Dive: Data Dictionary Latch Serialization (dict_operation_lock)

Contention on wait/synch/sxlock/innodb/dict_operation_lock indicates that threads are waiting on InnoDB's internal data dictionary synchronization mechanisms while executing metadata-related operations. Since MySQL 8.0, the data dictionary is transactional and integrated into InnoDB; however, certain DDL and dictionary-management activities still require serialized access to internal dictionary structures and tablespace metadata.

Official Documentation:

  MySQL 8.4 Reference Manual: MySQL Data Dictionary
  MySQL 8.4 Reference Manual: Atomic Data Dictionary Storage
  MySQL 8.4 Reference Manual: Performance Schema Wait Tables


Why This Wait Event Matters

Unlike row-lock contention, dictionary-lock contention can affect schema operations across the entire instance. Large-scale DDL operations, partition maintenance, tablespace modifications, and excessive object creation/drop activity may serialize at the dictionary layer, causing application-facing latency spikes even when CPU, memory, and storage metrics appear healthy.

Production Failure Scenario: Concurrent Maintenance Collision


  


As concurrent DDL operations accumulate, application threads may spend increasing amounts of time waiting on internal dictionary synchronization. The result is often elevated query latency, slower schema deployments, and prolonged maintenance windows.

Critical Operational Insights

1. Parallel DDL Is Frequently Self-Defeating
Executing multiple schema migrations in parallel rarely reduces overall deployment time. Instead, competing DDL operations often serialize internally, increasing contention and extending maintenance duration.

2. Partition Maintenance Is a Common Trigger
Automated partition-management jobs such as:

ALTER TABLE ... DROP PARTITION;
ALTER TABLE ... ADD PARTITION;
ALTER TABLE ... REORGANIZE PARTITION;

can introduce significant dictionary activity when executed concurrently with other administrative operations.

3. Runtime Schema Churn Does Not Scale
Applications that repeatedly execute:

CREATE TABLE ...;
DROP TABLE ...;
TRUNCATE TABLE ...;

during normal request processing place unnecessary pressure on the data dictionary subsystem and frequently become a bottleneck at scale.

4. Symptoms Are Often Misdiagnosed
Production teams frequently attribute the resulting slowdown to:

  Storage latency
  CPU saturation
  Metadata locks
  Replication lag

when the actual bottleneck is serialization within dictionary-management operations.

Diagnostic SQL

Identify Active Dictionary Waits:
SELECT THREAD_ID,
       EVENT_NAME,
       ROUND(TIMER_WAIT / 1000000000, 2) AS wait_ms,
       OPERATION,
       OBJECT_NAME
  FROM performance_schema.events_waits_current
 WHERE EVENT_NAME LIKE 'wait/synch/sxlock/innodb/dict_operation_lock%';

Inspect Active DDL Activity:
SELECT ID AS connection_id,
       USER,
       HOST,
       COMMAND,
       TIME AS runtime_sec,
       STATE,
       INFO AS ddl_statement
  FROM information_schema.processlist
 WHERE INFO REGEXP '^(ALTER|DROP|CREATE|TRUNCATE)'
   AND COMMAND &amp;lt;&amp;gt; 'Binlog Dump';

Identify Concurrent Schema Operations:
SELECT PROCESSLIST_ID,
       PROCESSLIST_USER,
       PROCESSLIST_HOST,
       PROCESSLIST_COMMAND,
       PROCESSLIST_STATE
  FROM performance_schema.threads
 WHERE PROCESSLIST_INFO REGEXP '^(ALTER|DROP|CREATE|TRUNCATE)';

Remediation &amp;amp; Operational Standards

1. Serialize Schema Deployments
Allow only one significant DDL operation per instance (or cluster writer) at a time. Avoid parallel schema deployment frameworks unless compatibility has been thoroughly validated.

2. Protect Automated Partition Maintenance
Use application-level locking to ensure partition-management jobs execute sequentially:

SELECT GET_LOCK('partition_maint_lock', 0) INTO @lock_acquired;

-- Continue only if @lock_acquired = 1

SELECT RELEASE_LOCK('partition_maint_lock');

Official Documentation: MySQL 8.4 Reference Manual: Locking Functions.

3. Eliminate Runtime Table Churn
Avoid creating and dropping permanent tables during normal request processing. Prefer:

  CREATE TEMPORARY TABLE
  Persistent staging tables
  Redis or application caches
  Queue-based processing architectures


4. Schedule Heavy DDL Windows
Large partition reorganizations, bulk table drops, archive cleanup operations, and schema deployments should be scheduled during dedicated maintenance windows whenever possible.

5. Audit Automation Pipelines
Review:

  Flyway/Liquibase pipelines
  Partition-rotation jobs
  Archival procedures
  Tenant-provisioning workflows

for overlapping DDL execution patterns.


  Key Operational Takeaway: wait/synch/sxlock/innodb/dict_operation_lock is typically not caused by user queries competing for data. It is usually a signal that multiple metadata-changing operations are competing for access to InnoDB's dictionary-management infrastructure. In high-concurrency environments, the most effective remediation is operational discipline: serialize DDL, coordinate partition maintenance, and eliminate unnecessary schema churn.




6. Deep-Dive: Group Replication Flow-Control Dynamics

In MySQL Group Replication and InnoDB Cluster deployments, the wait/synch/cond/sql/Group_replication_* wait-event family typically indicates distributed synchronization delays involving group communication, transaction certification, applier processing, or flow-control regulation. Under sustained write workloads, the cluster must continuously balance primary-node commit throughput against the processing capacity of all participating members.

Official Documentation:

  MySQL 8.4 Reference Manual: Group Replication Performance and Tuning
  MySQL 8.4 Reference Manual: Monitoring Group Replication


Production Failure Scenario: Asymmetric Node Throttling


  


Why This Wait Event Matters

Unlike traditional replication lag, Group Replication actively protects cluster consistency and member convergence. When a member falls behind:

  Certification queues increase.
  Applier queues accumulate.
  Flow Control activates.
  Primary commit throughput is throttled.
  Application write latency increases across the cluster.

Consequently, a single weak member can impact overall write performance even when the primary node remains healthy.

Diagnostic SQL

1. Inspect Certification Queues and Conflict Rates:
SELECT MEMBER_ID,
       COUNT_TRANSACTIONS_IN_QUEUE AS cert_queue_depth,
       COUNT_TRANSACTIONS_CHECKED AS total_checked,
       COUNT_CONFLICTS_DETECTED AS conflict_count,
       COUNT_TRANSACTIONS_ROWS_VALIDATING AS rows_validating
  FROM performance_schema.replication_group_member_stats;

Look for:

  Large queue depths
  Rapidly growing certification backlogs
  Elevated conflict rates
  Significant imbalance between members


2. Review Flow-Control Configuration:
SELECT VARIABLE_NAME,
       VARIABLE_VALUE
  FROM performance_schema.global_variables
 WHERE VARIABLE_NAME LIKE 'group_replication_flow_control%';

Verify:

  Flow-control mode
  Applier thresholds
  Certification thresholds
  Member-specific tuning deviations


Remediation &amp;amp; Architecture Standards

1. Maintain Balanced Cluster Sizing
Ensure all cluster members provide comparable:

  CPU capacity
  Memory allocation
  Storage performance
  Network throughput

A significantly slower member frequently becomes the trigger for cluster-wide Flow Control.

2. Investigate Before Raising Flow-Control Thresholds
Avoid using threshold increases as a first response. Instead:

  Identify lagging members
  Review queue growth trends
  Examine storage latency
  Review CPU utilization
  Validate network health

Only adjust flow-control thresholds following controlled performance testing.

3. Monitor Certification and Applier Queues
Regularly review performance_schema.replication_group_member_stats for:

  Queue growth
  Conflict detection rates
  Validation workload increases
  Sustained member imbalance

Early detection often prevents cluster-wide throttling events.

4. Audit Inter-Node Network Health
Validate:

  Latency consistency
  Packet loss
  Throughput capacity
  Network saturation


Standard Linux utilities assist in identifying communication bottlenecks:
ping &amp;lt;peer_ip&amp;gt;
iperf3 -c &amp;lt;peer_ip&amp;gt;

If Jumbo Frames are deployed, verify end-to-end MTU consistency across all network paths.


  Key Operational Takeaway: Group Replication flow-control waits are rarely caused by the primary writer alone. In most production incidents, the root cause is a lagging member that cannot certify or apply transactions at the cluster's incoming write rate. Flow Control intentionally sacrifices writer throughput to preserve cluster consistency, making the slowest member a critical determinant of overall cluster performance.




Summary &amp;amp; What's Next

Metadata-lock convoys, data-dictionary serialization, and Group Replication flow-control events represent three of the most disruptive forms of systemic contention in modern MySQL environments. Unlike conventional CPU, memory, or storage bottlenecks, these failures emerge from contention on shared internal coordination mechanisms and can affect large portions of the application stack while infrastructure metrics remain deceptively normal.


In Part 2: Storage Subsystems, Buffer Pool Latches, and Memory Spills
(Read Part 2),
the series shifts into storage-engine internals, including:



  Redo log checkpoint-age saturation and dynamic log sizing
  Doublewrite buffer behavior and storage durability characteristics
  Buffer pool mutex contention (buf_pool_mutex)
  Binary Log Group Commit optimization
  TempTable spills (FILE_TMPFILE)
  Undo purge lag and History List Length (HLL) growth


Continue to Part 2: Storage Subsystems &amp;amp; Memory Contention.7TKU66CE8V5W</description>
    <content:encoded><![CDATA[<p>High-concurrency database workloads rarely fail because of simple CPU or disk capacity limits. Instead, performance collapses occur when transactions serialize on shared internal database resources: metadata locks, data dictionary latches, group communication certification queues, and network buffers.</p>

<p>When contention occurs, external monitoring dashboards often show deceptive metrics such as near-zero CPU and disk activity alongside a massive surge in active connections and application timeouts. MySQL exposes these internal contention points through Performance Schema wait events.</p>

<p>This article—the first in a four-part deep dive—covers the foundational mechanics of Performance Schema wait telemetry, the governance of instrumentation overhead in high-throughput environments, and the diagnosis of Critical System Blocks: Metadata Locks (MDL), Data Dictionary latch serialization, and Group Replication Flow-Control stalls.</p>

<div>
  <strong>Operational Disclaimer:</strong> <em>The diagnostic queries, Performance Schema instrumentation configurations, engine parameter recommendations, stored routines, and architectural remediations presented across this series are provided for investigative engineering analysis and technical reference. Contention characteristics, wait-event profiles, and optimal parameter thresholds vary significantly depending on transactional access patterns, data volume, schema design, storage subsystem throughput, replication topology, cloud virtualization constraints (e.g., AWS Nitro, EBS bandwidth limits, Aurora multi-tenant architectures), and specific MySQL engine release baselines. Administrative overrides, dynamic parameter modifications, session terminations (KILL), and schema modifications should never be executed blindly against production clusters. All queries, DDL statements, and configuration changes must be reviewed, benchmarked, and validated in representative staging environments through established change-management procedures prior to production implementation.</em>
</div>

<h3>1. Why Wait Events Matter in High Concurrency</h3>

<p>In MySQL, every client query competes for finite shared resources. Rather than guessing whether a latency spike is driven by storage latency, table schema locks, or network transport, wait events quantify the exact time threads spend waiting:</p>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpX-5yt8p6q89Q8dbTZxkwskL9-ZfkRaySnfbrhK934gBxnqif915I8zkg441MRpE9PMK_FbONJp9IlUODYrnhtUO91CLoJE_eJZnh1lVudxVBHVNco3NZrc7KSweFr_DdBKz-VGhADws9cb83wXE4XSV_bcasIhZEs6uXKADn0p7MvqtleuauMROZYHs/s1600/image.png" alt="Why Wait Events Matter in High Concurrency">
</div>

<p><strong>Core Principle:</strong> Analyzing wait-event distributions and latency profiles transforms database troubleshooting from speculative trial-and-error into a deterministic diagnostic process.</p>

<p>Official documentation: <a href="https://dev.mysql.com/doc/refman/8.4/en/performance-schema-wait-tables.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Performance Schema Wait Event Tables</a>.</p>

<hr>

<h3>2. Performance Schema Prerequisites &amp; Overhead Governance</h3>

<p>By default, MySQL 8.4 LTS enables <code>performance_schema</code>, but detailed wait-event visibility is intentionally limited. Although numerous Performance Schema instruments are active, many wait-related instruments and the wait-event consumers (<code>events_waits_%</code>) remain disabled out of the box to reduce instrumentation overhead. Consequently, Performance Schema may initially expose statement-level and transaction-level diagnostics while providing little or no wait-event data until the relevant instruments and consumers are explicitly enabled.</p>

<h4>Instrumentation Enablement Matrix</h4>

<table>
  <thead>
    <tr>
      <th>Component</th>
      <th>Default State</th>
      <th>Operational Function</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>performance_schema</code></td>
      <td>ON</td>
      <td>Master telemetry engine toggle.</td>
    </tr>
    <tr>
      <td><code>setup_instruments (wait/%)</code></td>
      <td>SELECTIVELY ENABLED</td>
      <td>Low-level probes embedded in server source code (mutexes, rwlocks, file I/O, socket I/O, metadata locking, and other synchronization primitives). Some wait instruments are enabled by default, others are disabled.</td>
    </tr>
    <tr>
      <td><code>setup_consumers (events_waits_*)</code></td>
      <td>DISABLED</td>
      <td>In-memory aggregation tables and history buffers capturing wait-event telemetry. Wait-event consumers are disabled by default.</td>
    </tr>
  </tbody>
</table>

<h4>Measured Overhead: What Telemetry Costs</h4>

<p>Not all Performance Schema instruments impose the same overhead. High-frequency synchronization instruments such as mutexes and condition variables may execute millions of times per second on heavily concurrent OLTP systems. When instrumentation is enabled with <code>TIMED=YES</code>, additional high-resolution timing operations are performed for each instrumented event, increasing CPU consumption relative to untimed collection. The overhead estimates below represent illustrative field-observed ranges and should be interpreted as directional guidance rather than deterministic performance guarantees.</p>

<table>
  <thead>
    <tr>
      <th>Instrumentation Scope</th>
      <th>Typical CPU Overhead*</th>
      <th>Throughput Impact</th>
      <th>Workload Context</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Default Instruments (out-of-the-box)</td>
      <td>1–3%</td>
      <td>Negligible</td>
      <td>Standard production baseline</td>
    </tr>
    <tr>
      <td>Targeted (<code>wait/io/%</code>, <code>wait/lock/%</code>)</td>
      <td>3–8%</td>
      <td>Moderate</td>
      <td>Recommended for production triage</td>
    </tr>
    <tr>
      <td>All <code>wait/synch/mutex</code> with <code>TIMED=YES</code></td>
      <td>10–25%</td>
      <td>High</td>
      <td>Heavy concurrent OLTP workloads</td>
    </tr>
    <tr>
      <td>Blanket <code>wait/%</code> with <code>TIMED=YES</code></td>
      <td>15–40%</td>
      <td>Severe</td>
      <td>High-concurrency throughput regression</td>
    </tr>
    <tr>
      <td>Instruments <code>ENABLED=YES</code>, <code>TIMED=NO</code></td>
      <td>2–5%</td>
      <td>Minimal</td>
      <td>Frequency and contention counts only</td>
    </tr>
  </tbody>
</table>

<p><strong>* Performance Impact Disclaimer:</strong> The overhead percentages shown above are illustrative field-observed ranges derived from benchmarking exercises, vendor guidance, and production support experience across diverse environments. Actual impact may vary substantially depending on workload characteristics, transaction concurrency, CPU architecture, storage latency, enabled consumers, instrumentation scope, MySQL version, and cloud platform implementation details. These figures should be interpreted as directional guidance rather than deterministic performance guarantees.</p>

<p><strong>Operational Guideline:</strong> The <code>TIMED=YES</code> attribute on high-frequency synchronization instruments (particularly mutexes and condition variables) is often the largest contributor to instrumentation overhead. When investigating contention frequency, hotspot identification, or lock distribution rather than precise latency measurements, consider using <code>TIMED='NO'</code> to reduce timing-related overhead while still collecting event counts and contention statistics.</p>

<h4>Dynamic Runtime Activation (Diagnostic Session)</h4>

<p>To enable wait instrumentation dynamically during an incident window without restarting the server:</p>

<pre class="language-sql"><code class="language-sql">-- Dynamic Runtime Activation (Diagnostic Session)
-- Enable targeted wait instrumentation during an incident window without requiring a MySQL restart.

-- 1. Enable targeted wait instruments
UPDATE performance_schema.setup_instruments
   SET ENABLED = 'YES',
       TIMED   = 'YES'
 WHERE NAME LIKE 'wait/io/%'
    OR NAME LIKE 'wait/lock/%'
    OR NAME LIKE 'wait/synch/mutex/innodb/%';

-- 2. Enable wait-event consumers
UPDATE performance_schema.setup_consumers
   SET ENABLED = 'YES'
 WHERE NAME IN (
       'events_waits_current',
       'events_waits_history',
       'events_waits_history_long'
 );

-- 3. Verify enabled instrumentation
SELECT NAME,
       ENABLED,
       TIMED
  FROM performance_schema.setup_instruments
 WHERE NAME LIKE 'wait/%'
   AND ENABLED = 'YES'
 ORDER BY NAME
 LIMIT 20;

-- 4. Verify enabled consumers
SELECT NAME,
       ENABLED
  FROM performance_schema.setup_consumers
 WHERE NAME LIKE 'events_waits%';</code></pre>

<p><strong>Operational Note:</strong> Instrumentation changes affect only future wait events and do not retroactively populate historical data. Enable the required instruments and consumers, allow representative workload activity to execute, and then analyze wait-event telemetry.</p>

<p><strong>Production Guidance:</strong> When investigating contention frequency rather than precise latency measurements, consider enabling instruments with <code>TIMED='NO'</code> first. This significantly reduces instrumentation overhead while still exposing contention hotspots and event frequency distributions.</p>

<h4>Post-Investigation Cleanup Command</h4>

<p>Always return the instance to safe telemetry baselines following an investigation:</p>

<pre class="language-sql"><code class="language-sql">-- Post-Investigation Cleanup
-- Revert any instrumentation that was enabled specifically for the investigation.
-- Ideally, restore the pre-investigation configuration captured before enabling additional instruments and consumers.

-- Example: Disable instrumentation enabled for the investigation
UPDATE performance_schema.setup_instruments
   SET ENABLED = 'NO',
       TIMED   = 'NO'
 WHERE NAME LIKE 'wait/io/%'
    OR NAME LIKE 'wait/lock/%'
    OR NAME LIKE 'wait/synch/mutex/innodb/%';

-- Disable wait-event consumers
UPDATE performance_schema.setup_consumers
   SET ENABLED = 'NO'
 WHERE NAME IN (
       'events_waits_current',
       'events_waits_history',
       'events_waits_history_long'
 );</code></pre>

<p><strong>Operational Guideline:</strong> Before enabling additional Performance Schema instrumentation, capture the current contents of <code>performance_schema.setup_instruments</code> and <code>performance_schema.setup_consumers</code>. Following the investigation, restore the original configuration rather than assuming all wait instrumentation should be disabled. This avoids inadvertently disabling pre-existing monitoring required for operational observability.</p>

<hr>

<h3>3. High-Concurrency Taxonomy: Critical System Blocks</h3>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-3hH7NOugdlXicqNoFUjMCt0pASTU-Kr-I8jNhhvXCmPC_8o5VNkgwXrf5K-l2LQbg-riCwO-1kb50upqavHLnMlDgiuYqMr_HlGiBCPNmxBsCNDccq0m9qseCa9mErsyIP7jAID5oOMt68UIFqV0O2rVcbWNu234rT_Dr4zuWJnBIKJxc_EHGTg0ysA/w478-h478/image.png" alt="Critical System Blocks Architecture">
</div>

<h4>Critical System Blocks Summary</h4>

<table>
  <thead>
    <tr>
      <th>Wait Event</th>
      <th>Production Root Cause &amp; Impact</th>
      <th>Recommended Engineering Remediation</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>wait/lock/metadata/sql/mdl</code></td>
      <td>An administrative DDL operation requests an exclusive Metadata Lock (MDL) but is blocked by a pre-existing transaction holding a shared metadata lock. Once the exclusive request is queued, subsequent shared lock requests may also queue behind it, creating an MDL convoy that can rapidly escalate into application-wide latency, connection-pool exhaustion, and request timeouts.</td>
      <td>Follow the MDL Cascade Anatomy workflow below. Inspect <code>sys.schema_table_lock_waits</code>; identify and remediate the root blocking transaction; enforce <code>SET SESSION lock_wait_timeout = 5</code> for administrative sessions; and use online DDL options such as <code>ALGORITHM=INPLACE</code> and <code>LOCK=NONE</code> where supported by the operation and MySQL version.</td>
    </tr>
    <tr>
      <td><code>wait/synch/sxlock/innodb/dict_operation_lock</code></td>
      <td>Concurrent metadata-changing operations contend for InnoDB's internal Data Dictionary Operation Lock (<code>dict_operation_lock</code>), serializing DDL, tablespace operations, partition maintenance, and dictionary updates. Under sustained contention, dictionary access becomes a bottleneck, causing schema operations and dependent application threads to stall.</td>
      <td>Serialize schema deployments; stagger automated partition-maintenance jobs using advisory locks (<code>GET_LOCK()</code>); avoid high-frequency table creation and drop patterns; and eliminate runtime schema churn by pre-provisioning objects whenever possible.</td>
    </tr>
    <tr>
      <td><code>wait/synch/cond/sql/Group_replication_*</code></td>
      <td>Distributed synchronization stalls within Group Replication, typically involving transaction certification, flow control, applier backlogs, or group-communication delays. Common causes include asymmetric node capacity, network latency, storage-performance disparities, or overloaded replica members. These conditions can throttle commit throughput across the entire cluster.</td>
      <td>Validate inter-node network health; inspect <code>performance_schema.replication_group_member_stats</code>; review certification queues and flow-control activity; maintain uniform hardware sizing and storage performance across all cluster members; and investigate lagging members before increasing flow-control thresholds.</td>
    </tr>
  </tbody>
</table>

<hr>

<h3>4. Deep-Dive: Metadata Lock (MDL) &amp; Cascade Anatomy</h3>

<p>The wait event <code>wait/lock/metadata/sql/mdl</code> is among the most disruptive contention patterns in high-concurrency MySQL environments because a single blocked DDL statement can trigger a metadata-lock convoy that stalls large portions of the application tier while the database appears otherwise healthy. Database CPU, memory, and storage utilization may remain relatively normal even as application response times, connection counts, and timeout rates escalate.</p>

<p>Official Documentation: <a href="https://dev.mysql.com/doc/refman/8.4/en/metadata-locking.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Metadata Locking</a>.</p>

<h4>The Convoy Problem: Catalyst vs. Culprit</h4>

<p>In major production incidents, the operational culprit is rarely the DDL statement itself. More commonly, an idle transaction acquires a shared metadata lock, and a subsequent DDL request introduces an exclusive lock request that blocks all future access.</p>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixSi9b8JfOxFRAZTm_unDAiPBRvsChzNmCz8dlqThzs7IM0xu7OwLbVdtmJgbiCqIAZZBhrmrM3XU40WDzys42QsydTqHtfcXy5YS4V7hfOmbmMfDwmbfJzo3bPBfqFdraWE0Urtt7L3VQfrvi_0Mh3UR98jPKc-jw3kUFI1srajaB1_cyJnIqkDO9tfA/w461-h693/image.png" alt="Metadata Lock Convoy Problem">
</div>

<h4>Critical Operational Insights</h4>

<p><strong>1. The Sleeping Blocker Trap</strong></p>

<p>When troubleshooting via <code>SHOW PROCESSLIST</code>, engineers frequently focus on connections executing long-running queries (<code>Command: Query</code>). In reality, the blocking session is often a connection in <code>Command: Sleep</code> with an open transaction that has not yet committed or rolled back. Common causes include:</p>

<ul>
  <li>Connection pool leaks</li>
  <li>Disabled autocommit</li>
  <li>Application error paths</li>
  <li>Unclosed ORM transactions</li>
  <li>Long-running logical backups</li>
</ul>

<p>The blocking SQL may have completed hours ago.</p>

<p><strong>2. The 1-Year Default Timeout Trap (<code>lock_wait_timeout</code>)</strong></p>

<p>Many engineers confuse <code>innodb_lock_wait_timeout</code> with <code>lock_wait_timeout</code>. The distinction is critical:</p>

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Governs</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>innodb_lock_wait_timeout</code></td>
      <td>InnoDB row-lock waits only (Default: <strong>50 seconds</strong>)</td>
    </tr>
    <tr>
      <td><code>lock_wait_timeout</code></td>
      <td>Metadata lock (MDL) waits (Default: <strong>31,536,000 seconds / 1 year</strong>)</td>
    </tr>
  </tbody>
</table>

<p>As a result, an unattended migration can remain blocked for an extremely long period unless an explicit session timeout is configured. A blocked DDL statement can therefore hold a convoy open long enough to exhaust application connection pools and trigger widespread outages.</p>

<p><strong>3. Foreign-Key MDL Cascades</strong></p>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLQwoWvnTN-0r0AVjF3E03BW1TdPdAjtfEvkQSNiYhpqhoUNy81KoQYgo6-rzVUFbf3knCpCWg2wpoZc8qjsqMGN9yWa3xGC0IE8T-u6Eey6bpGAgdz2GUQ60D7Bz84I3Tao9HaIY_GMPSbfRfHTbI_qsqc0vGciHxGD_LcSNtLw2jUKSJdRo6iyzaozw/s1600/image.png" alt="Foreign Key MDL Cascades">
</div>

<p><strong>4. Connection Pool Amplification</strong></p>

<p>Modern application frameworks rely heavily on connection pools. A single sleeping transaction holding a metadata lock can eventually block hundreds of worker threads as requests accumulate behind a pending exclusive lock request. Consequently, MDL incidents often surface as:</p>

<ul>
  <li>Connection pool exhaustion</li>
  <li>API timeouts</li>
  <li>Elevated active session counts</li>
  <li>Request queue growth</li>
  <li>Application-wide degradation</li>
</ul>

<p>well before any database resource saturation becomes visible.</p>

<h4>Rapid Diagnostic SQL</h4>

<p><strong>Direct Triage via <code>sys.schema_table_lock_waits</code> (Recommended):</strong></p>

<pre class="language-sql"><code class="language-sql">-- Identify blocking and waiting sessions
SELECT object_schema,
       object_name,
       waiting_account,
       waiting_query,
       waiting_thread_id,
       waiting_pid,
       blocking_account,
       blocking_lock_type,
       blocking_lock_duration,
       blocking_thread_id,
       blocking_pid,
       sql_kill_blocking_connection
  FROM sys.schema_table_lock_waits;</code></pre>

<p><strong>Performance Schema Correlation:</strong></p>

<pre class="language-sql"><code class="language-sql">-- Correlate metadata lock requests with transactions and threads
SELECT ml.OBJECT_TYPE,
       ml.OBJECT_SCHEMA,
       ml.OBJECT_NAME,
       ml.LOCK_TYPE,
       ml.LOCK_STATUS,
       t.PROCESSLIST_ID AS connection_id,
       t.PROCESSLIST_USER AS user,
       t.PROCESSLIST_HOST AS host,
       t.PROCESSLIST_COMMAND AS command,
       t.PROCESSLIST_STATE AS state,
       trx.trx_started,
       TIMESTAMPDIFF(SECOND, trx.trx_started, NOW()) AS trx_age_seconds,
       esc.SQL_TEXT AS last_executed_sql
  FROM performance_schema.metadata_locks ml
  JOIN performance_schema.threads t
    ON ml.OWNER_THREAD_ID = t.THREAD_ID
  LEFT JOIN information_schema.innodb_trx trx
    ON t.PROCESSLIST_ID = trx.trx_mysql_thread_id
  LEFT JOIN performance_schema.events_statements_current esc
    ON t.THREAD_ID = esc.THREAD_ID
 WHERE ml.OBJECT_NAME = 'orders'
 ORDER BY ml.LOCK_STATUS ASC,
          trx.trx_started ASC;</code></pre>

<h4>Production Deployment Guardrails</h4>

<p><strong>1. Enforce Fail-Fast DDL Sessions</strong></p>

<pre class="language-sql"><code class="language-sql">SET SESSION lock_wait_timeout = 5;

ALTER TABLE orders
  ADD COLUMN promo_code VARCHAR(20),
  ALGORITHM = INPLACE,
  LOCK = NONE;</code></pre>

<p>Use online DDL clauses such as <code>ALGORITHM=INPLACE</code> and <code>LOCK=NONE</code> only where supported by the operation and MySQL version.</p>

<p><strong>2. Perform Pre-Flight Validation</strong></p>

<p>Before executing schema changes, check for:</p>
<ul>
  <li>Long-running transactions in <code>information_schema.innodb_trx</code></li>
  <li>Existing metadata lock contention in <code>sys.schema_table_lock_waits</code></li>
  <li>Sleeping sessions with open transactions</li>
  <li>Backup or maintenance jobs that may prolong lock lifetimes</li>
</ul>

<p><strong>3. Avoid Major DDL During Logical Backups</strong></p>

<p><code>mysqldump --single-transaction</code> generally avoids table locking for InnoDB tables, but it still creates a long-running consistent-read transaction. Large logical backup operations can:</p>
<ul>
  <li>Extend transaction lifetimes</li>
  <li>Retain metadata dependencies longer than expected</li>
  <li>Increase DDL scheduling conflicts</li>
  <li>Amplify metadata-lock contention windows</li>
</ul>

<p>Avoid scheduling large schema changes during extensive logical backup activity.</p>

<div>
  <strong>Key Operational Takeaway:</strong> The session causing an MDL outage is rarely the <code>ALTER TABLE</code> statement itself. In most production incidents, the root cause is an overlooked sleeping transaction holding a shared metadata lock. The DDL statement merely becomes the catalyst that transforms an isolated transaction into a system-wide metadata-lock convoy.
</div>

<hr>

<h3>5. Deep-Dive: Data Dictionary Latch Serialization (<code>dict_operation_lock</code>)</h3>

<p>Contention on <code>wait/synch/sxlock/innodb/dict_operation_lock</code> indicates that threads are waiting on InnoDB's internal data dictionary synchronization mechanisms while executing metadata-related operations. Since MySQL 8.0, the data dictionary is transactional and integrated into InnoDB; however, certain DDL and dictionary-management activities still require serialized access to internal dictionary structures and tablespace metadata.</p>

<p>Official Documentation:</p>
<ul>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/data-dictionary.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: MySQL Data Dictionary</a></li>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/data-dictionary-usage-differences.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Atomic Data Dictionary Storage</a></li>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/performance-schema-wait-tables.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Performance Schema Wait Tables</a></li>
</ul>

<h4>Why This Wait Event Matters</h4>

<p>Unlike row-lock contention, dictionary-lock contention can affect schema operations across the entire instance. Large-scale DDL operations, partition maintenance, tablespace modifications, and excessive object creation/drop activity may serialize at the dictionary layer, causing application-facing latency spikes even when CPU, memory, and storage metrics appear healthy.</p>

<h4>Production Failure Scenario: Concurrent Maintenance Collision</h4>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgr42SYeBNBna2LuBKjc7h0VbaXkjWYSzipK6DGPJvUdwFBZgxO_h8RpuVLGDWdhGqe3bU0re8enwIowVpdhIQ4NnmkBUtUVt5MbdRJHx4e8F1p3QcEgYCHmLCyXvT8_8z7t0tz7ZgAnE62iF_2RelYTFUGZqBrSUCUyGunG1TN_ae5sbl1anxGRoSEPWs/w485-h729/image.png" alt="Concurrent Maintenance Collision">
</div>

<p>As concurrent DDL operations accumulate, application threads may spend increasing amounts of time waiting on internal dictionary synchronization. The result is often elevated query latency, slower schema deployments, and prolonged maintenance windows.</p>

<h4>Critical Operational Insights</h4>

<p><strong>1. Parallel DDL Is Frequently Self-Defeating</strong><br>
Executing multiple schema migrations in parallel rarely reduces overall deployment time. Instead, competing DDL operations often serialize internally, increasing contention and extending maintenance duration.</p>

<p><strong>2. Partition Maintenance Is a Common Trigger</strong><br>
Automated partition-management jobs such as:</p>

<pre class="language-sql"><code class="language-sql">ALTER TABLE ... DROP PARTITION;
ALTER TABLE ... ADD PARTITION;
ALTER TABLE ... REORGANIZE PARTITION;</code></pre>

<p>can introduce significant dictionary activity when executed concurrently with other administrative operations.</p>

<p><strong>3. Runtime Schema Churn Does Not Scale</strong><br>
Applications that repeatedly execute:</p>

<pre class="language-sql"><code class="language-sql">CREATE TABLE ...;
DROP TABLE ...;
TRUNCATE TABLE ...;</code></pre>

<p>during normal request processing place unnecessary pressure on the data dictionary subsystem and frequently become a bottleneck at scale.</p>

<p><strong>4. Symptoms Are Often Misdiagnosed</strong><br>
Production teams frequently attribute the resulting slowdown to:</p>
<ul>
  <li>Storage latency</li>
  <li>CPU saturation</li>
  <li>Metadata locks</li>
  <li>Replication lag</li>
</ul>
<p>when the actual bottleneck is serialization within dictionary-management operations.</p>

<h4>Diagnostic SQL</h4>

<p><strong>Identify Active Dictionary Waits:</strong></p>
<pre class="language-sql"><code class="language-sql">SELECT THREAD_ID,
       EVENT_NAME,
       ROUND(TIMER_WAIT / 1000000000, 2) AS wait_ms,
       OPERATION,
       OBJECT_NAME
  FROM performance_schema.events_waits_current
 WHERE EVENT_NAME LIKE 'wait/synch/sxlock/innodb/dict_operation_lock%';</code></pre>

<p><strong>Inspect Active DDL Activity:</strong></p>
<pre class="language-sql"><code class="language-sql">SELECT ID AS connection_id,
       USER,
       HOST,
       COMMAND,
       TIME AS runtime_sec,
       STATE,
       INFO AS ddl_statement
  FROM information_schema.processlist
 WHERE INFO REGEXP '^(ALTER|DROP|CREATE|TRUNCATE)'
   AND COMMAND &lt;&gt; 'Binlog Dump';</code></pre>

<p><strong>Identify Concurrent Schema Operations:</strong></p>
<pre class="language-sql"><code class="language-sql">SELECT PROCESSLIST_ID,
       PROCESSLIST_USER,
       PROCESSLIST_HOST,
       PROCESSLIST_COMMAND,
       PROCESSLIST_STATE
  FROM performance_schema.threads
 WHERE PROCESSLIST_INFO REGEXP '^(ALTER|DROP|CREATE|TRUNCATE)';</code></pre>

<h4>Remediation &amp; Operational Standards</h4>

<p><strong>1. Serialize Schema Deployments</strong><br>
Allow only one significant DDL operation per instance (or cluster writer) at a time. Avoid parallel schema deployment frameworks unless compatibility has been thoroughly validated.</p>

<p><strong>2. Protect Automated Partition Maintenance</strong><br>
Use application-level locking to ensure partition-management jobs execute sequentially:</p>

<pre class="language-sql"><code class="language-sql">SELECT GET_LOCK('partition_maint_lock', 0) INTO @lock_acquired;

-- Continue only if @lock_acquired = 1

SELECT RELEASE_LOCK('partition_maint_lock');</code></pre>

<p>Official Documentation: <a href="https://dev.mysql.com/doc/refman/8.4/en/locking-functions.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Locking Functions</a>.</p>

<p><strong>3. Eliminate Runtime Table Churn</strong><br>
Avoid creating and dropping permanent tables during normal request processing. Prefer:</p>
<ul>
  <li><code>CREATE TEMPORARY TABLE</code></li>
  <li>Persistent staging tables</li>
  <li>Redis or application caches</li>
  <li>Queue-based processing architectures</li>
</ul>

<p><strong>4. Schedule Heavy DDL Windows</strong><br>
Large partition reorganizations, bulk table drops, archive cleanup operations, and schema deployments should be scheduled during dedicated maintenance windows whenever possible.</p>

<p><strong>5. Audit Automation Pipelines</strong><br>
Review:</p>
<ul>
  <li>Flyway/Liquibase pipelines</li>
  <li>Partition-rotation jobs</li>
  <li>Archival procedures</li>
  <li>Tenant-provisioning workflows</li>
</ul>
<p>for overlapping DDL execution patterns.</p>

<div>
  <strong>Key Operational Takeaway:</strong> <code>wait/synch/sxlock/innodb/dict_operation_lock</code> is typically not caused by user queries competing for data. It is usually a signal that multiple metadata-changing operations are competing for access to InnoDB's dictionary-management infrastructure. In high-concurrency environments, the most effective remediation is operational discipline: serialize DDL, coordinate partition maintenance, and eliminate unnecessary schema churn.
</div>

<hr>

<h3>6. Deep-Dive: Group Replication Flow-Control Dynamics</h3>

<p>In MySQL Group Replication and InnoDB Cluster deployments, the <code>wait/synch/cond/sql/Group_replication_*</code> wait-event family typically indicates distributed synchronization delays involving group communication, transaction certification, applier processing, or flow-control regulation. Under sustained write workloads, the cluster must continuously balance primary-node commit throughput against the processing capacity of all participating members.</p>

<p>Official Documentation:</p>
<ul>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/group-replication-performance.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Group Replication Performance and Tuning</a></li>
  <li><a href="https://dev.mysql.com/doc/refman/8.4/en/group-replication-replication-group-member-stats.html" target="_blank" rel="noopener noreferrer">MySQL 8.4 Reference Manual: Monitoring Group Replication</a></li>
</ul>

<h4>Production Failure Scenario: Asymmetric Node Throttling</h4>

<div>
  <img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdhXgmg9Xyobd31cs9__XKYNvL2lByfiazlyoiJKcMJGjyboAi8rp2Hp4caIWtx16ceoPE8OUbpjmH_TGyz34N9RhJDai3maHPeLT69Xnoz16ca_yXV6wzotvjTv9CTGz5xVFBO7BoqWOau8cOF87xWAZMMV1cqIkwaMxJOhLmICqINL1HoaQzcoEIL54/s1600/image.png" alt="Asymmetric Node Throttling">
</div>

<h4>Why This Wait Event Matters</h4>

<p>Unlike traditional replication lag, Group Replication actively protects cluster consistency and member convergence. When a member falls behind:</p>
<ol>
  <li>Certification queues increase.</li>
  <li>Applier queues accumulate.</li>
  <li>Flow Control activates.</li>
  <li>Primary commit throughput is throttled.</li>
  <li>Application write latency increases across the cluster.</li>
</ol>
<p>Consequently, a single weak member can impact overall write performance even when the primary node remains healthy.</p>

<h4>Diagnostic SQL</h4>

<p><strong>1. Inspect Certification Queues and Conflict Rates:</strong></p>
<pre class="language-sql"><code class="language-sql">SELECT MEMBER_ID,
       COUNT_TRANSACTIONS_IN_QUEUE AS cert_queue_depth,
       COUNT_TRANSACTIONS_CHECKED AS total_checked,
       COUNT_CONFLICTS_DETECTED AS conflict_count,
       COUNT_TRANSACTIONS_ROWS_VALIDATING AS rows_validating
  FROM performance_schema.replication_group_member_stats;</code></pre>

<p>Look for:</p>
<ul>
  <li>Large queue depths</li>
  <li>Rapidly growing certification backlogs</li>
  <li>Elevated conflict rates</li>
  <li>Significant imbalance between members</li>
</ul>

<p><strong>2. Review Flow-Control Configuration:</strong></p>
<pre class="language-sql"><code class="language-sql">SELECT VARIABLE_NAME,
       VARIABLE_VALUE
  FROM performance_schema.global_variables
 WHERE VARIABLE_NAME LIKE 'group_replication_flow_control%';</code></pre>

<p>Verify:</p>
<ul>
  <li>Flow-control mode</li>
  <li>Applier thresholds</li>
  <li>Certification thresholds</li>
  <li>Member-specific tuning deviations</li>
</ul>

<h4>Remediation &amp; Architecture Standards</h4>

<p><strong>1. Maintain Balanced Cluster Sizing</strong><br>
Ensure all cluster members provide comparable:</p>
<ul>
  <li>CPU capacity</li>
  <li>Memory allocation</li>
  <li>Storage performance</li>
  <li>Network throughput</li>
</ul>
<p>A significantly slower member frequently becomes the trigger for cluster-wide Flow Control.</p>

<p><strong>2. Investigate Before Raising Flow-Control Thresholds</strong><br>
Avoid using threshold increases as a first response. Instead:</p>
<ul>
  <li>Identify lagging members</li>
  <li>Review queue growth trends</li>
  <li>Examine storage latency</li>
  <li>Review CPU utilization</li>
  <li>Validate network health</li>
</ul>
<p>Only adjust flow-control thresholds following controlled performance testing.</p>

<p><strong>3. Monitor Certification and Applier Queues</strong><br>
Regularly review <code>performance_schema.replication_group_member_stats</code> for:</p>
<ul>
  <li>Queue growth</li>
  <li>Conflict detection rates</li>
  <li>Validation workload increases</li>
  <li>Sustained member imbalance</li>
</ul>
<p>Early detection often prevents cluster-wide throttling events.</p>

<p><strong>4. Audit Inter-Node Network Health</strong><br>
Validate:</p>
<ul>
  <li>Latency consistency</li>
  <li>Packet loss</li>
  <li>Throughput capacity</li>
  <li>Network saturation</li>
</ul>

<p>Standard Linux utilities assist in identifying communication bottlenecks:</p>
<pre class="language-bash"><code class="language-bash">ping &lt;peer_ip&gt;
iperf3 -c &lt;peer_ip&gt;</code></pre>

<p>If Jumbo Frames are deployed, verify end-to-end MTU consistency across all network paths.</p>

<div>
  <strong>Key Operational Takeaway:</strong> Group Replication flow-control waits are rarely caused by the primary writer alone. In most production incidents, the root cause is a lagging member that cannot certify or apply transactions at the cluster's incoming write rate. Flow Control intentionally sacrifices writer throughput to preserve cluster consistency, making the slowest member a critical determinant of overall cluster performance.
</div>

<hr>

<h3>Summary &amp; What's Next</h3>

<p>Metadata-lock convoys, data-dictionary serialization, and Group Replication flow-control events represent three of the most disruptive forms of systemic contention in modern MySQL environments. Unlike conventional CPU, memory, or storage bottlenecks, these failures emerge from contention on shared internal coordination mechanisms and can affect large portions of the application stack while infrastructure metrics remain deceptively normal.</p>

<p>
In <strong>Part 2: Storage Subsystems, Buffer Pool Latches, and Memory Spills</strong>
(<a href="https://ushastry.blogspot.com/2026/11/diagnosing-mysql-at-scale-part-2.html" target="_blank" rel="noopener noreferrer">Read Part 2</a>),
the series shifts into storage-engine internals, including:
</p>

<ul>
  <li>Redo log checkpoint-age saturation and dynamic log sizing</li>
  <li>Doublewrite buffer behavior and storage durability characteristics</li>
  <li>Buffer pool mutex contention (<code>buf_pool_mutex</code>)</li>
  <li>Binary Log Group Commit optimization</li>
  <li>TempTable spills (<code>FILE_TMPFILE</code>)</li>
  <li>Undo purge lag and History List Length (HLL) growth</li>
</ul>

<p>Continue to <strong><a href="https://ushastry.blogspot.com/2026/11/diagnosing-mysql-at-scale-part-2.html" target="_blank" rel="noopener noreferrer">Part 2: Storage Subsystems &amp; Memory Contention</a></strong>.</p><div class="blogger-post-footer">7TKU66CE8V5W</div>]]></content:encoded>
    <pubDate>Thu, 05 Nov 2026 18:38:38 +0000</pubDate>
    <dc:creator>Umesh Shastry</dc:creator>
    <category>Diagnostics</category>
    <category>Group Replication</category>
    <category>Innodb</category>
    <category>Metadata Lock</category>
    <category>Performance Schema</category>
  </item>

  <item>
    <title>WooCommerce Analytics with DuckDB</title>
    <guid isPermaLink="false">https://www.percona.com/?p=55557</guid>
    <link>https://www.percona.com/blog/woocommerce-analytics-with-duckdb/</link>
    <description>MySQL and DuckDB
Unlike row-oriented databases, which store data for a single record sequentially on disk, favoring single-row lookups, columnar databases store all the values from a column contiguously in storage, allowing only the selected column to be read from disk.
The default MySQL engine is InnoDB, which is ACID-compliant and great for OLTP (Online Transaction Processing) workloads. That said, OLAP (Online Analytical Processing) queries are built differently; they scan, group, and summarize millions or billions of rows to produce reports. In these cases, the InnoDB engine can be slow to process such queries. This has been a debt in MySQL over the years; it was common to setup ETL/CDC pipelines to export or stream the data to another engine, usually a columnar database system, to run reporting and BI queries on it.
DuckDB is a portable analytical database system, designed to be fast and reliable, similar to SQLite.
Recently, various vendors (MariaDB, Alibaba) started including DuckDB inside MySQL as a storage engine. A table declared with ENGINE=DuckDB stores data in columnar format, and queries are handled by the DuckDB engine.
This is useful in several ways; the main one is to use MySQL asynchronous replication to stream data from the OLTP, using the InnoDB engine, to a Replica where tables are in DuckDB, thereby offloading the OLAP workload to the replica. To showcase this, we thought that the best example was an on-line store which naturally requires both: short, fast, scalable transactions for the store frontend, and analytics for the store’s reporting and Business Intelligence, and what better than the free and open-source software that serves between 20 and 40% of the world’s e-commerce websites: WooCommerce.
WooCommerce
There are around 4M WooCommerce stores worldwide, the most of any e-commerce platform, handling between 20 and 39% of all e-commerce sites, and they use MySQL as their underlying database.
One of the reasons stores have been adopting computers for over 50 years, is the value of having accumulated information about your business’ transactions. This data collection allows business owners to understand trends and find new opportunities, by turning raw data into metrics that marketing and sales departments can act upon. This process is based on the long-term accumulation of large amounts of data and metadata, and that accumulation is often incompatible with transactional/row-oriented storage engines.
WooCommerce has a built-in analytics module to create those reports that provide metrics, so it is the perfect product to test the performance difference between InnoDB and DuckDB.
WooCommerce analytics does not run its reports against live order tables because querying these tables would be too slow. Instead, it keeps a set of aggregations in a number of denormalized “lookup” tables, which are optimized for this purpose. This is a limitation when running reports with InnoDB, as querying the raw tables would kill performance, and thus the reports are limited to what you can get from these lookup tables.
The following lookup tables are created for WooCommerce Analytics:mysql&amp;gt; SHOW TABLES LIKE '%wc_%lookup'; SHOW TABLES LIKE '%wc_order_stats';
+-------------------------------------+
| Tables_in_woocommerce (%wc_%lookup) |
+-------------------------------------+
| wp_wc_category_lookup               |
| wp_wc_customer_lookup               |
| wp_wc_order_coupon_lookup           |
| wp_wc_order_product_lookup          |
| wp_wc_order_tax_lookup              |
| wp_wc_product_attributes_lookup     |
| wp_wc_product_meta_lookup           |
+-------------------------------------+
7 rows in set (0.00 sec)

+-----------------------------------------+
| Tables_in_woocommerce (%wc_order_stats) |
+-----------------------------------------+
| wp_wc_order_stats                       |
+-----------------------------------------+
1 row in set (0.00 sec)The wp_wc_order_stats table is the spine of the revenue and orders reports; it contains one row per order, including dates, status, customer, number of items sold, and the monetary totals.
In the store we’re using, the wc_order_stats table contains 400000 rows, and there are 5000 products. This data was loaded using the WooCommerce Smooth Generator plugin.
The lookup tables are derived data, so WooCommerce has to keep them in sync with the orders they summarize, which it does in the background (which means you have to run a scheduled process that is what keeps these lookup tables updated).
MySQL Deployment
We deployed a Source/Replica MySQL topology with both databases on the same host, each with its respective Docker image.
Source
The Source runs Percona Server for MySQL 8.0.46, which we chose to match the replica’s major version.
WooCommerce tables use the MySQL default engine, which is InnoDB.
We configured an InnoDB Buffer Pool of 4 GiB:[mysqld]
innodb_buffer_pool_size=4G
Replica
For this test, as a Replica we deployed AliSQL 8.0.44-2, Alibaba’s MySQL branch based on MySQL 8.0.44, which includes the DuckDB storage engine along with other features such as Vector Search and Flashback. At this moment, this is the latest available version.
We configured the DuckDB instance with the following settings:[mysqld]
duckdb_memory_limit=4G    # maximum memory DuckDB is allowed to use
force_innodb_to_duckdb=ON # force-replace the InnoDB engine with DuckDB
duckdb_threads=0          # total number of threads used by DuckDB. 0 equals system CPU cores.You can find full reference for DuckDB settings here https://github.com/alibaba/AliSQL/blob/master/wiki/duckdb/duckdb_variables-en.md  
The force_innodb_to_duckdb option is mandatory in this process, as it is the way to tell MySQL to use DuckDB engine for tables created via replication or mysqldump.
After loading the 5000 products and 400K orders, the size of the WooCommerce database directory using the InnoDB engine was 7.3 GiB:$ du -sch woocommerce/
7.3G woocommerce/While on the replica where tables have been created with DuckDB, the dataset fits in just 601 MiB:$ du -sch duckdb.db
601M duckdb.dbThat is a solid 12x improvement on disk footprint!
Sending reports queries directly to the Replica
We wanted to find a way to make the process as transparent as possible, but WooCommerce uses WordPress’s single database connection object for everything, which means that the reports’ code will use the global $wpdb, and there is no setting that would allow us to send this reporting/analytics traffic to a different server, or an easy way to introduce a second connection object to the code. This means you are stuck with a single server, so there would be no easy way to make use of both InnoDB and DuckDB copies.
Thus, we had to come up with some solution to run the reports against the replica. We like simple stuff, so instead of changing the existing WooCommerce installation at all, we run a whole second copy of the same WooCommerce installation that connects to the replica, and we push the analytics traffic to that second WooCommerce copy.
The original install of your store stays exactly as it is, with its wp-config.php which already points to the primary (InnoDB/transactional) database:define( 'DB_HOST', 'ps8.example.local' );   // Percona Server for MySQL 8.0.46, InnoDBAnd the second install is just a full copy of the first, in its own directory, with just one line of code changed: the database host.$ cp -aL /var/www/html /var/www/html2;  # we assume your WooCommerce install is /var/www/html 
$ chown -R apache:apache /var/www/html2 # and we also assume you run Apache; should not be hard to adapt for Nginx or other web serversIn the copy’s wp-config.php we change only the host and leave the table prefix, the authentication keys, everything else untouched:define( 'DB_HOST', 'duckdb8.example.local'' );  // AliSQL 8.0.44 with the DuckDB storage engine, the replicaThe second install gets a dedicated pool, listening on its own socket, and its own virtual host on an internal port, handing .php to that pool:# /etc/httpd/conf.d/html2.conf
&amp;lt;VirtualHost 127.0.0.1:8080&amp;gt;
    DocumentRoot /var/www/html2
    &amp;lt;Directory &quot;/var/www/html2&quot;&amp;gt;
        AllowOverride All
        Require all granted
    &amp;lt;/Directory&amp;gt;
    &amp;lt;FilesMatch \.php$&amp;gt;
        SetHandler &quot;proxy:unix:/run/php-fpm/www2.sock|fcgi://localhost&quot;
    &amp;lt;/FilesMatch&amp;gt;
&amp;lt;/VirtualHost&amp;gt;With that in place, we have two addresses for (essentially) the same WooCommerce, each using a different database:
– http://127.0.0.1:80/ is the store, on Percona Server for MySQL 8.0.46 (InnoDB).
– http://127.0.0.1:8080/ is the copy, on DuckDB replica.
Every analytics call we benchmark goes to one or the other, so the same report is run against InnoDB and against DuckDB from the same codebase, changing nothing but the database underneath.
Benchmark Methodology
Each report was run several times, and we kept the fastest run for each. 
Before comparing times, we verified that both engines returned the same result set: the same number of rows and the same values, once serialized and sorted. A few reports differed because the DuckDB engine computes decimals as double-precision floating-point rather than exact DECIMAL, so some monetary sums ended up a fraction of a cent apart. While this might be of concern when dealing with real money, for reporting the precision DuckDB provides should be more than enough, as the difference between the reports was always below 1 cent. 
The two engines never ran at the same time. Every report ran first on InnoDB and then on DuckDB, so they never competed for CPU, memory, or disk (at the web server layer). 
A few reports whose SQL relied on MySQL-only syntax failed outright on DuckDB and were discarded rather than rewritten, since the point was to run the same query on both engines.  
We drove the benchmarks through WooCommerce’s own Analytics REST API, sending the same report requests to each engine and timing the responses.
Each report was run with the force_cache_refresh parameter, so the result is computed from the database on every call rather than served from WooCommerce’s cache.
Built-in Analytics Reports
We started with the built-in WooCommerce Analytics reports. All reports used the same window, from 2024-01-01 to 2026-09-01, over a store of 400,000 orders. The stats reports were aggregated by month; the detail reports returned up to 100 rows per page.

Every built-in report ran faster on DuckDB, from about 10x to nearly 63x, and all of them finished in under a second against InnoDB’s 5 to 16. The biggest improvement resulted from the category reports, where the heavy grouping and aggregation show the columnar engine’s strength.
Built-in Analytics Segmented Reports
The built-in reports are not limited to a single total over a date range. Each stats report takes an interval, so you can ask for the same figures broken down by hour, day, week, or month, and several of them also accept a segmentby parameter, which splits the totals along a dimension. Segmenting by category, for example, returns revenue or order numbers per product category rather than as a single combined figure, which is how the Analytics screens generate their breakdowns. 
The finer the interval and the more segments involved, the more the report asks of the database. These are the reports we use to compare InnoDB and DuckDB because they are where the underlying engine makes the most difference.

Every segmented report ran faster on DuckDB, ranging from about 14x to 83x faster. Segmenting by category and time pushes InnoDB into multi-minute queries, while DuckDB still answers in seconds, so the heavier the breakdown, the wider the gap.
WooCommerce Customer Analytics Plugin
To see whether the engine helps with real third-party code, not just the built-in reports, we also benchmarked the Customer Analytics for WooCommerce plugin from Coddium.
It extends WooCommerce Analytics with customer metrics the built-in reports do not provide: customer lifetime value, churn rate, RFM segments, cohort retention, repeat purchase rate, order sequence, and acquisition sources, among others. 
Like the built-in reports, each of these reads from the same WooCommerce Analytics lookup tables, making it a good test of whether a plugin written for MySQL runs faster on the DuckDB engine without any changes of its own.

Even with a third-party plugin we did not modify, all eleven reports ran faster on DuckDB, from about 2x to nearly 70x faster.
Ad-hoc Query on Live Data
As mentioned, the built-in analytics reports are served from precomputed lookup tables, which only answer the questions they were designed to answer. For anything else, you have to query the live order tables directly. So we tested an expensive query against those tables to compare the performance. 
In this case, we wanted to know how many items were sold, broken down by product category, city, and day of the week, across the store’s orders in the date range. We came up with the following query:SELECT
  tt.term_taxonomy_id                            AS category_id,
  LOWER(a.city)                                  AS city,
  DAYOFWEEK(o.date_created_gmt)                  AS dow,
  SUM(CAST(im_qty.meta_value AS DECIMAL(10,2)))  AS items_sold
FROM woocommerce.wp_wc_orders o
JOIN woocommerce.wp_wc_order_addresses a
  ON a.order_id = o.id AND a.address_type = 'billing'
JOIN woocommerce.wp_woocommerce_order_items oi
  ON oi.order_id = o.id AND oi.order_item_type = 'line_item'
JOIN woocommerce.wp_woocommerce_order_itemmeta im_prod
  ON im_prod.order_item_id = oi.order_item_id AND im_prod.meta_key = '_product_id'
JOIN woocommerce.wp_woocommerce_order_itemmeta im_qty
  ON im_qty.order_item_id = oi.order_item_id AND im_qty.meta_key = '_qty'
JOIN woocommerce.wp_term_relationships tr
  ON tr.object_id = CAST(im_prod.meta_value AS DECIMAL(20,0))
JOIN woocommerce.wp_term_taxonomy tt
  ON tt.term_taxonomy_id = tr.term_taxonomy_id AND tt.taxonomy = 'product_cat'
WHERE o.type = 'shop_order'
  AND o.status IN ('wc-completed','wc-processing')
  AND o.date_created_gmt &amp;gt;= '2024-01-01' AND o.date_created_gmt &amp;lt; '2026-09-01'
GROUP BY category_id, LOWER(a.city), dow;The execution time difference between both engines can be found below:

Across three runs each, DuckDB was consistently about 4.1 seconds and InnoDB about 4 minutes 16 seconds.
The results are not identical, and the two extra groups come from how each engine compares text. Since the city column uses a case- and accent-insensitive collation that InnoDB applies, it folds an accented spelling and its plain form into a single group. The DuckDB engine does not apply the collation and compares text byte-for-byte, keeping them apart. 
No orders are missing on either side; the same rows are simply bucketed into two more groups.
Limitations
We ran into the following limitations when performing the above benchmarks:


DuckDB SQL compatibility


Since the DuckDB storage engine executes queries in its own engine, the SQL syntax should be portable. A query that runs fine on InnoDB can fail on DuckDB when it uses a keyword or data type that DuckDB does not recognize. For example, the CAST function in MySQL accepts the UNSIGNED keyword, but when it’s pushed into DuckDB, it fails with the following error:ERROR 7501 (HY000): [DuckDB] Catalog Error: Type with name UNSIGNED does not exist!
Did you mean &quot;JSON&quot;?.Such queries need to be rewritten so DuckDB can process them. This can be automated with query-rewrite tools such as ProxySQL Query Rewrite feature or MySQL’s Rewriter Query Rewrite plugin, which we will cover in a separate blog post.


Collation


Unlike InnoDB, DuckDB does not apply the column collation and forces binary comparison. This can produce different result sets between engines. 
For example, the following SELECT COUNT result set differs between DuckDB and InnoDB:# DuckDB
mysql&amp;gt; SELECT COUNT(DISTINCT city) FROM woocommerce.wp_wc_order_addresses WHERE address_type='billing' AND city&amp;lt;&amp;gt;'';
+----------------------+
| COUNT(DISTINCT city) |
+----------------------+
|               114412 |
+----------------------+
1 row in set (0.02 sec)

# InnoDB
mysql&amp;gt; SELECT COUNT(DISTINCT city) FROM woocommerce.wp_wc_order_addresses WHERE address_type='billing' AND city&amp;lt;&amp;gt;'';
+----------------------+
| COUNT(DISTINCT city) |
+----------------------+
|               114375 |
+----------------------+
1 row in set (1.38 sec)This is caused by DuckDB bypassing the column collation.
If we force binary comparison when running the query on InnoDB, we obtain the same result set as DuckDB:# InnoDB BINARY
mysql&amp;gt; SELECT COUNT(DISTINCT BINARY city) FROM woocommerce.wp_wc_order_addresses WHERE address_type='billing' AND city&amp;lt;&amp;gt;'';
+-----------------------------+
| COUNT(DISTINCT BINARY city) |
+-----------------------------+
|                      114412 |
+-----------------------------+
1 row in set, 1 warning (1.34 sec)To prevent this, group and join should be done on numeric IDs rather than text.


Decimals computed as Double


To improve performance, DuckDB does not support DECIMAL precision by default, so monetary columns are summed in floating point instead. This can lead to differences at sub-cent digits. If you need exact decimal values, you can disable the duckdb_use_double_for_decimal variable when importing the data. 
We have not yet fully tested performance implications of disabling this, but we plan to cover this in another blogpost, in the near future.


Foreign keys


Traditionally in MySQL, cascaded foreign key operations such as ON DELETE CASCADE were handled internally by InnoDB, below the SQL layer, so they were not written to the binlog files. 
Because the DuckDB storage engine does not perform these cascades itself, changes to child rows would be lost when replicating from InnoDB to DuckDB. 
Starting with MySQL 9.6, foreign key cascades are handled at the SQL layer by default, so the affected rows are recorded in the binary log, making replication across different engines transparent. 
The following MySQL blog post explains this change in detail:
https://blogs.oracle.com/mysql/no-more-hidden-changes-how-mysql-9-6-transforms-foreign-key-management
WooCommerce does not use cascading foreign keys, so this limitation does not apply in this case.


Observability


DuckDB exposes only a few of status variables. Status variables mostly show counters for statements (Com_duckdb_select, Com_duckdb_insert) and rows processed in batches. 
InnoDB, by comparison, exposes dozens of status variables covering the buffer pool, I/O, row operations, and locking, and is instrumented in detail through the Performance Schema. 
So there is little visibility into what the DuckDB engine is doing internally, such as memory use, disk spilling, or query execution, which makes monitoring and tuning harder than on InnoDB. 
This gap makes sense though, given how much longer InnoDB has existed; DuckDB has had far less time to build equivalent instrumentation.
Why Columnar engine wins?
The reason is physical, and it comes down to how the data is stored on disk. InnoDB is a row store; it keeps all of a row’s columns together, which is ideal for transactional work, where you read or write an entire row at a time.
Analytics asks the opposite question. A report rarely needs the whole row; it sums one column, counts another, and groups by a third, across millions of rows.
In a row store, answering that means reading every full row from the B-tree and discarding most of it.
A columnar engine stores each column separately, so it reads only the fields a query touches, compresses them far better because the values share a type, and processes them in batches instead of row by row. Reading less, reading it in compressed form, and processing it in bulk is what turns InnoDB’s minutes into DuckDB’s seconds. The heavier the aggregation, the wider the gap, which is exactly what the benchmarks showed.
P.S. DuckDB is a potential solution for analytics use cases, and this post is part of our testing. At this time, Percona does not have a production DuckDB build and does not support it.
The post WooCommerce Analytics with DuckDB appeared first on Percona.</description>
    <content:encoded><![CDATA[<h3>MySQL and DuckDB</h3>
<p><span>Unlike row-oriented databases, which store data for a single record sequentially on disk, favoring single-row lookups, columnar databases store all the values from a column contiguously in storage, allowing only the selected column to be read from disk.</span></p>
<p><span>The default MySQL engine is InnoDB, which is ACID-compliant and great for OLTP (Online Transaction Processing) workloads. That said, OLAP (Online Analytical Processing) queries are built differently; they scan, group, and summarize millions or billions of rows to produce reports. In these cases, the InnoDB engine can be slow to process such queries. This has been a debt in MySQL over the years; it was common to setup ETL/CDC pipelines to export or stream the data to another engine, usually a columnar database system, to run reporting and BI queries on it.</span></p>
<p><span>DuckDB is a portable analytical database system, designed to be fast and reliable, similar to SQLite.</span></p>
<p><span>Recently, various vendors (MariaDB, Alibaba) started including DuckDB inside MySQL as a storage engine. A table declared with ENGINE=DuckDB stores data in columnar format, and queries are handled by the DuckDB engine.</span></p>
<p><span>This is useful in several ways; the main one is to use MySQL asynchronous replication to stream data from the OLTP, using the InnoDB engine, to a Replica where tables are in DuckDB, thereby offloading the OLAP workload to the replica. To showcase this, we thought that the best example was an on-line store which naturally requires both: short, fast, scalable transactions for the store frontend, and analytics for the store’s reporting and Business Intelligence, and what better than the free and open-source software that serves between 20 and 40% of the world’s e-commerce websites: WooCommerce.</span></p>
<h3><span>WooCommerce</span></h3>
<p><span>There are around 4M WooCommerce stores worldwide, the most of any e-commerce platform, handling between 20 and 39% of all e-commerce sites, and they use MySQL as their underlying database.</span></p>
<p><span>One of the reasons stores have been adopting computers for over 50 years, is the value of having accumulated information about your business’ transactions. This data collection allows business owners to understand trends and find new opportunities, by turning raw data into metrics that marketing and sales departments can act upon. This process is based on the long-term accumulation of large amounts of data and metadata, and that accumulation is often incompatible with transactional/row-oriented storage engines.</span></p>
<p><span>WooCommerce has a built-in analytics module to create those reports that provide metrics, so it is the perfect product to test the performance difference between InnoDB and DuckDB.</span></p>
<p><span>WooCommerce analytics does not run its reports against live order tables because querying these tables would be too slow. Instead, it keeps a set of aggregations in a number of denormalized “lookup” tables, which are optimized for this purpose. This is a limitation when running reports with InnoDB, as querying the raw tables would kill performance, and thus the reports are limited to what you can get from these lookup tables.</span></p>
<p><span>The following lookup tables are created for WooCommerce Analytics:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">mysql&gt; SHOW TABLES LIKE '%wc_%lookup'; SHOW TABLES LIKE '%wc_order_stats';
+-------------------------------------+
| Tables_in_woocommerce (%wc_%lookup) |
+-------------------------------------+
| wp_wc_category_lookup               |
| wp_wc_customer_lookup               |
| wp_wc_order_coupon_lookup           |
| wp_wc_order_product_lookup          |
| wp_wc_order_tax_lookup              |
| wp_wc_product_attributes_lookup     |
| wp_wc_product_meta_lookup           |
+-------------------------------------+
7 rows in set (0.00 sec)

+-----------------------------------------+
| Tables_in_woocommerce (%wc_order_stats) |
+-----------------------------------------+
| wp_wc_order_stats                       |
+-----------------------------------------+
1 row in set (0.00 sec)</pre><p><span>The wp_wc_order_stats table is the spine of the revenue and orders reports; it contains one row per order, including dates, status, customer, number of items sold, and the monetary totals.</span></p>
<p><span>In the store we’re using, the wc_order_stats table contains 400000 rows, and there are 5000 products. This data was loaded using the </span><a href="https://github.com/woocommerce/wc-smooth-generator"><span>WooCommerce Smooth Generator plugin</span></a><span>.</span></p>
<p><span>The lookup tables are derived data, so WooCommerce has to keep them in sync with the orders they summarize, which it does in the background (which means you have to run a scheduled process that is what keeps these lookup tables updated).</span></p>
<h3><span>MySQL Deployment</span></h3>
<p><span>We deployed a Source/Replica MySQL topology with both databases on the same host, each with its respective Docker image.</span></p>
<h4><span>Source</span></h4>
<p><span>The Source runs Percona Server for MySQL 8.0.46, which we chose to match the replica’s major version.</span></p>
<p><span>WooCommerce tables use the MySQL default engine, which is InnoDB.</span></p>
<p><span>We configured an InnoDB Buffer Pool of 4 GiB:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">[mysqld]
innodb_buffer_pool_size=4G</pre><p></p>
<h4><span>Replica</span></h4>
<p><span>For this test, as a Replica we deployed AliSQL 8.0.44-2, Alibaba’s MySQL branch based on MySQL 8.0.44, which includes the </span><a href="https://github.com/alibaba/AliSQL/blob/master/wiki/duckdb/duckdb.md"><span>DuckDB</span></a><span> storage engine along with other features such as Vector Search and Flashback. At this moment, this is the latest available version.</span></p>
<p><span>We configured the DuckDB instance with the following settings:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">[mysqld]
duckdb_memory_limit=4G    # maximum memory DuckDB is allowed to use
force_innodb_to_duckdb=ON # force-replace the InnoDB engine with DuckDB
duckdb_threads=0          # total number of threads used by DuckDB. 0 equals system CPU cores.</pre><p><span>You can find full reference for DuckDB settings here </span><a href="https://github.com/alibaba/AliSQL/blob/master/wiki/duckdb/duckdb_variables-en.md"><span>https://github.com/alibaba/AliSQL/blob/master/wiki/duckdb/duckdb_variables-en.md</span></a><span>  </span></p>
<p><span>The force_innodb_to_duckdb option is mandatory in this process, as it is the way to tell MySQL to use DuckDB engine for tables created via replication or mysqldump.</span></p>
<p><span>After loading the 5000 products and 400K orders, the size of the WooCommerce database directory using the InnoDB engine was 7.3 GiB:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">$ du -sch woocommerce/
7.3G woocommerce/</pre><p><span>While on the replica where tables have been created with DuckDB, the dataset fits in just 601 MiB:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">$ du -sch duckdb.db
601M duckdb.db</pre><p><span>That is a solid 12x improvement on disk footprint!</span></p>
<h3><span>Sending reports queries directly to the Replica</span></h3>
<p><span>We wanted to find a way to make the process as transparent as possible, but WooCommerce uses WordPress’s single database connection object for everything, which means that the reports’ code will use the global $wpdb, and there is no setting that would allow us to send this reporting/analytics traffic to a different server, or an easy way to introduce a second connection object to the code. This means you are stuck with a single server, so there would be no easy way to make use of both InnoDB and DuckDB copies.</span></p>
<p><span>Thus, we had to come up with some solution to run the reports against the replica. We like simple stuff, so instead of changing the existing WooCommerce installation at all, we run a whole second copy of the same WooCommerce installation that connects to the replica, and we push the analytics traffic to that second WooCommerce copy.</span></p>
<p><span>The original install of your store stays exactly as it is, with its <code>wp-config.php</code> which already points to the primary (InnoDB/transactional) database:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">define( 'DB_HOST', 'ps8.example.local' );   // Percona Server for MySQL 8.0.46, InnoDB</pre><p><span>And the second install is just a full copy of the first, in its own directory, with just one line of code changed: the database host.</span></p><pre class="urvanov-syntax-highlighter-plain-tag">$ cp -aL /var/www/html /var/www/html2;  # we assume your WooCommerce install is /var/www/html 
$ chown -R apache:apache /var/www/html2 # and we also assume you run Apache; should not be hard to adapt for Nginx or other web servers</pre><p><span>In the copy’s wp-config.php we change only the host and leave the table prefix, the authentication keys, everything else untouched:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">define( 'DB_HOST', 'duckdb8.example.local'' );  // AliSQL 8.0.44 with the DuckDB storage engine, the replica</pre><p><span>The second install gets a dedicated pool, listening on its own socket, and its own virtual host on an internal port, handing <code>.php</code> to that pool:</span></p><pre class="urvanov-syntax-highlighter-plain-tag"># /etc/httpd/conf.d/html2.conf
&lt;VirtualHost 127.0.0.1:8080&gt;
    DocumentRoot /var/www/html2
    &lt;Directory "/var/www/html2"&gt;
        AllowOverride All
        Require all granted
    &lt;/Directory&gt;
    &lt;FilesMatch \.php$&gt;
        SetHandler "proxy:unix:/run/php-fpm/www2.sock|fcgi://localhost"
    &lt;/FilesMatch&gt;
&lt;/VirtualHost&gt;</pre><p><span>With that in place, we have two addresses for (essentially) the same WooCommerce, each using a different database:</span></p>
<p><span>– <code>http://127.0.0.1:80/</code> is the store, on Percona Server for MySQL 8.0.46 (InnoDB).</span></p>
<p><span>– <code>http://127.0.0.1:8080/</code> is the copy, on DuckDB replica.</span></p>
<p><span>Every analytics call we benchmark goes to one or the other, so the same report is run against InnoDB and against DuckDB from the same codebase, changing nothing but the database underneath.</span></p>
<h3><span>Benchmark Methodology</span></h3>
<p><span>Each report was run several times, and we kept the fastest run for each. </span></p>
<p><span>Before comparing times, we verified that both engines returned the same result set: the same number of rows and the same values, once serialized and sorted. A few reports differed because the DuckDB engine computes decimals as double-precision floating-point rather than exact DECIMAL, so some monetary sums ended up a fraction of a cent apart. While this might be of concern when dealing with real money, for reporting the precision DuckDB provides should be more than enough, as the difference between the reports was always below 1 cent. </span></p>
<p><span>The two engines never ran at the same time. Every report ran first on InnoDB and then on DuckDB, so they never competed for CPU, memory, or disk (at the web server layer). </span></p>
<p><span>A few reports whose SQL relied on MySQL-only syntax failed outright on DuckDB and were discarded rather than rewritten, since the point was to run the same query on both engines.  </span></p>
<p><span>We drove the benchmarks through WooCommerce’s own Analytics REST API, sending the same report requests to each engine and timing the responses.</span></p>
<p><span>Each report was run with the force_cache_refresh parameter, so the result is computed from the database on every call rather than served from WooCommerce’s cache.</span></p>
<h3><span>Built-in Analytics Reports</span></h3>
<p><span>We started with the built-in </span><a href="https://woocommerce.com/products/woocommerce-analytics/"><span>WooCommerce Analytics</span></a><span> reports. All reports used the same window, from 2024-01-01 to 2026-09-01, over a store of 400,000 orders. The stats reports were aggregated by month; the detail reports returned up to 100 rows per page.</span></p>
<p><img loading="lazy" decoding="async" class="alignnone wp-image-55564 size-medium_large" src="https://www.percona.com/wp-content/uploads/2026/11/builtin-reports-light-768x802.png" alt="" width="768" height="802" srcset="https://www.percona.com/wp-content/uploads/2026/11/builtin-reports-light-768x802.png 768w, https://www.percona.com/wp-content/uploads/2026/11/builtin-reports-light-287x300.png 287w, https://www.percona.com/wp-content/uploads/2026/11/builtin-reports-light-981x1024.png 981w, https://www.percona.com/wp-content/uploads/2026/11/builtin-reports-light-1471x1536.png 1471w, https://www.percona.com/wp-content/uploads/2026/11/builtin-reports-light.png 1640w" sizes="auto, (max-width: 768px) 100vw, 768px"></p>
<p><span>Every built-in report ran faster on DuckDB, from about 10x to nearly 63x, and all of them finished in under a second against InnoDB’s 5 to 16. The biggest improvement resulted from the category reports, where the heavy grouping and aggregation show the columnar engine’s strength.</span></p>
<h3><span>Built-in Analytics Segmented Reports</span></h3>
<p><span>The built-in reports are not limited to a single total over a date range. Each stats report takes an interval, so you can ask for the same figures broken down by hour, day, week, or month, and several of them also accept a segmentby parameter, which splits the totals along a dimension. Segmenting by category, for example, returns revenue or order numbers per product category rather than as a single combined figure, which is how the Analytics screens generate their breakdowns. </span></p>
<p><span>The finer the interval and the more segments involved, the more the report asks of the database. These are the reports we use to compare InnoDB and DuckDB because they are where the underlying engine makes the most difference.</span></p>
<p><img loading="lazy" decoding="async" class="alignnone wp-image-55567 size-medium_large" src="https://www.percona.com/wp-content/uploads/2026/11/segmented-reports-light-768x524.png" alt="" width="768" height="524" srcset="https://www.percona.com/wp-content/uploads/2026/11/segmented-reports-light-768x524.png 768w, https://www.percona.com/wp-content/uploads/2026/11/segmented-reports-light-300x205.png 300w, https://www.percona.com/wp-content/uploads/2026/11/segmented-reports-light-1024x699.png 1024w, https://www.percona.com/wp-content/uploads/2026/11/segmented-reports-light-1536x1049.png 1536w, https://www.percona.com/wp-content/uploads/2026/11/segmented-reports-light.png 1640w" sizes="auto, (max-width: 768px) 100vw, 768px"></p>
<p><span>Every segmented report ran faster on DuckDB, ranging from about 14x to 83x faster. Segmenting by category and time pushes InnoDB into multi-minute queries, while DuckDB still answers in seconds, so the heavier the breakdown, the wider the gap.</span></p>
<h3><span>WooCommerce Customer Analytics Plugin</span></h3>
<p><span>To see whether the engine helps with real third-party code, not just the built-in reports, we also benchmarked the </span><a href="https://coddium.com/customer-analytics-for-woocommerce/"><span>Customer Analytics for WooCommerce</span></a><span> plugin from Coddium.</span></p>
<p><span>It extends WooCommerce Analytics with customer metrics the built-in reports do not provide: customer lifetime value, churn rate, RFM segments, cohort retention, repeat purchase rate, order sequence, and acquisition sources, among others. </span></p>
<p><span>Like the built-in reports, each of these reads from the same WooCommerce Analytics lookup tables, making it a good test of whether a plugin written for MySQL runs faster on the DuckDB engine without any changes of its own.</span></p>
<p><img loading="lazy" decoding="async" class="alignnone wp-image-55574 size-medium_large" src="https://www.percona.com/wp-content/uploads/2026/11/plugin-reports-light-1000w-768x1111.webp" alt="" width="768" height="1111" srcset="https://www.percona.com/wp-content/uploads/2026/11/plugin-reports-light-1000w-768x1111.webp 768w, https://www.percona.com/wp-content/uploads/2026/11/plugin-reports-light-1000w-207x300.webp 207w, https://www.percona.com/wp-content/uploads/2026/11/plugin-reports-light-1000w-708x1024.webp 708w, https://www.percona.com/wp-content/uploads/2026/11/plugin-reports-light-1000w.webp 1000w" sizes="auto, (max-width: 768px) 100vw, 768px"></p>
<p><span>Even with a third-party plugin we did not modify, all eleven reports ran faster on DuckDB, from about 2x to nearly 70x faster.</span></p>
<h3><span>Ad-hoc Query on Live Data</span></h3>
<p><span>As mentioned, the built-in analytics reports are served from precomputed lookup tables, which only answer the questions they were designed to answer. For anything else, you have to query the live order tables directly. So we tested an expensive query against those tables to compare the performance. </span></p>
<p><span>In this case, we wanted to know how many items were sold, broken down by product category, city, and day of the week, across the store’s orders in the date range. We came up with the following query:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">SELECT
  tt.term_taxonomy_id                            AS category_id,
  LOWER(a.city)                                  AS city,
  DAYOFWEEK(o.date_created_gmt)                  AS dow,
  SUM(CAST(im_qty.meta_value AS DECIMAL(10,2)))  AS items_sold
FROM woocommerce.wp_wc_orders o
JOIN woocommerce.wp_wc_order_addresses a
  ON a.order_id = o.id AND a.address_type = 'billing'
JOIN woocommerce.wp_woocommerce_order_items oi
  ON oi.order_id = o.id AND oi.order_item_type = 'line_item'
JOIN woocommerce.wp_woocommerce_order_itemmeta im_prod
  ON im_prod.order_item_id = oi.order_item_id AND im_prod.meta_key = '_product_id'
JOIN woocommerce.wp_woocommerce_order_itemmeta im_qty
  ON im_qty.order_item_id = oi.order_item_id AND im_qty.meta_key = '_qty'
JOIN woocommerce.wp_term_relationships tr
  ON tr.object_id = CAST(im_prod.meta_value AS DECIMAL(20,0))
JOIN woocommerce.wp_term_taxonomy tt
  ON tt.term_taxonomy_id = tr.term_taxonomy_id AND tt.taxonomy = 'product_cat'
WHERE o.type = 'shop_order'
  AND o.status IN ('wc-completed','wc-processing')
  AND o.date_created_gmt &gt;= '2024-01-01' AND o.date_created_gmt &lt; '2026-09-01'
GROUP BY category_id, LOWER(a.city), dow;</pre><p><span>The execution time difference between both engines can be found below:</span></p>
<p><img loading="lazy" decoding="async" class="alignnone wp-image-55568 size-medium_large" src="https://www.percona.com/wp-content/uploads/2026/11/adhoc-live-query-light-768x407.png" alt="" width="768" height="407" srcset="https://www.percona.com/wp-content/uploads/2026/11/adhoc-live-query-light-768x407.png 768w, https://www.percona.com/wp-content/uploads/2026/11/adhoc-live-query-light-300x159.png 300w, https://www.percona.com/wp-content/uploads/2026/11/adhoc-live-query-light-1024x543.png 1024w, https://www.percona.com/wp-content/uploads/2026/11/adhoc-live-query-light-1536x815.png 1536w, https://www.percona.com/wp-content/uploads/2026/11/adhoc-live-query-light.png 1640w" sizes="auto, (max-width: 768px) 100vw, 768px"></p>
<p><span>Across three runs each, DuckDB was consistently about 4.1 seconds and InnoDB about 4 minutes 16 seconds.</span></p>
<p><span>The results are not identical, and the two extra groups come from how each engine compares text. Since the city column uses a case- and accent-insensitive collation that InnoDB applies, it folds an accented spelling and its plain form into a single group. The DuckDB engine does not apply the collation and compares text byte-for-byte, keeping them apart. </span></p>
<p><span>No orders are missing on either side; the same rows are simply bucketed into two more groups.</span></p>
<h3><span>Limitations</span></h3>
<p><span>We ran into the following limitations when performing the above benchmarks:</span></p>
<ul>
<li aria-level="1">
<h4><span>DuckDB SQL compatibility</span></h4>
</li>
</ul>
<p><span>Since the DuckDB storage engine executes queries in its own engine, the SQL syntax should be portable. A query that runs fine on InnoDB can fail on DuckDB when it uses a keyword or data type that DuckDB does not recognize. For example, the CAST function in MySQL accepts the UNSIGNED keyword, but when it’s pushed into DuckDB, it fails with the following error:</span></p><pre class="urvanov-syntax-highlighter-plain-tag">ERROR 7501 (HY000): [DuckDB] Catalog Error: Type with name UNSIGNED does not exist!
Did you mean "JSON"?.</pre><p><span>Such queries need to be rewritten so DuckDB can process them. This can be automated with query-rewrite tools such as </span><a href="https://proxysql.com/documentation/query-rewrite/"><span>ProxySQL Query Rewrite</span></a><span> feature or </span><a href="https://dev.mysql.com/doc/refman/8.4/en/rewriter-query-rewrite-plugin.html"><span>MySQL’s Rewriter Query Rewrite plugin</span></a><span>, which we will cover in a separate blog post.</span></p>
<ul>
<li aria-level="1">
<h4><span>Collation</span></h4>
</li>
</ul>
<p><span>Unlike InnoDB, DuckDB does not apply the column collation and forces binary comparison. This can produce different result sets between engines. </span></p>
<p><span>For example, the following SELECT COUNT result set differs between DuckDB and InnoDB:</span></p><pre class="urvanov-syntax-highlighter-plain-tag"># DuckDB
mysql&gt; SELECT COUNT(DISTINCT city) FROM woocommerce.wp_wc_order_addresses WHERE address_type='billing' AND city&lt;&gt;'';
+----------------------+
| COUNT(DISTINCT city) |
+----------------------+
|               114412 |
+----------------------+
1 row in set (0.02 sec)

# InnoDB
mysql&gt; SELECT COUNT(DISTINCT city) FROM woocommerce.wp_wc_order_addresses WHERE address_type='billing' AND city&lt;&gt;'';
+----------------------+
| COUNT(DISTINCT city) |
+----------------------+
|               114375 |
+----------------------+
1 row in set (1.38 sec)</pre><p>This is caused by DuckDB bypassing the column collation.</p>
<p>If we force binary comparison when running the query on InnoDB, we obtain the same result set as DuckDB:</p><pre class="urvanov-syntax-highlighter-plain-tag"># InnoDB BINARY
mysql&gt; SELECT COUNT(DISTINCT BINARY city) FROM woocommerce.wp_wc_order_addresses WHERE address_type='billing' AND city&lt;&gt;'';
+-----------------------------+
| COUNT(DISTINCT BINARY city) |
+-----------------------------+
|                      114412 |
+-----------------------------+
1 row in set, 1 warning (1.34 sec)</pre><p><span>To prevent this, group and join should be done on numeric IDs rather than text.</span></p>
<ul>
<li aria-level="1">
<h4><span>Decimals computed as Double</span></h4>
</li>
</ul>
<p><span>To improve performance, DuckDB does not support DECIMAL precision by default, so monetary columns are summed in floating point instead. This can lead to differences at sub-cent digits. If you need exact decimal values, you can disable the duckdb_use_double_for_decimal variable when importing the data. </span></p>
<p><span>We have not yet fully tested performance implications of disabling this, but we plan to cover this in another blogpost, in the near future.</span></p>
<ul>
<li aria-level="1">
<h4><span>Foreign keys</span></h4>
</li>
</ul>
<p><span>Traditionally in MySQL, cascaded foreign key operations such as ON DELETE CASCADE were handled internally by InnoDB, below the SQL layer, so they were not written to the binlog files. </span></p>
<p><span>Because the DuckDB storage engine does not perform these cascades itself, changes to child rows would be lost when replicating from InnoDB to DuckDB. </span></p>
<p><span>Starting with MySQL 9.6, foreign key cascades are handled at the SQL layer by default, so the affected rows are recorded in the binary log, making replication across different engines transparent. </span></p>
<p>The following MySQL blog post explains this change in detail:</p>
<p><a href="https://blogs.oracle.com/mysql/no-more-hidden-changes-how-mysql-9-6-transforms-foreign-key-management">https://blogs.oracle.com/mysql/no-more-hidden-changes-how-mysql-9-6-transforms-foreign-key-management</a></p>
<p><span>WooCommerce does not use cascading foreign keys, so this limitation does not apply in this case.</span></p>
<ul>
<li aria-level="1">
<h4><span>Observability</span></h4>
</li>
</ul>
<p><span>DuckDB exposes only a few of status variables. Status variables mostly show counters for statements (<code>Com_duckdb_select</code>, <code>Com_duckdb_insert</code>) and rows processed in batches. </span></p>
<p><span>InnoDB, by comparison, exposes dozens of status variables covering the buffer pool, I/O, row operations, and locking, and is instrumented in detail through the Performance Schema. </span></p>
<p><span>So there is little visibility into what the DuckDB engine is doing internally, such as memory use, disk spilling, or query execution, which makes monitoring and tuning harder than on InnoDB. </span></p>
<p><span>This gap makes sense though, given how much longer InnoDB has existed; DuckDB has had far less time to build equivalent instrumentation.</span></p>
<h3><span>Why Columnar engine wins?</span></h3>
<p><span>The reason is physical, and it comes down to how the data is stored on disk. InnoDB is a row store; it keeps all of a row’s columns together, which is ideal for transactional work, where you read or write an entire row at a time.</span></p>
<p><span>Analytics asks the opposite question. A report rarely needs the whole row; it sums one column, counts another, and groups by a third, across millions of rows.</span></p>
<p><span>In a row store, answering that means reading every full row from the B-tree and discarding most of it.</span></p>
<p><span>A columnar engine stores each column separately, so it reads only the fields a query touches, compresses them far better because the values share a type, and processes them in batches instead of row by row. Reading less, reading it in compressed form, and processing it in bulk is what turns InnoDB’s minutes into DuckDB’s seconds. The heavier the aggregation, the wider the gap, which is exactly what the benchmarks showed.</span></p>
<p><span>P.S. DuckDB is a potential solution for analytics use cases, and this post is part of our testing. At this time, Percona does not have a production DuckDB build and does not support it.</span></p>
<p>The post <a href="https://www.percona.com/blog/woocommerce-analytics-with-duckdb/">WooCommerce Analytics with DuckDB</a> appeared first on <a href="https://www.percona.com/">Percona</a>.</p>]]></content:encoded>
    <pubDate>Thu, 05 Nov 2026 16:31:19 +0000</pubDate>
    <dc:creator>MySQL Performance Blog</dc:creator>
    <category>Benchmarks</category>
    <category>Database Trends</category>
    <category>MySQL</category>
    <category>Open Source</category>
    <category>Percona Server for MySQL</category>
    <category>Storage Engine</category>
    <category>DuckDB</category>
    <category>WooCommerce</category>
  </item>

  <item>
    <title>Village News: MySQL News + Events (5 November 2026)</title>
    <guid isPermaLink="false">6aec93f8387a220001129c71</guid>
    <link>https://villagesql.com/blog/village-news-mysql-news-events-5-november-2026/</link>
    <description>
Welcome back to Village News, our curated roundup of MySQL and database news. This week, AWS contributed a fix for large-transaction binlog stalls to community MySQL, and MariaDB launched Enterprise Platform 2027.
If you want to get these updates, just subscribe to the blog.
Enjoy!
MySQL News
Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated)
AWS contributes a large-transaction binary log optimization to MySQL
AWS Database Blog
TL;DR - BOLT, long always-on in Aurora, stops a multi-gigabyte transaction from blocking other commits while it is copied into the binlog, and it speeds up crash recovery. It is planned for a future MySQL innovation release.
Vinicius Grippa: &quot;MySQL Is Not Dead, But It Needs Physiotherapy&quot;
OurSQL Foundation Blog
TL;DR - Ten months after he publicly asked whether MySQL was dead, the Readyset engineer and Oracle ACE Director talks through MySQL's strengths, where Postgres is ahead on AI, and who is filling the gaps.
An analytical replica for MySQL, on Oracle's HeatWave benchmark
Daniel Guzman-Burgos, dbtrail Blog
TL;DR - On Oracle's own HeatWave test, DBTrail, an open source binlog-to-Parquet/DuckDB replica, ran 12x more analytic queries per minute than Percona Server alone and cut the OLTP p99 from 42 to 31 ms.
A New KMIP Library in C++, and the Only Free One
Oleksiy Lukin, Percona Community Blog
TL;DR - Percona open sourced kmipclient, a native C++ KMIP client for talking to key management systems. It already backs Percona Server for MySQL's KMIP keyring component in 8.4.10+, 9.7.1+ and 26.10.
Amazon Aurora MySQL 3.14 (compatible with MySQL 8.0.46) is generally available
AWS What's New
TL;DR - Aurora MySQL 3.14 adds post-quantum TLS key exchange. Separately, Aurora MySQL 8.4.10 adds GTID tags and up to 50,000 connections per instance.
The Galera Crossroads, Part 2: Swap the Binaries, Keep the Cluster
Marco Tusa, TusaCentral
TL;DR - A step-by-step, node-by-node migration from Codership MySQL/Galera 8.4 to Percona XtraDB Cluster 8.4.11, with zero downtime and no new hardware. It covers strict mode, cluster-traffic encryption and SST settings.
How VillageSQL does open source
Adam Levin, VillageSQL
TL;DR - How the GPLv2 VillageSQL Server project is run: a CLA that grants a license rather than ownership, GitHub Issues as the source of truth, a public project board with weekly triage, and the same review process for every contributor.
MyVector Featured in the MySQL Ecosystem Project Spotlight of the Week
Alkin Tezuysal, AskDba
TL;DR - The MySQL community team's LinkedIn spotlight features MyVector, an open source plugin that adds vector storage and HNSW similarity search to MySQL through SQL.
Multi-Node Unused Index Verification
Arunjith Aravindan, Percona Blog
TL;DR - sys.schema_unused_indexes sees only one server, and its counters reset on restart. A new check_unused_indexes script reports an index only when every node in the topology agrees it is unused.
Database News
Introducing MariaDB Enterprise Platform 2027: one foundation for the agentic enterprise
Tina Yang, MariaDB
TL;DR - The platform adds agent memory, a semantic cache, AI retrieval, BYOC deployment and Raft-based multi-region clustering. A companion benchmark post claims Enterprise Server 12.3 LTS reaches about 4x the HammerDB TPROC-C throughput of 10.6.
Patroni, Ubuntu 24.04, and two unplanned PostgreSQL failovers
Franck Pachot
TL;DR - An unattended zlib update made Ubuntu's needrestart restart Patroni in the middle of a load, causing a failover, and then a second one when the new primary was patched. Includes the journal-based root-cause analysis.
The hidden latency inside your database
Simeon Griggs, PlanetScale Blog
TL;DR - Network-attached storage puts a network hop inside every database I/O. This piece explains when durable local NVMe pays off and when network storage is fine.
MariaDB Performance Claim : Technically Correct, but Misleading
Jean-François Gagné
TL;DR - A response to last week's &quot;MariaDB 11.8 became about 3× faster&quot; post. The fix recovers a regression from 10.5, so against 10.4 the gain is about 9%, and the author argues for fairer vendor claims across the ecosystem.
Lakebase Postgres is now generally available on Google Cloud
Cassie Murray, Databricks Blog
TL;DR - Databricks' serverless Postgres, with autoscaling, scale to zero, instant branching and PITR on storage shared with the lakehouse, is now GA on Google Cloud.
Still on MariaDB 10.6? Why your next upgrade is a security decision
Anna Widenius, MariaDB Foundation Blog
TL;DR - MariaDB Community Server 10.6 reached end of life on 6 July 2026, and 10.6.28 was its final release. Security fixes now land only in 10.11, 11.4, 11.8 and 12.3.
Introducing DBArena: open, reproducible benchmarks for hosted Postgres
Aditya Maruvada and Daniel Mitterdorfer, Supabase Blog
TL;DR - This Supabase-sponsored site benchmarks Supabase (heap and OrioleDB), RDS and Cloud SQL Postgres at matched vCPU and I/O, normalized for price. The methodology and tooling are open source.
Upcoming Database Events
MySQL Contributor Summit
November 4-5, 2026
Virtual
KubeCon + CloudNativeCon North America (VillageSQL is a sponsor)
November 9-12, 2026
Salt Lake City, Utah
PASS Data Community Summit
November 9-11, 2026
Hyatt Regency Seattle
Seattle, WA
pgDay México 2026
November 13, 2026
Puerta Polanco
Mexico City, Mexico
DuckDB × MySQL Day
November 17, 2026
Online
PGConf.Asia 2026
November 17-18, 2026
Hong Kong
OurSQL Foundation Webinar: What Happened to the MySQL Query Cache?
November 18, 2026, 10:00 AM ET / 16:00 CET
Online
PgConf.Nepal 2026
November 18-21, 2026
Pulchowk Campus, Tribhuvan University
Kathmandu, Nepal
PGConf.PL 2026
November 24, 2026
ARCHE Dwór Uphagena
Gdańsk, Poland
PostgreSQL Conference Japan 2026
November 27, 2026
AP Tokyo Yaesu
Tokyo, Japan
AWS re:Invent 2026
November 30 – December 4, 2026
Las Vegas, NV
OurSQL Foundation Webinar: What Is the Fastest Database? Open-Source Benchmarking with HammerDB
December 3, 2026, 10:00 AM ET / 16:00 CET
Online
Open Source Summit Japan
December 7-9, 2026
Tokyo, Japan
CIDR 2027
January 24-27, 2027
Mövenpick Hotel Amsterdam City Centre
Amsterdam, The Netherlands
MySQL Belgian Days 2027
January 28-29, 2027
ICAB Incubator
Brussels, Belgium
FOSDEM PGDay 2027
January 28-29, 2027
Brussels Marriott Hotel Grand Place
Brussels, Belgium
FOSDEM 2027
January 30-31, 2027
ULB Solbosch Campus
Brussels, Belgium
CERN PGDay 2027
February 12, 2027
CERN Council Chamber
Geneva, Switzerland
PGConf India 2027
March 2-5, 2027
Sheraton Grand Hotel at Brigade Gateway
Bengaluru, India
FOSSASIA PGDay 2027
March 9, 2027
Bangkok, Thailand
KubeCon + CloudNativeCon Europe 2027
March 15-18, 2027
Barcelona, Spain
Nordic PGDay 2027
March 16, 2027
Courtyard Kungsholmen
Stockholm, Sweden
SCaLE 24x
April 1-4, 2027
Pasadena Convention Center
Pasadena, CA
DuckCon #8
April 7, 2027
De Wester (Westergas)
Amsterdam, The Netherlands
PgDay Boston 2027
April 10, 2027
Tufts University Joyce Cummings Center
Boston, MA
Postgres Conference 2027
April 21-23, 2027
San Jose Marriott
San Jose, CA
PGConf.dev 2027
May 11-14, 2027
Plaza Centre-Ville
Montréal, QC, Canada
Percona Live Bay Area 2027
May 11-13, 2027
Computer History Museum
Mountain View, CA</description>
    <content:encoded><![CDATA[<h1></h1>
<img src="https://storage.ghost.io/c/db/c7/dbc78ca4-dcfe-468b-b2fa-030396d2a98e/content/images/2026/11/village-news-banner.jpg" alt="Village News: MySQL News + Events (5 November 2026)"><p>Welcome back to Village News, our curated roundup of MySQL and database news. This week, AWS contributed a fix for large-transaction binlog stalls to community MySQL, and MariaDB launched Enterprise Platform 2027.</p>
<p>If you want to get these updates, just <a href="https://villagesql.com/blog/blog#/portal/signup">subscribe</a> to the blog.</p>
<p>Enjoy!</p>
<h2>MySQL News</h2>
<p><em>Note: Aggregated MySQL news can be found at <a href="https://planet.oursqlcommunity.org/?source=villagesql">Planet for MySQL Community</a> and <a href="https://planet.mysql.com/">Planet MySQL</a> (Oracle curated)</em></p>
<p><strong><a href="https://aws.amazon.com/blogs/database/aws-contributes-a-large-transaction-binary-log-optimization-to-mysql/">AWS contributes a large-transaction binary log optimization to MySQL</a></strong><br>
<em>AWS Database Blog</em></p>
<p><strong>TL;DR -</strong> BOLT, long always-on in Aurora, stops a multi-gigabyte transaction from blocking other commits while it is copied into the binlog, and it speeds up crash recovery. It is planned for a future MySQL innovation release.</p>
<p><strong><a href="https://oursqlfoundation.org/blog/vinicius-grippa-interview/">Vinicius Grippa: "MySQL Is Not Dead, But It Needs Physiotherapy"</a></strong><br>
<em>OurSQL Foundation Blog</em></p>
<p><strong>TL;DR -</strong> Ten months after he publicly asked whether MySQL was dead, the Readyset engineer and Oracle ACE Director talks through MySQL's strengths, where Postgres is ahead on AI, and who is filling the gaps.</p>
<p><strong><a href="https://blog.dbtrail.com/mysql-heatwave-benchmark-analytical-replica/">An analytical replica for MySQL, on Oracle's HeatWave benchmark</a></strong><br>
<em>Daniel Guzman-Burgos, dbtrail Blog</em></p>
<p><strong>TL;DR -</strong> On Oracle's own HeatWave test, DBTrail, an open source binlog-to-Parquet/DuckDB replica, ran 12x more analytic queries per minute than Percona Server alone and cut the OLTP p99 from 42 to 31 ms.</p>
<p><strong><a href="https://percona.community/blog/2026/11/03/kmipclient-new-kmip-library-in-cpp/">A New KMIP Library in C++, and the Only Free One</a></strong><br>
<em>Oleksiy Lukin, Percona Community Blog</em></p>
<p><strong>TL;DR -</strong> Percona open sourced kmipclient, a native C++ KMIP client for talking to key management systems. It already backs Percona Server for MySQL's KMIP keyring component in 8.4.10+, 9.7.1+ and 26.10.</p>
<p><strong><a href="https://aws.amazon.com/about-aws/whats-new/2026/11/amazon-aurora-mysql-314-available/">Amazon Aurora MySQL 3.14 (compatible with MySQL 8.0.46) is generally available</a></strong><br>
<em>AWS What's New</em></p>
<p><strong>TL;DR -</strong> Aurora MySQL 3.14 adds post-quantum TLS key exchange. Separately, <a href="https://aws.amazon.com/about-aws/whats-new/2026/11/amazon-aurora-mysql-8410-available/">Aurora MySQL 8.4.10</a> adds GTID tags and up to 50,000 connections per instance.</p>
<p><strong><a href="https://www.tusacentral.net/joomla/index.php/mysql-blogs/273-the-galera-crossroads-part-2-swap-the-binaries-keep-the-cluster">The Galera Crossroads, Part 2: Swap the Binaries, Keep the Cluster</a></strong><br>
<em>Marco Tusa, TusaCentral</em></p>
<p><strong>TL;DR -</strong> A step-by-step, node-by-node migration from Codership MySQL/Galera 8.4 to Percona XtraDB Cluster 8.4.11, with zero downtime and no new hardware. It covers strict mode, cluster-traffic encryption and SST settings.</p>
<p><strong><a href="https://villagesql.com/blog/blog/open-source/">How VillageSQL does open source</a></strong><br>
<em>Adam Levin, VillageSQL</em></p>
<p><strong>TL;DR -</strong> How the GPLv2 VillageSQL Server project is run: a CLA that grants a license rather than ownership, GitHub Issues as the source of truth, a public project board with weekly triage, and the same review process for every contributor.</p>
<p><strong><a href="https://askdba.net/myvector-mysql-ecosystem-project-spotlight-of-the-week/">MyVector Featured in the MySQL Ecosystem Project Spotlight of the Week</a></strong><br>
<em>Alkin Tezuysal, AskDba</em></p>
<p><strong>TL;DR -</strong> The MySQL community team's LinkedIn spotlight features MyVector, an open source plugin that adds vector storage and HNSW similarity search to MySQL through SQL.</p>
<p><strong><a href="https://www.percona.com/blog/multi-node-unused-index-verification/">Multi-Node Unused Index Verification</a></strong><br>
<em>Arunjith Aravindan, Percona Blog</em></p>
<p><strong>TL;DR -</strong> <code>sys.schema_unused_indexes</code> sees only one server, and its counters reset on restart. A new <code>check_unused_indexes</code> script reports an index only when every node in the topology agrees it is unused.</p>
<h2>Database News</h2>
<p><strong><a href="https://mariadb.com/resources/blog/introducing-mariadb-enterprise-platform-2027-one-foundation-for-the-agentic-enterprise/">Introducing MariaDB Enterprise Platform 2027: one foundation for the agentic enterprise</a></strong><br>
<em>Tina Yang, MariaDB</em></p>
<p><strong>TL;DR -</strong> The platform adds agent memory, a semantic cache, AI retrieval, BYOC deployment and Raft-based multi-region clustering. A <a href="https://mariadb.com/resources/blog/mariadb-enterprise-server-12-3-lts-about-4x-more-oltp-throughput-than-10-6/">companion benchmark post</a> claims Enterprise Server 12.3 LTS reaches about 4x the HammerDB TPROC-C throughput of 10.6.</p>
<p><strong><a href="https://dev.to/franckpachot/patroni-ubuntu-2404-and-two-unplanned-postgresql-failovers-52gl">Patroni, Ubuntu 24.04, and two unplanned PostgreSQL failovers</a></strong><br>
<em>Franck Pachot</em></p>
<p><strong>TL;DR -</strong> An unattended zlib update made Ubuntu's needrestart restart Patroni in the middle of a load, causing a failover, and then a second one when the new primary was patched. Includes the journal-based root-cause analysis.</p>
<p><strong><a href="https://planetscale.com/blog/the-hidden-latency-inside-your-database">The hidden latency inside your database</a></strong><br>
<em>Simeon Griggs, PlanetScale Blog</em></p>
<p><strong>TL;DR -</strong> Network-attached storage puts a network hop inside every database I/O. This piece explains when durable local NVMe pays off and when network storage is fine.</p>
<p><strong><a href="https://jfg-mysql.blogspot.com/2026/11/mariadb-technically-correct-but-misleading-performance-claim.html">MariaDB Performance Claim : Technically Correct, but Misleading</a></strong><br>
<em>Jean-François Gagné</em></p>
<p><strong>TL;DR -</strong> A response to last week's "MariaDB 11.8 became about 3× faster" post. The fix recovers a regression from 10.5, so against 10.4 the gain is about 9%, and the author argues for fairer vendor claims across the ecosystem.</p>
<p><strong><a href="https://www.databricks.com/blog/lakebase-postgres-now-generally-available-google-cloud">Lakebase Postgres is now generally available on Google Cloud</a></strong><br>
<em>Cassie Murray, Databricks Blog</em></p>
<p><strong>TL;DR -</strong> Databricks' serverless Postgres, with autoscaling, scale to zero, instant branching and PITR on storage shared with the lakehouse, is now GA on Google Cloud.</p>
<p><strong><a href="https://mariadb.org/still-on-mariadb-10-6-why-your-next-upgrade-is-a-security-decision/">Still on MariaDB 10.6? Why your next upgrade is a security decision</a></strong><br>
<em>Anna Widenius, MariaDB Foundation Blog</em></p>
<p><strong>TL;DR -</strong> MariaDB Community Server 10.6 reached end of life on 6 July 2026, and 10.6.28 was its final release. Security fixes now land only in 10.11, 11.4, 11.8 and 12.3.</p>
<p><strong><a href="https://supabase.com/blog/introducing-dbarena">Introducing DBArena: open, reproducible benchmarks for hosted Postgres</a></strong><br>
<em>Aditya Maruvada and Daniel Mitterdorfer, Supabase Blog</em></p>
<p><strong>TL;DR -</strong> This Supabase-sponsored site benchmarks Supabase (heap and OrioleDB), RDS and Cloud SQL Postgres at matched vCPU and I/O, normalized for price. The methodology and tooling are open source.</p>
<h2>Upcoming Database Events</h2>
<p><strong><a href="https://blogs.oracle.com/mysql/join-mysql-public-discussion-6-and-the-november-contributor-summit">MySQL Contributor Summit</a></strong><br>
November 4-5, 2026<br>
Virtual</p>
<p><strong><a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/">KubeCon + CloudNativeCon North America</a></strong> (<strong>VillageSQL is a sponsor</strong>)<br>
November 9-12, 2026<br>
Salt Lake City, Utah</p>
<p><strong><a href="https://passdatacommunitysummit.com/">PASS Data Community Summit</a></strong><br>
November 9-11, 2026<br>
Hyatt Regency Seattle<br>
Seattle, WA</p>
<p><strong><a href="https://2026.pgday.com.mx/">pgDay México 2026</a></strong><br>
November 13, 2026<br>
Puerta Polanco<br>
Mexico City, Mexico</p>
<p><strong><a href="https://oursqlfoundation.org/events/duckdb-mysql-day/">DuckDB × MySQL Day</a></strong><br>
November 17, 2026<br>
Online</p>
<p><strong><a href="https://www.pgconf.asia/EN/2026/">PGConf.Asia 2026</a></strong><br>
November 17-18, 2026<br>
Hong Kong</p>
<p><strong><a href="https://oursqlfoundation.org/community/webinars/mysql-query-cache/">OurSQL Foundation Webinar: What Happened to the MySQL Query Cache?</a></strong><br>
November 18, 2026, 10:00 AM ET / 16:00 CET<br>
Online</p>
<p><strong><a href="https://2026.pgconf.org.np/">PgConf.Nepal 2026</a></strong><br>
November 18-21, 2026<br>
Pulchowk Campus, Tribhuvan University<br>
Kathmandu, Nepal</p>
<p><strong><a href="https://2026.pgconf.pl/">PGConf.PL 2026</a></strong><br>
November 24, 2026<br>
ARCHE Dwór Uphagena<br>
Gdańsk, Poland</p>
<p><strong><a href="https://www.postgresql.jp/jpug-pgcon2026">PostgreSQL Conference Japan 2026</a></strong><br>
November 27, 2026<br>
AP Tokyo Yaesu<br>
Tokyo, Japan</p>
<p><strong><a href="https://aws.amazon.com/events/reinvent">AWS re:Invent 2026</a></strong><br>
November 30 – December 4, 2026<br>
Las Vegas, NV</p>
<p><strong><a href="https://oursqlfoundation.org/community/webinars/fastest-database-hammerdb/">OurSQL Foundation Webinar: What Is the Fastest Database? Open-Source Benchmarking with HammerDB</a></strong><br>
December 3, 2026, 10:00 AM ET / 16:00 CET<br>
Online</p>
<p><strong><a href="https://events.linuxfoundation.org/open-source-summit-japan/">Open Source Summit Japan</a></strong><br>
December 7-9, 2026<br>
Tokyo, Japan</p>
<p><strong><a href="https://www.cidrdb.org/cidr2027/">CIDR 2027</a></strong><br>
January 24-27, 2027<br>
Mövenpick Hotel Amsterdam City Centre<br>
Amsterdam, The Netherlands</p>
<p><strong><a href="https://sessionize.com/mysql-belgian-days-2027/">MySQL Belgian Days 2027</a></strong><br>
January 28-29, 2027<br>
ICAB Incubator<br>
Brussels, Belgium</p>
<p><strong><a href="https://2027.fosdempgday.org/">FOSDEM PGDay 2027</a></strong><br>
January 28-29, 2027<br>
Brussels Marriott Hotel Grand Place<br>
Brussels, Belgium</p>
<p><strong><a href="https://fosdem.org/2027/">FOSDEM 2027</a></strong><br>
January 30-31, 2027<br>
ULB Solbosch Campus<br>
Brussels, Belgium</p>
<p><strong><a href="https://indico.cern.ch/event/1720940/">CERN PGDay 2027</a></strong><br>
February 12, 2027<br>
CERN Council Chamber<br>
Geneva, Switzerland</p>
<p><strong><a href="https://pgconf.in/">PGConf India 2027</a></strong><br>
March 2-5, 2027<br>
Sheraton Grand Hotel at Brigade Gateway<br>
Bengaluru, India</p>
<p><strong><a href="https://www.postgresql.org/about/event/fossasia-pgday-2027-returns-to-thailand-on-march-9-cfp-open-2611/">FOSSASIA PGDay 2027</a></strong><br>
March 9, 2027<br>
Bangkok, Thailand</p>
<p><strong><a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/">KubeCon + CloudNativeCon Europe 2027</a></strong><br>
March 15-18, 2027<br>
Barcelona, Spain</p>
<p><strong><a href="https://2027.nordicpgday.org/">Nordic PGDay 2027</a></strong><br>
March 16, 2027<br>
Courtyard Kungsholmen<br>
Stockholm, Sweden</p>
<p><strong><a href="https://www.socallinuxexpo.org/scale/24x">SCaLE 24x</a></strong><br>
April 1-4, 2027<br>
Pasadena Convention Center<br>
Pasadena, CA</p>
<p><strong><a href="https://duckdb.org/events/2027/04/07/duckcon8/">DuckCon #8</a></strong><br>
April 7, 2027<br>
De Wester (Westergas)<br>
Amsterdam, The Netherlands</p>
<p><strong><a href="https://2027.pgdayboston.org/">PgDay Boston 2027</a></strong><br>
April 10, 2027<br>
Tufts University Joyce Cummings Center<br>
Boston, MA</p>
<p><strong><a href="https://postgresconf.org/conferences/2027">Postgres Conference 2027</a></strong><br>
April 21-23, 2027<br>
San Jose Marriott<br>
San Jose, CA</p>
<p><strong><a href="https://2027.pgconf.dev/">PGConf.dev 2027</a></strong><br>
May 11-14, 2027<br>
Plaza Centre-Ville<br>
Montréal, QC, Canada</p>
<p><strong><a href="https://perconalive.com/">Percona Live Bay Area 2027</a></strong><br>
May 11-13, 2027<br>
Computer History Museum<br>
Mountain View, CA</p>]]></content:encoded>
    <pubDate>Thu, 05 Nov 2026 14:40:46 +0000</pubDate>
    <dc:creator>VillageSQL</dc:creator>
  </item>

  <item>
    <title>MySQL Community Spotlight - Jonathan Desrosiers</title>
    <guid isPermaLink="false">planet-manual-6e48f529580e247f804f1fdb9397e250</guid>
    <link>https://github.com/mysql/mysql-community/wiki/Jonathan-Desrosiers</link>
    <description>Jonathan shares how he discovered MySQL through building websites and WordPress, his experience at the EU Summit for the MySQL Community, and the challenges of balancing backwards compatibility with modern MySQL features.</description>
    <pubDate>Wed, 04 Nov 2026 19:02:17 +0000</pubDate>
    <dc:creator>MySQL Community Team</dc:creator>
  </item>

  <item>
    <title>MySQL High Availability and Data Replication Made Easy: Tungsten on AWS and Google Cloud</title>
    <guid isPermaLink="false">2063 at https://www.continuent.com</guid>
    <link>https://www.continuent.com/resources/blog/mysql-high-availability-and-data-replication-made-easy-tungsten-aws-and-google-cloud</link>
    <description>Deploy MySQL high availability and data replication on AWS and Google Cloud with Tungsten Clustering and Replicator v8.0.4. Explore automated failover, disaster recovery, cross-platform replication, and simplified deployment through cloud marketplaces.</description>
    <pubDate>Wed, 04 Nov 2026 16:35:10 +0000</pubDate>
    <dc:creator>Continuent</dc:creator>
  </item>

  <item>
    <title>Engineered to Run Agents and Apps: Join Nutanix at KubeCon + CloudNativeCon NA 2026</title>
    <guid isPermaLink="false">https://www.nutanix.com/blog/engineered-to-run-agents-and-apps</guid>
    <link>https://www.nutanix.com/blog/engineered-to-run-agents-and-apps</link>
    <description>The cloud native community is gearing up for KubeCon + CloudNativeCon North America 2026 in Salt Lake City from November 9–12, and Nutanix is proud to be a Diamond Sponsor.</description>
    <pubDate>Wed, 04 Nov 2026 09:45:00 +0000</pubDate>
    <dc:creator>Nutanix</dc:creator>
    <category>Blog Category</category>
    <category>Nutanix Database Service (NDB)</category>
  </item>

  <item>
    <title>Vinicius Grippa: “MySQL Is Not Dead, But It Needs Physiotherapy”</title>
    <guid isPermaLink="false">https://oursqlfoundation.org/blog/vinicius-grippa-interview/</guid>
    <link>https://oursqlfoundation.org/blog/vinicius-grippa-interview/</link>
    <description>In January, Vinicius Grippa publicly asked whether MySQL was dead. Ten months later we asked him again and talked through what MySQL does well, what it’s still missing, and who is filling the gaps.
Grippa spent years in Percona Support, a job where, in his words, you learn “the community, open source, the technology itself, and how software evolves” all at once. Today he works at Readyset, a caching layer for MySQL and Postgres. He also maintains the open source tool MyFlames, is an Oracle ACE Director, and helps run the MySQL community in Brazil.</description>
    <content:encoded><![CDATA[<p><em>In January, Vinicius Grippa <a href="https://comunidademysql.com.br/mysql-abandonado-fud-realidade-oracle/">publicly asked whether MySQL was dead</a>. Ten months later we asked him again and talked through what MySQL does well, what it’s still missing, and who is filling the gaps.</em></p>
<p><em>Grippa spent years in Percona Support, a job where, in his words, you learn “the community, open source, the technology itself, and how software evolves” all at once. Today he works at Readyset, a caching layer for MySQL and Postgres. He also maintains the open source tool MyFlames, is an Oracle ACE Director, and helps run the MySQL community in Brazil.</em></p>]]></content:encoded>
    <pubDate>Wed, 04 Nov 2026 00:00:00 +0000</pubDate>
    <dc:creator>OurSQL Foundation</dc:creator>
  </item>

  <item>
    <title>MySQL Shorts -  Episode 105 - MySQL Null-Safe Equality Operator for Accurate Filtering</title>
    <guid isPermaLink="false">planet-manual-6b3c22c2522e93b0218fc7586b782415</guid>
    <link>https://www.youtube.com/watch?v=2qBJPYaMzwU&amp;list=PLWx5a9Tn2EvG4C90YFJ9eU61IpALeE0SN</link>
    <description>MySQL's null-safe equality operator helps you write SQL queries that correctly match NULL values and retrieve records that ordinary equality comparisons miss. See how to filter assigned and unassigned support tickets using null-safe comparisons, and explore the operator's syntax and behavior in the MySQL Reference Manual: https://dev.mysql.com/doc/refman/9.7/...

Learn how the MySQL null-safe equality operator handles NULL values in SQL queries and helps retrieve records that standard equality comparisons can miss. This demo explains the difference between ordinary equality (=) and null-safe equality, showing how each operator evaluates NULL values and affects query results.</description>
    <pubDate>Tue, 03 Nov 2026 14:13:41 +0000</pubDate>
    <dc:creator>MySQL Community Team</dc:creator>
  </item>

  <item>
    <title>MySQL Community Spotlight - Vinicius Grippa</title>
    <guid isPermaLink="false">planet-manual-20a52328a2fbf21c828b41ae05823356</guid>
    <link>https://github.com/mysql/mysql-community/wiki/Vinicius-Grippa</link>
    <description>Vinicius shares how he got started with MySQL, the community members who inspire him, and his enthusiasm for performance troubleshooting, community events, and AI.</description>
    <pubDate>Tue, 03 Nov 2026 12:30:02 +0000</pubDate>
    <dc:creator>MySQL Community Team</dc:creator>
  </item>

  <item>
    <title>FOSDEM 2027: Bringing the M-Ecosystem Together</title>
    <guid isPermaLink="false">planet-manual-a4907711f9b6a970ad6374de618fa0ab</guid>
    <link>https://mariadb.org/fosdem-2027-bringing-the-m-ecosystem-together/</link>
    <description>One ecosystem, many projects, shared challenges. Join us in Brussels on Sunday, January 31, 2027, for the M-Ecosystem Devroom at FOSDEM!

Building on the tradition of the MariaDB, MySQL &amp; Friends Devroom, this new chapter expands its scope and ambition, bringing together developers and contributors from across the open-source database ecosystem to share knowledge, tackle common challenges, and foster collaboration across projects.</description>
    <content:encoded><![CDATA[One ecosystem, many projects, shared challenges. Join us in Brussels on Sunday, January 31, 2027, for the M-Ecosystem Devroom at FOSDEM!

Building on the tradition of the MariaDB, MySQL & Friends Devroom, this new chapter expands its scope and ambition, bringing together developers and contributors from across the open-source database ecosystem to share knowledge, tackle common challenges, and foster collaboration across projects.]]></content:encoded>
    <pubDate>Tue, 03 Nov 2026 12:04:58 +0000</pubDate>
    <dc:creator>MariaDB</dc:creator>
  </item>

  <item>
    <title>A New KMIP Library in C++, and the Only Free One</title>
    <guid isPermaLink="false">https://percona.community/blog/2026/11/03/kmipclient-new-kmip-library-in-cpp/</guid>
    <link>https://percona.community/blog/2026/11/03/kmipclient-new-kmip-library-in-cpp/</link>
    <description>Percona has open sourced kmipclient, a new KMIP library in C++ for applications that need to talk to a key management system over a standard protocol instead of a single vendor’s proprietary API.
Encryption is one of the fundamental tools for protecting sensitive data. But encryption is only part of the problem. The keys that unlock that data need to be protected separately from the data itself.
If an attacker gets access to both the encrypted data and the keys required to decrypt it, encryption does not buy much. That is a bit like taping your house key to the front door: the lock is real, but it is not doing much work. Keeping keys under their own administrative and security boundary also makes key rotation, access control, auditing, and revocation easier to manage.
This is why dedicated key management systems (KMS) exist. A KMS keeps cryptographic keys in a separate security domain and gives applications controlled access to them, so an application can store encrypted data while the keys stay under the control of the key management infrastructure.
There is another problem, though: different KMS products expose different APIs.
The Key Management Interoperability Protocol (KMIP) addresses that by defining a standard protocol for communication between applications and key management systems. KMIP is developed by the KMIP Technical Committee at OASIS (the Organization for the Advancement of Structured Information Standards), which publishes the KMIP specifications, profiles, and test cases. The protocol has evolved through several versions, from KMIP 1.0 through 1.4 and the 2.x series, with KMIP 3.0 now in development.
Several open source KMIP implementations already exist, in Python, Go, and C. We searched extensively and found none of them native C++: libkmip itself is a C library, and the handful of C++-adjacent projects out there are either proprietary SDKs or early-stage work with no C++ bindings yet. What was missing was a modern, freely available native C++ implementation with a clean API, current KMIP support, and an architecture that would make further development reasonably painless.
So we built one. kmipclient is built on top of its own protocol library, kmipcore, and tested against five independent KMIP-capable servers.
Why build a new KMIP library in C++
At the time, Percona was already maintaining and extending an existing KMIP implementation: a fork of libkmip, a C library, wrapped in a thin C++ layer called kmippp. (The old code is still there, on the legacy-libkmip branch, for anyone curious about the starting point.) It worked, but extending it was not much fun.
The problem was not that KMIP is difficult to implement. The problem was the shape of the C data model. libkmip allocates and frees every request and response type through its own dedicated pair of functions, so adding one KMIP operation means writing a new alloc function, a new free function, and a new parse function, then wiring all three into a C++ wrapper on top. Every small feature turned into a little archaeology dig: find the right allocator, find its matching free function, and hope you freed the thing you actually allocated. The specification implementation was minimal, and even a small addition was expensive to make correctly.
Adding support for the KMIP Activate and Deactivate operations was a good example. The work meant reading the specification, adding the missing functionality to libkmip, and then adding a matching wrapper in kmippp. Three layers, for one operation.
That was the point where it became clear: this was costing more in support and development than it should.
What was needed was simple to describe: code that was easier to understand, maintain, test, and extend, without fighting the architecture at every step. So a small weekend project started, not to replace the existing implementation immediately, but to see what a clean, maintainable, extensible native C++ KMIP implementation could look like.
The weekend project grew.
Splitting the protocol from the client
One of the early decisions was to separate the KMIP protocol from the client that talks to a server.
kmipcore is the protocol library. It holds the KMIP data model, the objects, requests and responses, and the TTLV (Tag-Type-Length-Value) serialization and parsing that KMIP uses on the wire. It has no networking code at all.
kmipclient is the client library built on top of kmipcore. It provides the higher-level API applications use, handles the communication layer, and adds features such as a thread-safe connection pool.
Networking and logging are both dependency injection points rather than fixed choices. Networking is abstracted behind a replaceable NetClient interface, with an OpenSSL-based implementation provided out of the box. Logging follows the same pattern: a Logger interface emits structured records that an application can route into its own logging sink instead of one bolted on by the library. Either way, an application can plug kmipclient into networking and logging infrastructure it already runs, rather than standing up a parallel one just for this library.
This separation turned out to matter more than expected. kmipcore can be tested on its own, with no KMS and no network connection involved. kmipclient can then concentrate on the practical problems of talking to a real server. And because kmipcore represents the KMIP protocol itself rather than any one client’s needs, the same library could also become the foundation for a KMIP server implementation later.
The diagram below shows what that split replaced. The old stack needed three layers in C and C++ to reach component_keyring_kmip, Percona Server for MySQL’s keyring component for KMIP. The new stack needs one: kmipcore, with kmipclient sitting on top of it.


Old stack on the left, new stack on the right. kmipcore is where new KMIP operations actually get implemented.
A separate utils module, still in progress, is growing alongside the two libraries to hold key and X.509 certificate handling and other shared helpers that do not belong in either the protocol or the client layer.
The first time a new operation took an afternoon instead of a week, the split already felt worth it.
The specification comes first
KMIP is a fairly large and detailed protocol, and interoperability depends on implementing what the specification says, not just what happens to work against one particular server.
The KMIP specification was the primary source of truth throughout the project. kmipclient supports KMIP 1.4 and KMIP 2.0, including protocol negotiation, while still letting an application select a protocol version explicitly when it needs to.
Real KMS implementations are useful for testing, but interoperability testing validates a specification-based implementation. It is not a replacement for reading the specification. A server accepting what you send does not prove that what you sent was correct according to KMIP, only that this particular server tolerated it.
Designed to be extended
Fully implementing the KMIP specification is a project of its own, and that was never the near-term goal. The practical goal was narrower: implement what is needed today, while keeping the cost of implementing the next piece of the specification as low as possible.
This is where the split between protocol and client pays off. KMIP objects and operations map directly to C++ types, and adding a new operation mostly means implementing the corresponding protocol objects in kmipcore, rather than touching several layers of an older C implementation and a separate C++ wrapper on top of it.
The top-level API is aimed at engineers who care about the result, not the protocol or the cryptography underneath it: activate a key, fetch one, rotate one, without reading the KMIP specification or knowing what a wrapping algorithm is. The protocol details are still there underneath, for the few call sites that actually need them.
The library also supports batching multiple request and response items into a single exchange, which high-level operations use directly. op_get_key(), for example, sends a Get and a Get Attributes request as one batch and returns both the key material and its attributes from a single round trip:

cpp
Copy
Copied!



auto key = client.op_get_key(id);
std::cout  &quot;Key: 0x&quot;;
print_hex(key-&amp;gt;value());
std::cout  &quot;Attributes:&quot;  std::endl;
for (const auto &amp;amp;[attr_name, attr_value] :
 key-&amp;gt;attributes().as_string_map()) {
 std::cout  &quot; &quot;  attr_name  &quot;: &quot;  attr_value  std::endl;
}


From kmipclient/examples/example_get.cpp.
High-level operations stay just as direct for simpler calls. Activating a key, for instance, is a single method call that returns the server-confirmed key ID or throws:

cpp
Copy
Copied!



std::string KmipClient::op_activate(const std::string &amp;amp;id) const {
 auto request = make_request_message();
 const auto batch_item_id =
 request.add_batch_item(kmipcore::ActivateRequest(id));

 kmipcore::secure_bytes response_bytes;
 io-&amp;gt;do_exchange(
 request.serialize(), response_bytes, request.getMaxResponseSize()
 );

 kmipcore::ResponseParser rf(response_bytes, request);
 return rf
 .getResponseByBatchItemIdkmipcore::ActivateResponseBatchItem&amp;gt;(
 batch_item_id
 )
 .getUniqueIdentifier();
}


From kmipclient/src/KmipClient.cpp.
The library also provides lower-level protocol access for applications that need something more specialized than the high-level API covers.
From a weekend project to production
Testing turned the prototype into a library other teams could depend on.
The test suite has two layers. Unit tests exercise the protocol and data model on their own, including serialization, parsing, and specification-conformance checks. Integration tests then run against five independent KMIP server implementations: PyKMIP (open source), Cosmian KMS (source-available), HashiCorp Vault Enterprise, Fortanix Data Security Manager, and Thales CipherTrust Manager.


Cosmian KMS runs in CI, so the integration suite exercises it on every change. CI started out against PyKMIP; the switch to Cosmian KMS came later, driven by its more complete coverage of the KMIP 2.0 specification. Cosmian KMS is source-available rather than open source: its code is public under the Business Source License (BUSL-1.1), which grants free use for non-production work like a CI pipeline outright, free production use up to two vCPUs, and converts to the Mozilla Public License (MPL) 2.0, an open source license, four years after each version’s publication.
The other four servers are not practical to spin up on every commit, so Percona’s QA engineers run the full integration suite against all five servers by hand after each update, in addition to the automated CI run. Every kmipclient release is verified against all five this way, which gives an interoperability matrix and catches regressions that unit tests and CI alone would miss.
kmipclient now backs component_keyring_kmip in Percona Server for MySQL’s KMIP keyring component, shipping in the 8.4.10 and 8.4.11, 9.7.1 and 9.7.2, and 26.10 releases, with an MTR (MySQL Test Run) suite that exercises it against a Cosmian KMS instance in Docker. It is a general-purpose KMIP client library, though: MySQL was its first important user, not the reason it exists. Teams who previously set up Percona Server for MySQL with HashiCorp Vault Enterprise’s KMIP secrets engine through our earlier keyring plugin walkthrough are now running on top of this library.
Growing through real-world contributions
The project stopped being a one-person exercise early on.
Thanks are due to several Percona engineers who helped along the way: colleagues from the MongoDB team gave an early hand with KMIP itself and with getting test servers running; colleagues from the PostgreSQL team contributed fixes and improvements covering secure handling of sensitive data, protocol correctness, and network robustness; colleagues from the MySQL team reviewed the library’s design, API, and usability; and colleagues from the QA team have done extensive testing of both the library and the MySQL KMIP keyring built on it, across all five supported KMIP servers.
Contributions crossing three product teams and QA, for a library that started as a weekend project on one engineer’s laptop, says something about how real integration work exposes problems that are hard to find alone. The code gets reviewed, exercised, and occasionally broken in ways a single developer’s test environment rarely produces, which is one of the more effective forms of testing there is.
How AI fit into the process
A significant part of the implementation was done with AI coding assistance, mainly GitHub Copilot. The architecture and API were designed by hand first; AI did a lot of the repetitive implementation work afterward.
The approach was deliberately incremental. The KMIP specifications went in first, so the protocol itself was the primary source of truth, not an impression of what the existing C code already did. The existing C library went in next, for practical context on functionality already in use. Then came a hand-written prototype with the new API, interfaces, stubs, and implementation skeletons already defined.
From there, the process followed a pattern: explain the architecture and goals, check that the model’s understanding matched, have it write documentation, then read that documentation back and fix the inconsistencies it surfaced. Development plans came next, broken into a rough work breakdown, followed by the API itself through several rounds of review and revision. Implementation went step by step, function by function, with every step checked rather than accepted on faith.
AI also served as an additional code-review pass: a colleague ran the implementation through several different models for a second set of perspectives. That is not a substitute for human engineering judgment, though. Another colleague reviewed the complete library the traditional way, looking past individual pieces of code at the overall design, API, usability, and maintainability.
AI did a lot of the typing. It did not design the system or make the engineering calls.
There is an irony worth admitting here. Extending libkmip by hand, allocator by allocator, is exactly what made the old architecture feel so expensive in the first place. That kind of grind, hunting down the right alloc function and its matching free function, is precisely the work an AI assistant handles well today. Started now, with these tools already in hand, the choice might not be to write a new library at all. It might be to keep extending the old one.
What’s next for kmipclient
The project is not done. A few things are already in motion: folding shared helpers into the utils module and tidying up the key-related API, extending the operation API further, rounding out X.509 certificate support, and filling in more of the KMIP specification. A secure memory de-allocation change is currently in review, and memory allocation and general performance work is next on the list.
None of that changes the underlying goal: implement what applications need now, while keeping the next addition to the specification cheap.
Try the new KMIP library in C++
kmipclient is a native, open source C++ implementation of KMIP, built on kmipcore, its underlying protocol library. It started as a weekend project aimed at making KMIP development in C++ easier to understand, test, and extend. It has since become a production component in Percona Server for MySQL, tested against five independent KMIP server implementations, with contributions from engineers across three product teams.
It still is not an attempt to implement every corner of the KMIP specification at once. It is a foundation for the functionality applications need today, built so the next piece costs less than the last one did.
Building it requires a C++20 compiler, CMake 3.10 or newer, and OpenSSL. The kmipclient keeps the same dual Apache-2.0/BSD license that libkmip used, out of respect for libkmip’s original author and for the projects already depending on that license.
The project is open source and available on GitHub: percona/kmipclient. Issues, questions, and contributions are welcome.
And, as is usually true of software, it is not quite finished yet.

Related reading

KMIP keyring component in Percona Server for MySQL
Setting up Percona Server for MySQL with HashiCorp Vault Enterprise’s KMIP secrets engine
kmipclient on GitHub
</description>
    <content:encoded><![CDATA[<p>Percona has open sourced <code>kmipclient</code>, a new KMIP library in C++ for applications that need to talk to a key management system over a standard protocol instead of a single vendor’s proprietary API.</p>
<p>Encryption is one of the fundamental tools for protecting sensitive data. But encryption is only part of the problem. The keys that unlock that data need to be protected separately from the data itself.</p>
<p>If an attacker gets access to both the encrypted data and the keys required to decrypt it, encryption does not buy much. That is a bit like taping your house key to the front door: the lock is real, but it is not doing much work. Keeping keys under their own administrative and security boundary also makes key rotation, access control, auditing, and revocation easier to manage.</p>
<p>This is why dedicated <strong>key management systems (KMS)</strong> exist. A KMS keeps cryptographic keys in a separate security domain and gives applications controlled access to them, so an application can store encrypted data while the keys stay under the control of the key management infrastructure.</p>
<p>There is another problem, though: different KMS products expose different APIs.</p>
<p>The <strong>Key Management Interoperability Protocol (KMIP)</strong> addresses that by defining a standard protocol for communication between applications and key management systems. KMIP is developed by the <a href="https://www.oasis-open.org/committees/kmip/" target="_blank" rel="noopener noreferrer">KMIP Technical Committee at OASIS</a> (the Organization for the Advancement of Structured Information Standards), which publishes the KMIP specifications, profiles, and test cases. The protocol has evolved through several versions, from KMIP 1.0 through 1.4 and the 2.x series, with <a href="https://docs.oasis-open.org/kmip/kmip-spec/v3.0/kmip-spec-v3.0.html" target="_blank" rel="noopener noreferrer">KMIP 3.0 now in development</a>.</p>
<p>Several open source KMIP implementations already exist, in Python, Go, and C. We searched extensively and found none of them native C++: <code>libkmip</code> itself is a C library, and the handful of C++-adjacent projects out there are either proprietary SDKs or early-stage work with no C++ bindings yet. What was missing was a modern, freely available native C++ implementation with a clean API, current KMIP support, and an architecture that would make further development reasonably painless.</p>
<p>So we built one. <code>kmipclient</code> is built on top of its own protocol library, <code>kmipcore</code>, and tested against five independent KMIP-capable servers.</p>
<h2>Why build a new KMIP library in C++</h2>
<p>At the time, Percona was already maintaining and extending an existing KMIP implementation: a fork of <code>libkmip</code>, a C library, wrapped in a thin C++ layer called <code>kmippp</code>. (The old code is still there, on the <a href="https://github.com/percona/kmipclient/tree/legacy-libkmip" target="_blank" rel="noopener noreferrer"><code>legacy-libkmip</code> branch</a>, for anyone curious about the starting point.) It worked, but extending it was not much fun.</p>
<p>The problem was not that KMIP is difficult to implement. The problem was the shape of the C data model. <code>libkmip</code> allocates and frees every request and response type through its own dedicated pair of functions, so adding one KMIP operation means writing a new alloc function, a new free function, and a new parse function, then wiring all three into a C++ wrapper on top. Every small feature turned into a little archaeology dig: find the right allocator, find its matching free function, and hope you freed the thing you actually allocated. The specification implementation was minimal, and even a small addition was expensive to make correctly.</p>
<p>Adding support for the KMIP <strong>Activate</strong> and <strong>Deactivate</strong> operations was a good example. The work meant reading the specification, adding the missing functionality to <code>libkmip</code>, and then adding a matching wrapper in <code>kmippp</code>. Three layers, for one operation.</p>
<p>That was the point where it became clear: this was costing more in support and development than it should.</p>
<p>What was needed was simple to describe: code that was easier to understand, maintain, test, and extend, without fighting the architecture at every step. So a small weekend project started, not to replace the existing implementation immediately, but to see what a clean, maintainable, extensible native C++ KMIP implementation could look like.</p>
<p>The weekend project grew.</p>
<h2>Splitting the protocol from the client</h2>
<p>One of the early decisions was to separate the <strong>KMIP protocol</strong> from the <strong>client</strong> that talks to a server.</p>
<p><code>kmipcore</code> is the protocol library. It holds the KMIP data model, the objects, requests and responses, and the TTLV (Tag-Type-Length-Value) serialization and parsing that KMIP uses on the wire. It has no networking code at all.</p>
<p><code>kmipclient</code> is the client library built on top of <code>kmipcore</code>. It provides the higher-level API applications use, handles the communication layer, and adds features such as a thread-safe connection pool.</p>
<p>Networking and logging are both dependency injection points rather than fixed choices. Networking is abstracted behind a replaceable <code>NetClient</code> interface, with an OpenSSL-based implementation provided out of the box. Logging follows the same pattern: a <code>Logger</code> interface emits structured records that an application can route into its own logging sink instead of one bolted on by the library. Either way, an application can plug <code>kmipclient</code> into networking and logging infrastructure it already runs, rather than standing up a parallel one just for this library.</p>
<p>This separation turned out to matter more than expected. <code>kmipcore</code> can be tested on its own, with no KMS and no network connection involved. <code>kmipclient</code> can then concentrate on the practical problems of talking to a real server. And because <code>kmipcore</code> represents the KMIP protocol itself rather than any one client’s needs, the same library could also become the foundation for a KMIP server implementation later.</p>
<p>The diagram below shows what that split replaced. The old stack needed three layers in C and C++ to reach <code>component_keyring_kmip</code>, Percona Server for MySQL’s keyring component for KMIP. The new stack needs one: <code>kmipcore</code>, with <code>kmipclient</code> sitting on top of it.</p>
<p>
<figure><img src="https://percona.community/blog/2026/11/kmipclient-architecture-old-vs-new.svg" alt="Old C stack (libkmip, kmip_bio, kmippp, three layers) versus new C++ stack (kmipcore, kmipclient, one layer) feeding into Percona Server for MySQL’s component_keyring_kmip"></figure></p>
<p><em>Old stack on the left, new stack on the right. <code>kmipcore</code> is where new KMIP operations actually get implemented.</em></p>
<p>A separate <code>utils</code> module, still in progress, is growing alongside the two libraries to hold key and X.509 certificate handling and other shared helpers that do not belong in either the protocol or the client layer.</p>
<p>The first time a new operation took an afternoon instead of a week, the split already felt worth it.</p>
<h2>The specification comes first</h2>
<p>KMIP is a fairly large and detailed protocol, and interoperability depends on implementing what the specification says, not just what happens to work against one particular server.</p>
<p>The <a href="https://www.oasis-open.org/committees/kmip/" target="_blank" rel="noopener noreferrer">KMIP specification</a> was the primary source of truth throughout the project. <code>kmipclient</code> supports <a href="https://docs.oasis-open.org/kmip/spec/v1.4/kmip-spec-v1.4.html" target="_blank" rel="noopener noreferrer">KMIP 1.4</a> and <a href="https://docs.oasis-open.org/kmip/kmip-spec/v2.0/kmip-spec-v2.0.html" target="_blank" rel="noopener noreferrer">KMIP 2.0</a>, including protocol negotiation, while still letting an application select a protocol version explicitly when it needs to.</p>
<p>Real KMS implementations are useful for testing, but interoperability testing validates a specification-based implementation. It is not a replacement for reading the specification. A server accepting what you send does not prove that what you sent was correct according to KMIP, only that this particular server tolerated it.</p>
<h2>Designed to be extended</h2>
<p>Fully implementing the KMIP specification is a project of its own, and that was never the near-term goal. The practical goal was narrower: implement what is needed today, while keeping the cost of implementing the next piece of the specification as low as possible.</p>
<p>This is where the split between protocol and client pays off. KMIP objects and operations map directly to C++ types, and adding a new operation mostly means implementing the corresponding protocol objects in <code>kmipcore</code>, rather than touching several layers of an older C implementation and a separate C++ wrapper on top of it.</p>
<p>The top-level API is aimed at engineers who care about the result, not the protocol or the cryptography underneath it: activate a key, fetch one, rotate one, without reading the KMIP specification or knowing what a wrapping algorithm is. The protocol details are still there underneath, for the few call sites that actually need them.</p>
<p>The library also supports batching multiple request and response items into a single exchange, which high-level operations use directly. <code>op_get_key()</code>, for example, sends a Get and a Get Attributes request as one batch and returns both the key material and its attributes from a single round trip:</p>
<div class="code-block">
<div class="code-block__header"><span class="code-block__lang">cpp</span><button class="code-block__copy" type="button" data-copy-target="codeblock-0" aria-label="Copy code to clipboard">
<span class="code-block__copy-default">Copy</span>
<span class="code-block__copy-success" aria-hidden="true">Copied!</span>
</button>
</div>
<div class="code-block__content">
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-cpp" data-lang="cpp"><span class="line"><span class="cl"><span class="k">auto</span> <span class="n">key</span> <span class="o">=</span> <span class="n">client</span><span class="p">.</span><span class="n">op_get_key</span><span class="p">(</span><span class="n">id</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"> <span class="s">"Key: 0x"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="n">print_hex</span><span class="p">(</span><span class="n">key</span><span class="o">-&gt;</span><span class="n">value</span><span class="p">());</span>
</span></span><span class="line"><span class="cl"><span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"> <span class="s">"Attributes:"</span> <span class="o"> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> <span class="p">(</span><span class="k">const</span> <span class="k">auto</span> <span class="o">&amp;</span><span class="p">[</span><span class="n">attr_name</span><span class="p">,</span> <span class="n">attr_value</span><span class="p">]</span> <span class="o">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">key</span><span class="o">-&gt;</span><span class="n">attributes</span><span class="p">().</span><span class="n">as_string_map</span><span class="p">())</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"> <span class="s">" "</span> <span class="o"> <span class="n">attr_name</span> <span class="o"> <span class="s">": "</span> <span class="o"> <span class="n">attr_value</span> <span class="o"> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></span></span></span></span></span></span></span></span></code></pre></div>
</div>
</div>
<p><em>From <code>kmipclient/examples/example_get.cpp</code>.</em></p>
<p>High-level operations stay just as direct for simpler calls. Activating a key, for instance, is a single method call that returns the server-confirmed key ID or throws:</p>
<div class="code-block">
<div class="code-block__header"><span class="code-block__lang">cpp</span><button class="code-block__copy" type="button" data-copy-target="codeblock-1" aria-label="Copy code to clipboard">
<span class="code-block__copy-default">Copy</span>
<span class="code-block__copy-success" aria-hidden="true">Copied!</span>
</button>
</div>
<div class="code-block__content">
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-cpp" data-lang="cpp"><span class="line"><span class="cl"><span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">KmipClient</span><span class="o">::</span><span class="n">op_activate</span><span class="p">(</span><span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="o">&amp;</span><span class="n">id</span><span class="p">)</span> <span class="k">const</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="k">auto</span> <span class="n">request</span> <span class="o">=</span> <span class="n">make_request_message</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"> <span class="k">const</span> <span class="k">auto</span> <span class="n">batch_item_id</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl"> <span class="n">request</span><span class="p">.</span><span class="n">add_batch_item</span><span class="p">(</span><span class="n">kmipcore</span><span class="o">::</span><span class="n">ActivateRequest</span><span class="p">(</span><span class="n">id</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">kmipcore</span><span class="o">::</span><span class="n">secure_bytes</span> <span class="n">response_bytes</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"> <span class="n">io</span><span class="o">-&gt;</span><span class="n">do_exchange</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">request</span><span class="p">.</span><span class="n">serialize</span><span class="p">(),</span> <span class="n">response_bytes</span><span class="p">,</span> <span class="n">request</span><span class="p">.</span><span class="n">getMaxResponseSize</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">kmipcore</span><span class="o">::</span><span class="n">ResponseParser</span> <span class="n">rf</span><span class="p">(</span><span class="n">response_bytes</span><span class="p">,</span> <span class="n">request</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">rf</span>
</span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="n">getResponseByBatchItemId</span><span class="o"><span class="n">kmipcore</span><span class="o">::</span><span class="n">ActivateResponseBatchItem</span><span class="o">&gt;</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">batch_item_id</span>
</span></span><span class="line"><span class="cl"> <span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="n">getUniqueIdentifier</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></span></code></pre></div>
</div>
</div>
<p><em>From <code>kmipclient/src/KmipClient.cpp</code>.</em></p>
<p>The library also provides lower-level protocol access for applications that need something more specialized than the high-level API covers.</p>
<h2>From a weekend project to production</h2>
<p>Testing turned the prototype into a library other teams could depend on.</p>
<p>The test suite has two layers. Unit tests exercise the protocol and data model on their own, including serialization, parsing, and specification-conformance checks. Integration tests then run against five independent KMIP server implementations: <a href="https://github.com/OpenKMIP/PyKMIP" target="_blank" rel="noopener noreferrer">PyKMIP</a> (open source), <a href="https://github.com/Cosmian/kms" target="_blank" rel="noopener noreferrer">Cosmian KMS</a> (source-available), HashiCorp Vault Enterprise, Fortanix Data Security Manager, and Thales CipherTrust Manager.</p>
<p>
<figure><img src="https://percona.community/blog/2026/11/kmipclient-testing-pipeline.svg" alt="Diagram showing unit tests feeding into integration tests, which split into an automated CI pipeline running Cosmian KMS on every commit, and a manual QA pass running all five KMIP servers on every release"></figure></p>
<p>Cosmian KMS runs in CI, so the integration suite exercises it on every change. CI started out against PyKMIP; the switch to Cosmian KMS came later, driven by its more complete coverage of the KMIP 2.0 specification. Cosmian KMS is source-available rather than open source: its code is public under the Business Source License (BUSL-1.1), which grants free use for non-production work like a CI pipeline outright, free production use up to two vCPUs, and converts to the Mozilla Public License (MPL) 2.0, an open source license, four years after each version’s publication.</p>
<p>The other four servers are not practical to spin up on every commit, so Percona’s QA engineers run the full integration suite against all five servers by hand after each update, in addition to the automated CI run. Every <code>kmipclient</code> release is verified against all five this way, which gives an interoperability matrix and catches regressions that unit tests and CI alone would miss.</p>
<p><code>kmipclient</code> now backs <code>component_keyring_kmip</code> in <a href="https://docs.percona.com/percona-server/8.0/using-kmip.html" target="_blank" rel="noopener noreferrer">Percona Server for MySQL’s KMIP keyring component</a>, shipping in the 8.4.10 and 8.4.11, 9.7.1 and 9.7.2, and 26.10 releases, with an MTR (MySQL Test Run) suite that exercises it against a Cosmian KMS instance in Docker. It is a general-purpose KMIP client library, though: MySQL was its first important user, not the reason it exists. Teams who previously set up Percona Server for MySQL with HashiCorp Vault Enterprise’s KMIP secrets engine through <a href="https://www.percona.com/blog/using-percona-server-for-mysql-8-0-and-percona-xtrabackup-8-0-with-hashicorp-vault-enterprise-kmip-secrets-engine/" target="_blank" rel="noopener noreferrer">our earlier keyring plugin walkthrough</a> are now running on top of this library.</p>
<h2>Growing through real-world contributions</h2>
<p>The project stopped being a one-person exercise early on.</p>
<p>Thanks are due to several Percona engineers who helped along the way: colleagues from the MongoDB team gave an early hand with KMIP itself and with getting test servers running; colleagues from the PostgreSQL team contributed fixes and improvements covering secure handling of sensitive data, protocol correctness, and network robustness; colleagues from the MySQL team reviewed the library’s design, API, and usability; and colleagues from the QA team have done extensive testing of both the library and the MySQL KMIP keyring built on it, across all five supported KMIP servers.</p>
<p>Contributions crossing three product teams and QA, for a library that started as a weekend project on one engineer’s laptop, says something about how real integration work exposes problems that are hard to find alone. The code gets reviewed, exercised, and occasionally broken in ways a single developer’s test environment rarely produces, which is one of the more effective forms of testing there is.</p>
<h2>How AI fit into the process</h2>
<p>A significant part of the implementation was done with AI coding assistance, mainly GitHub Copilot. The architecture and API were designed by hand first; AI did a lot of the repetitive implementation work afterward.</p>
<p>The approach was deliberately incremental. The KMIP specifications went in first, so the protocol itself was the primary source of truth, not an impression of what the existing C code already did. The existing C library went in next, for practical context on functionality already in use. Then came a hand-written prototype with the new API, interfaces, stubs, and implementation skeletons already defined.</p>
<p>From there, the process followed a pattern: explain the architecture and goals, check that the model’s understanding matched, have it write documentation, then read that documentation back and fix the inconsistencies it surfaced. Development plans came next, broken into a rough work breakdown, followed by the API itself through several rounds of review and revision. Implementation went step by step, function by function, with every step checked rather than accepted on faith.</p>
<p>AI also served as an additional code-review pass: a colleague ran the implementation through several different models for a second set of perspectives. That is not a substitute for human engineering judgment, though. Another colleague reviewed the complete library the traditional way, looking past individual pieces of code at the overall design, API, usability, and maintainability.</p>
<p>AI did a lot of the typing. It did not design the system or make the engineering calls.</p>
<p>There is an irony worth admitting here. Extending <code>libkmip</code> by hand, allocator by allocator, is exactly what made the old architecture feel so expensive in the first place. That kind of grind, hunting down the right alloc function and its matching free function, is precisely the work an AI assistant handles well today. Started now, with these tools already in hand, the choice might not be to write a new library at all. It might be to keep extending the old one.</p>
<h2>What’s next for kmipclient</h2>
<p>The project is not done. A few things are already in motion: folding shared helpers into the <code>utils</code> module and tidying up the key-related API, extending the operation API further, rounding out X.509 certificate support, and filling in more of the KMIP specification. A secure memory de-allocation change is currently in review, and memory allocation and general performance work is next on the list.</p>
<p>None of that changes the underlying goal: implement what applications need now, while keeping the next addition to the specification cheap.</p>
<h2>Try the new KMIP library in C++</h2>
<p><code>kmipclient</code> is a native, open source C++ implementation of KMIP, built on <code>kmipcore</code>, its underlying protocol library. It started as a weekend project aimed at making KMIP development in C++ easier to understand, test, and extend. It has since become a production component in Percona Server for MySQL, tested against five independent KMIP server implementations, with contributions from engineers across three product teams.</p>
<p>It still is not an attempt to implement every corner of the KMIP specification at once. It is a foundation for the functionality applications need today, built so the next piece costs less than the last one did.</p>
<p>Building it requires a C++20 compiler, CMake 3.10 or newer, and OpenSSL. The <code>kmipclient</code> keeps the same dual Apache-2.0/BSD license that <code>libkmip</code> used, out of respect for <code>libkmip</code>’s original author and for the projects already depending on that license.</p>
<p>The project is open source and available on GitHub: <strong><a href="https://github.com/percona/kmipclient" target="_blank" rel="noopener noreferrer">percona/kmipclient</a></strong>. Issues, questions, and contributions are welcome.</p>
<p>And, as is usually true of software, it is not quite finished yet.</p>
<hr>
<h2>Related reading</h2>
<ul>
<li><a href="https://docs.percona.com/percona-server/8.0/using-kmip.html" target="_blank" rel="noopener noreferrer">KMIP keyring component in Percona Server for MySQL</a></li>
<li><a href="https://www.percona.com/blog/using-percona-server-for-mysql-8-0-and-percona-xtrabackup-8-0-with-hashicorp-vault-enterprise-kmip-secrets-engine/" target="_blank" rel="noopener noreferrer">Setting up Percona Server for MySQL with HashiCorp Vault Enterprise’s KMIP secrets engine</a></li>
<li><a href="https://github.com/percona/kmipclient" target="_blank" rel="noopener noreferrer">kmipclient on GitHub</a></li>
</ul>]]></content:encoded>
    <pubDate>Tue, 03 Nov 2026 09:00:00 +0000</pubDate>
    <dc:creator>Percona Community</dc:creator>
    <category>KMIP</category>
    <category>Encryption</category>
    <category>Security</category>
    <category>C++</category>
    <category>Open Source</category>
    <category>MySQL</category>
    <category>Percona Server for MySQL</category>
  </item>

  <item>
    <title>Naming Things is Hard</title>
    <guid isPermaLink="false">planet-manual-be6417135552ad2c07e1a77b9cae6b16</guid>
    <link>https://stroz.dev/posts/2026/october/naming-is-hard/</link>
    <description>I shared a meme on LinkedIn because it made me laugh. Then I kept looking at the constraint name at the bottom.

PK__LAW__3214EC07B14E510A.

I would like to cooperate, officer, but I may need a minute to figure out what I did.

PK gives me a clue that this is a primary key constraint. LAW suggests the table involved. The rest gives me something to copy into a search box. A name such as pk_law_id would give me a much better starting point for understanding the violation.

That is a lot to ask of a meme, but it gets at something I care about when I build databases and write code: names should help the person reading them understand what is happening.</description>
    <pubDate>Tue, 03 Nov 2026 06:00:00 +0000</pubDate>
    <dc:creator>Scott Stroz</dc:creator>
  </item>

  <item>
    <title>MariaDB Performance Claim : Technically Correct, but Misleading</title>
    <guid isPermaLink="false">tag:blogger.com,1999:blog-9188714267863327820.post-6579759465038062932</guid>
    <link>https://jfg-mysql.blogspot.com/2026/11/mariadb-technically-correct-but-misleading-performance-claim.html</link>
    <description>A recent post on the MariaDB Foundation Blog has the impressive title How MariaDB 11.8 became about 3× faster on some in-memory workloads, fixing a regression dating back to 10.5.&amp;amp;nbsp; The claim is technically correct when viewed in isolation, but I believe it becomes misleading when the broader context is taken into account.&amp;amp;nbsp; The details are below.



The post mentions a benchmark on 11.8</description>
    <content:encoded><![CDATA[A recent post on the MariaDB Foundation Blog has the impressive title How MariaDB 11.8 became about 3× faster on some in-memory workloads, fixing a regression dating back to 10.5.&amp;nbsp; The claim is technically correct when viewed in isolation, but I believe it becomes misleading when the broader context is taken into account.&amp;nbsp; The details are below.



The post mentions a benchmark on 11.8]]></content:encoded>
    <pubDate>Mon, 02 Nov 2026 20:32:55 +0000</pubDate>
    <dc:creator>Jean-François Gagné</dc:creator>
    <category>Benchmarks</category>
    <category>M-Ecosystem</category>
    <category>MariaDB</category>
    <category>MySQL</category>
    <category>OurSQL Foundation</category>
    <category>Performance Regression</category>
  </item>

</channel>
</rss>
