Planet MySQL https://planet.mysql.com Sun, 25 Oct 2026 08:32:55 +0000 en Planet MySQL - https://planet.mysql.com Optimize OLTP and OLAP Workloads with MySQL HeatWave Autopilot 6dbe256dac1ed36c0e79d740107a974f https://blogs.oracle.com/mysql/optimize-oltp-and-olap-workloads-with-mysql-heatwave-autopilot As businesses grow, their databases must support both fast transactions and timely analytics. OLTP workloads handle everyday operations such as orders and payments, while OLAP workloads analyze large datasets for reporting and planning. Without appropriate optimization, growing data volumes and competing resource demands can slow transactions and delay insights. MySQL HeatWave supports online transaction processing […] Fri, 23 Oct 2026 18:27:17 +0000 Oracle MySQL Group MySQL MySQL Basics/How-To MySQL HeatWave OCI What’s New in MySQL NDB Cluster 9.x – Better Visibility and Control in MySQL NDB Cluster 5088bd809dc1d42c81aeb98bca818a45 https://blogs.oracle.com/mysql/whats-new-in-mysql-ndb-cluster-9-x-better-visibility-and-control-in-mysql-ndb-cluster MySQL NDB Cluster 9.x continues to focus on the qualities that matter most in production: clear operational insight, predictable behaviour under failure, efficient application access, and faster recovery. This release series adds practical observability across the cluster, strengthens several exceptional paths, and expands the control available to users. Better visibility into cluster operation Diagnosing a […] Thu, 22 Oct 2026 17:05:48 +0000 Oracle MySQL Group MySQL MySQL Community MySQL NDB News mysql MySQL 9.7 LTS mysqlcommunity Benchmarking vector indexes in MySQL with vector-dev-bench 6ad905f36a70ab0001bf83eb https://villagesql.com/blog/vector-dev-bench/ PostgreSQL has a rapidly evolving vector ecosystem. MySQL is earlier on that road, but the community is hard at work catching up. MariaDB ships its own HNSW index, Google engineers have shared a ScaNN-based vector index on MySQL's own vector index proposal, Percona is building an HNSW index inside InnoDB, and VillageSQL is building vector search as an extension with vsql-vector. We want vector search on MySQL to be something that teams can choose with confidence (and not have to switch to PostgreSQL to get). Getting there takes more than one good implementation. The teams building these indexes need to measure them the same way, check each other's numbers, and fix what the numbers show. Working with Percona, we published vector-dev-bench. It runs on a laptop, it drives VillageSQL, MariaDB, PostgreSQL with pgvector, and MySQL built from Google's ScaNN pull request, and it gives an answer in minutes. Doing things in the open A benchmark number is only as good as the method behind it. A public method lets anyone rerun numbers, read the exact SQL each engine received, and tell us where the method may be unfair. It also makes collaboration practical. When another team can run the same harness against their own build, we can compare results without a bunch of upfront coordination. With vector-dev-bench, an engine is one entry in a Python dictionary plus a start script, so adding a new one is a small change. We've seen this work already. Percona's vector-bench is a full comparison framework, and we added VillageSQL to our copy of it. Along the way, we noticed that pgvector loaded its rows with COPY while the MySQL engines used INSERT, so the build times were measuring two different kinds of statements. We switched pgvector's incremental build to batched INSERTs so that every engine loads rows the same way. vector-bench adopted the change, and its code credits our commit. That is the kind of exchange we want more of. Two tools for two jobs vector-bench and vector-dev-bench answer different questions, and we use both. vector-bench is built to produce a comparison that you can publish. It runs each engine in Docker, built from pinned sources where it compiles them, runs real datasets such as SIFT, GloVe, and GIST, and measures recall against throughput, build cost, concurrency from 1 to 32 clients, filtered search, and recall after deletes and reinserts. It runs every engine twice, once with identical resources and once with each vendor's recommended settings, and it records the hardware and versions with every result. Its main profile takes about two days, which is the right cost for a number that other people will cite. vector-dev-bench is built for the quick, iterative loop in which you change the index, rebuild it, and want to know whether recall moved. It runs against engines you've already built, does not use containers, and uses synthetic data by default. It builds a 20,000-vector index in about 7 seconds. Its README is plain about the trade-off: the numbers are for relative comparisons, and a published comparison belongs in a framework like vector-bench. vector-dev-bench Measurement vector-dev-bench is three command-line tools over one shared module: Tool What it measures recall_bench.py It measures recall@k and queries per second across a sweep of ef_search values. It exits with an error when recall falls below a threshold (0.95 by default), so a script can use it as a pass/fail check. build_bench.py It measures index build time. --build-threads N builds the index from N client connections at once. rw_bench.py It measures reads during writes. Reader threads run nearest-neighbor queries while writer threads insert into the same index, and the tool reports read throughput second by second along with p50, p95, and p99 latency. Recall is the measure that matters here, because an approximate nearest-neighbor index returns approximately the right answer by design. A test that compares exact output, the way MySQL's own test framework does, fails on a correct index. vector-dev-bench computes the true nearest neighbors itself with NumPy and reports what fraction of them each engine returned. You can also pass --dataset to use a standard ann-benchmarks dataset with its published ground truth. The engines are profiles: Profile Engine vsql_vector VillageSQL with vsql-vector (SVECTOR column, HNSW index) mariadb MariaDB 11.7 or later (MHNSW) pgvector PostgreSQL 15 or later with pgvector google MySQL 9.7.1 built from Google's vector search commit (ScaNN index), which its authors describe as an experimental proof of concept varchar_ctrl A plain VARCHAR column, as a control that separates a server problem from a harness problem VillageSQL, MariaDB, and the Google build all speak the MySQL protocol, and the harness drives all three through the same PyMySQL connection code. The client cost is the same for each engine, but it isn't zero. A gap between two of these engines holds for the machine that measured it, and it can change on different hardware, so rerun it on the hardware you care about before you trust it. pgvector takes a different path. The harness sends each batch of pgvector inserts or queries through the psql command-line client, so the timed window also includes starting psql and opening its connection. The parallel options (--readers and --build-threads) fall back to one serial client, and rw_bench.py does not run against pgvector. A VillageSQL-versus-pgvector number from vector-dev-bench measures the two clients as well as the two engines. Treat it as a hint about where to look, and don't quote it as a verdict on either engine. What a run looks like We ran a recall sweep against VillageSQL 0.0.7 with the vsql-vector-pk build of the extension (whose HNSW index stores each row's primary key), on 20,000 random 32-dimension vectors. The tool builds the index once and then queries it at four ef_search values. The README and RUNBOOK cover building each engine and running the tools. Here is the output from an Apple M5 laptop with 32 GB of memory: build_time_s=7.03 (index built once; sweeping ef_search) ef_search recall@10 qps 50 0.7870 8829.3 100 0.9035 5903.0 200 0.9645 3526.0 400 0.9875 2044.4 This is the curve every HNSW index trades along. A wider search finds more of the true neighbors and answers fewer queries per second. The shape is also a check on the run itself: recall that stays at exactly 1.0 when ef_search drops usually means the optimizer chose a full table scan and the index wasn't used. MariaDB and PostgreSQL run the same sweep through their own profiles, and run_three_way.sh runs all three engines one after another so that only one server is busy at a time. Reads while the index is changing Benchmarks such as ann-benchmarks build the index once and then query a corpus that never changes. Production tables change all the time. rw_bench.py covers that case: it runs readers alone, adds writers for a window in the middle, and then runs readers alone again. This run used 4 readers and 4 writers against the same 20,000-vector index: total reads=47700 writes=2000 rows write_rate=432 rows/s over 4.6s read-only mean=4564 qps read+write mean=2738 qps impact=40% drop read latency (us) p50 p95 p99 max n read-only 797 1399 1805 87018 32038 read+write 1440 1803 1908 8991 10988 The per-second table in the full output shows reads dropping when the writers start at second 4 and recovering once they stop at second 8. For an index that lives inside a transactional engine, this is one of the numbers we care about most, because it shows what a busy table costs the queries that read it. When to use it, and when not to Use vector-dev-bench when you're changing an index and want to know within minutes whether recall, build time, or read throughput moved. Use it as a recall check in a script, since recall_bench.py exits with an error below its threshold. Use it to try an idea against MariaDB or pgvector on the same machine (for pgvector, recall and build time only) before you invest in a full run. Use a full framework such as vector-bench when the number is going to leave the room. That includes anything published, anything that compares absolute latency, and anything that depends on filtered search or recall after deletes. Latency under concurrency counts as absolute latency. A comparison with pgvector needs the full framework most of all, because pgvector is the engine whose client differs. Every number from vector-dev-bench passes through a client, so absolute latency includes the client and the round trip. Synthetic vectors give a trustworthy relative signal on the machine that ran them, but they don't predict recall on real embeddings. Add your engine The repository is Apache-2.0 licensed, and an AGENTS.md file walks a coding agent through running it. If you're building a vector index for MySQL, or for anything that speaks the MySQL or PostgreSQL protocol, add a profile and a start script and send a pull request. If you think our method is unfair to your engine, open an issue and tell us why. Getting vector search on MySQL to the point where teams can pick it with confidence is work for everyone in the MySQL community, and it starts with numbers we can all reproduce. You can get VillageSQL from the website, read how custom indexes work in the custom indexes documentation, and see our Percona Live demo in Vector search in MySQL: an early look at HNSW and custom indexes in VillageSQL.

PostgreSQL has a rapidly evolving vector ecosystem. MySQL is earlier on that road, but the community is hard at work catching up. MariaDB ships its own HNSW index, Google engineers have shared a ScaNN-based vector index on MySQL's own vector index proposal, Percona is building an HNSW index inside InnoDB, and VillageSQL is building vector search as an extension with vsql-vector.

We want vector search on MySQL to be something that teams can choose with confidence (and not have to switch to PostgreSQL to get). Getting there takes more than one good implementation. The teams building these indexes need to measure them the same way, check each other's numbers, and fix what the numbers show.

Working with Percona, we published vector-dev-bench. It runs on a laptop, it drives VillageSQL, MariaDB, PostgreSQL with pgvector, and MySQL built from Google's ScaNN pull request, and it gives an answer in minutes.

Doing things in the open

A benchmark number is only as good as the method behind it. A public method lets anyone rerun numbers, read the exact SQL each engine received, and tell us where the method may be unfair.

It also makes collaboration practical. When another team can run the same harness against their own build, we can compare results without a bunch of upfront coordination. With vector-dev-bench, an engine is one entry in a Python dictionary plus a start script, so adding a new one is a small change.

We've seen this work already. Percona's vector-bench is a full comparison framework, and we added VillageSQL to our copy of it. Along the way, we noticed that pgvector loaded its rows with COPY while the MySQL engines used INSERT, so the build times were measuring two different kinds of statements. We switched pgvector's incremental build to batched INSERTs so that every engine loads rows the same way. vector-bench adopted the change, and its code credits our commit. That is the kind of exchange we want more of.

Two tools for two jobs

vector-bench and vector-dev-bench answer different questions, and we use both.

vector-bench is built to produce a comparison that you can publish. It runs each engine in Docker, built from pinned sources where it compiles them, runs real datasets such as SIFT, GloVe, and GIST, and measures recall against throughput, build cost, concurrency from 1 to 32 clients, filtered search, and recall after deletes and reinserts. It runs every engine twice, once with identical resources and once with each vendor's recommended settings, and it records the hardware and versions with every result. Its main profile takes about two days, which is the right cost for a number that other people will cite.

vector-dev-bench is built for the quick, iterative loop in which you change the index, rebuild it, and want to know whether recall moved. It runs against engines you've already built, does not use containers, and uses synthetic data by default. It builds a 20,000-vector index in about 7 seconds. Its README is plain about the trade-off: the numbers are for relative comparisons, and a published comparison belongs in a framework like vector-bench.

vector-dev-bench Measurement

vector-dev-bench is three command-line tools over one shared module:

Tool What it measures
recall_bench.py It measures recall@k and queries per second across a sweep of ef_search values. It exits with an error when recall falls below a threshold (0.95 by default), so a script can use it as a pass/fail check.
build_bench.py It measures index build time. --build-threads N builds the index from N client connections at once.
rw_bench.py It measures reads during writes. Reader threads run nearest-neighbor queries while writer threads insert into the same index, and the tool reports read throughput second by second along with p50, p95, and p99 latency.

Recall is the measure that matters here, because an approximate nearest-neighbor index returns approximately the right answer by design. A test that compares exact output, the way MySQL's own test framework does, fails on a correct index. vector-dev-bench computes the true nearest neighbors itself with NumPy and reports what fraction of them each engine returned. You can also pass --dataset to use a standard ann-benchmarks dataset with its published ground truth.

The engines are profiles:

Profile Engine
vsql_vector VillageSQL with vsql-vector (SVECTOR column, HNSW index)
mariadb MariaDB 11.7 or later (MHNSW)
pgvector PostgreSQL 15 or later with pgvector
google MySQL 9.7.1 built from Google's vector search commit (ScaNN index), which its authors describe as an experimental proof of concept
varchar_ctrl A plain VARCHAR column, as a control that separates a server problem from a harness problem

VillageSQL, MariaDB, and the Google build all speak the MySQL protocol, and the harness drives all three through the same PyMySQL connection code. The client cost is the same for each engine, but it isn't zero. A gap between two of these engines holds for the machine that measured it, and it can change on different hardware, so rerun it on the hardware you care about before you trust it.

pgvector takes a different path. The harness sends each batch of pgvector inserts or queries through the psql command-line client, so the timed window also includes starting psql and opening its connection. The parallel options (--readers and --build-threads) fall back to one serial client, and rw_bench.py does not run against pgvector. A VillageSQL-versus-pgvector number from vector-dev-bench measures the two clients as well as the two engines. Treat it as a hint about where to look, and don't quote it as a verdict on either engine.

What a run looks like

We ran a recall sweep against VillageSQL 0.0.7 with the vsql-vector-pk build of the extension (whose HNSW index stores each row's primary key), on 20,000 random 32-dimension vectors. The tool builds the index once and then queries it at four ef_search values. The README and RUNBOOK cover building each engine and running the tools.

Here is the output from an Apple M5 laptop with 32 GB of memory:

build_time_s=7.03  (index built once; sweeping ef_search)
ef_search  recall@10    qps
50         0.7870       8829.3
100        0.9035       5903.0
200        0.9645       3526.0
400        0.9875       2044.4

This is the curve every HNSW index trades along. A wider search finds more of the true neighbors and answers fewer queries per second. The shape is also a check on the run itself: recall that stays at exactly 1.0 when ef_search drops usually means the optimizer chose a full table scan and the index wasn't used.

MariaDB and PostgreSQL run the same sweep through their own profiles, and run_three_way.sh runs all three engines one after another so that only one server is busy at a time.

Reads while the index is changing

Benchmarks such as ann-benchmarks build the index once and then query a corpus that never changes. Production tables change all the time. rw_bench.py covers that case: it runs readers alone, adds writers for a window in the middle, and then runs readers alone again. This run used 4 readers and 4 writers against the same 20,000-vector index:

total reads=47700  writes=2000 rows  write_rate=432 rows/s over 4.6s
read-only mean=4564 qps  read+write mean=2738 qps  impact=40% drop
read latency (us)   p50      p95      p99      max      n
  read-only             797     1399     1805    87018    32038
  read+write           1440     1803     1908     8991    10988

The per-second table in the full output shows reads dropping when the writers start at second 4 and recovering once they stop at second 8. For an index that lives inside a transactional engine, this is one of the numbers we care about most, because it shows what a busy table costs the queries that read it.

When to use it, and when not to

Use vector-dev-bench when you're changing an index and want to know within minutes whether recall, build time, or read throughput moved. Use it as a recall check in a script, since recall_bench.py exits with an error below its threshold. Use it to try an idea against MariaDB or pgvector on the same machine (for pgvector, recall and build time only) before you invest in a full run.

Use a full framework such as vector-bench when the number is going to leave the room. That includes anything published, anything that compares absolute latency, and anything that depends on filtered search or recall after deletes. Latency under concurrency counts as absolute latency. A comparison with pgvector needs the full framework most of all, because pgvector is the engine whose client differs. Every number from vector-dev-bench passes through a client, so absolute latency includes the client and the round trip. Synthetic vectors give a trustworthy relative signal on the machine that ran them, but they don't predict recall on real embeddings.

Add your engine

The repository is Apache-2.0 licensed, and an AGENTS.md file walks a coding agent through running it. If you're building a vector index for MySQL, or for anything that speaks the MySQL or PostgreSQL protocol, add a profile and a start script and send a pull request. If you think our method is unfair to your engine, open an issue and tell us why. Getting vector search on MySQL to the point where teams can pick it with confidence is work for everyone in the MySQL community, and it starts with numbers we can all reproduce.

You can get VillageSQL from the website, read how custom indexes work in the custom indexes documentation, and see our Percona Live demo in Vector search in MySQL: an early look at HNSW and custom indexes in VillageSQL.

]]> Thu, 22 Oct 2026 14:00:00 +0000 VillageSQL AI MySQL Extension Open Source VillageSQL Village News: MySQL News + Events (21 October 2026) 6ad915d56a70ab0001bf842a https://villagesql.com/blog/village-news-mysql-news-events-21-october-2026/ Welcome back to Village News, our curated roundup of MySQL and database news. This issue covers two weeks, with MySQL 26.10 landing at the end of them. If you want to get these updates, just subscribe to the blog. Enjoy! MySQL News Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated) MySQL 26.10 Is Here: New Capabilities for Developers and DBAs Oracle MySQL Blog TL;DR - MySQL 26.10 brings MySQL REST Service into the server as a component, so you no longer need MySQL Router to serve its HTTPS endpoints. Four Attempts and One RESET MASTER — Migrating a Terabyte of Production MySQL to Kubernetes (MOCO) Eugene Dimov, OurSQL Foundation Blog TL;DR - Ivinco rolled back three cutovers to MOCO, caused by a hardcoded VIP and Ceph write latency. After the move, a passwordless root account from the donor made the operator run RESET MASTER and wipe a replica's GTID history. MySQL vs PostgreSQL Internals (Part 3) – Row Locking Zhao Song TL;DR - Traces one SELECT … ORDER BY … FOR UPDATE through both engines. InnoDB locks rows as it scans and keeps the locks in memory. Postgres sorts a snapshot first, then locks and rechecks each tuple, recording the lock in the tuple header. Taming MySQL 8.4 Background Histogram Lock Wait Timeouts on Amazon RDS Umesh Shastry, MySQL :: under_the_hood TL;DR - RDS's monitoring agent holds a near-constant write lock on mysql.rds_heartbeat2, so 8.4's background histogram collection keeps timing out on it. The fix is to drop the histograms on that one table, not to change global settings. How fast is vector search inside MySQL? Alkin Tezuysal, AskDBA TL;DR - Benchmarks the MyVector HNSW plugin on three MySQL versions against MariaDB's native vector type. lefred's reply argues that MariaDB comes out ahead once you compare at matching recall. Announcing VillageSQL Server 0.0.7 VillageSQL Blog TL;DR - Builds now track MySQL 8.4.11, 9.7.2 and Percona Server 8.4.11-11. Extensions can now define custom indexes, including nearest-neighbor search, which lays the groundwork for vector search. “MySQL Is Still the Backbone”: Zongzhi Chen on Hot Rows, PolarDB, and the Agent Era OurSQL Foundation Blog TL;DR - An interview with a longtime MySQL contributor and early PolarDB engineer about why China runs on MySQL, what Singles Day does to hot rows, and where MySQL goes next. MySQL Online DDL "Duplicate Key" Optimization Libing Song TL;DR - Since 5.6, an online DDL can fail with a false Duplicate Entry error when concurrent DML fails on a unique key and leaves entries in the row log. This explains the cause and the fix shipped in AliSQL. InnoDB for the fast. DuckDB for the heavy. Same connection for both. Daniel Guzmán Burgos, dbtrail Blog TL;DR - dbtrail (Apache 2.0) reads the binlog into Parquet copies and routes each SELECT to InnoDB or DuckDB. 13 of 14 reports ran about 10x faster, while point lookups got slower. A companion post uses it for row-level recovery. How a Routine Linux fstrim Job Took Down MySQL: An InnoDB Crash Analysis Vinoth Kanna, Mydbops Blog TL;DR - A weekly fstrim sent about 5.2 TiB of discards to thick-provisioned disks and stalled I/O until InnoDB's 600-second semaphore watchdog killed mysqld. The fix is to disable fstrim.timer on such storage. MySQL Workbench 26.7 Dave Stokes, Dave's MySQL Stuff TL;DR - The new Workbench is a rewrite, not an update of the version retired in January. It lacks relationship maps and reverse engineering, and servers 8.0 and older are no longer supported. PGO or not PGO this is the dilemma. Step 1 Marco Tusa, Percona Blog TL;DR - A three-part look at profile-guided builds of MySQL 9.7.2. Steps 2 and 3 show why a build trained only on sysbench loses at high concurrency and recommend mixing MTR runs with moderate-concurrency training. Database News Neki update: $50 Metal, read-only replicas, Azure, and more Nick Van Wiggeren, PlanetScale Blog TL;DR - A month into preview, PlanetScale's sharded Postgres has added read-only replicas, Metal instances from $50/month, Azure regions, and Terraform and CLI controls. Make pg_stat_monitor stable again! Percona Community Blog TL;DR - pg_stat_monitor 2.4.0 fixes memory leaks that could end in an OOM kill, along with buffer overflows, a use-after-free, a lock-ordering bug and incorrect statistics. CedarDB Cloud Is Now Generally Available CedarDB Blog TL;DR - CedarDB, a Postgres-compatible database for both transactional and analytical workloads, is now available as a fully managed service. What gets TOASTed in Postgres Ahmed Darwich, PlanetScale Blog TL;DR - Postgres 19 makes LZ4 the default TOAST compression. LZ4 is usually the right choice, but pglz still compresses repetitive JSON smaller and reads large-value previews faster. In MariaDB 13.0, new-style optimizer hints go beyond MySQL’s Sergei Petrunia TL;DR - MariaDB 13.0 is stable, and its optimizer hints now go beyond MySQL's syntax, starting with path-based query-block addressing in QB_NAME. Load terabytes of data in minutes into Lakebase Postgres Databricks Blog TL;DR - Spark builds valid Postgres heap and index pages in parallel and writes them to storage, then the primary publishes them with one compact WAL record. Databricks reports loading 1 TB in under 5 minutes. Upcoming Database Events PGConf.EU October 20–23, 2026 (PostgreSQL Europe) Valencia, Spain MySQL Public Discussion #6 October 22, 2026, 17:00–18:00 UTC Online Oracle AI World October 25–28, 2026 Las Vegas, NV PG Down Under 2026 October 30, 2026 Surry Hills Sydney, Australia MySQL Contributor Summit November 4-5, 2026 Virtual KubeCon + CloudNativeCon North America (VillageSQL is a sponsor) November 9-12, 2026 Salt Lake City, Utah PASS Data Community Summit November 9-11, 2026 Hyatt Regency Seattle Seattle, WA DuckDB × MySQL Day November 17, 2026 Online PGConf.Asia 2026 November 17-18, 2026 Hong Kong PGConf.PL 2026 November 24, 2026 ARCHE Dwór Uphagena Gdańsk, Poland AWS re:Invent 2026 November 30 – December 4, 2026 Las Vegas, NV Open Source Summit Japan December 7-9, 2026 Tokyo, Japan CIDR 2027 January 24-27, 2027 Mövenpick Hotel Amsterdam City Centre Amsterdam, The Netherlands MySQL Belgian Days 2027 January 28-29, 2027 ICAB Incubator Brussels, Belgium FOSDEM PGDay 2027 January 28-29, 2027 Brussels Marriott Hotel Grand Place Brussels, Belgium FOSDEM 2027 January 30-31, 2027 ULB Solbosch Campus Brussels, Belgium CERN PGDay 2027 February 12, 2027 CERN Council Chamber Geneva, Switzerland PGConf India 2027 March 2-5, 2027 Sheraton Grand Hotel at Brigade Gateway Bengaluru, India FOSSASIA PGDay 2027 March 9, 2027 Bangkok, Thailand KubeCon + CloudNativeCon Europe 2027 March 15-18, 2027 Barcelona, Spain Nordic PGDay 2027 March 16, 2027 Courtyard Kungsholmen Stockholm, Sweden SCaLE 24x April 1-4, 2027 Pasadena Convention Center Pasadena, CA PgDay Boston 2027 April 10, 2027 Tufts University Joyce Cummings Center Boston, MA PGConf.dev 2027 May 11-14, 2027 Plaza Centre-Ville Montréal, QC, Canada

Welcome back to Village News, our curated roundup of MySQL and database news. This issue covers two weeks, with MySQL 26.10 landing at the end of them.

If you want to get these updates, just subscribe to the blog.

Enjoy!

MySQL News

Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated)

MySQL 26.10 Is Here: New Capabilities for Developers and DBAs
Oracle MySQL Blog

TL;DR - MySQL 26.10 brings MySQL REST Service into the server as a component, so you no longer need MySQL Router to serve its HTTPS endpoints.

Four Attempts and One RESET MASTER — Migrating a Terabyte of Production MySQL to Kubernetes (MOCO)
Eugene Dimov, OurSQL Foundation Blog

TL;DR - Ivinco rolled back three cutovers to MOCO, caused by a hardcoded VIP and Ceph write latency. After the move, a passwordless root account from the donor made the operator run RESET MASTER and wipe a replica's GTID history.

MySQL vs PostgreSQL Internals (Part 3) – Row Locking
Zhao Song

TL;DR - Traces one SELECT … ORDER BY … FOR UPDATE through both engines. InnoDB locks rows as it scans and keeps the locks in memory. Postgres sorts a snapshot first, then locks and rechecks each tuple, recording the lock in the tuple header.

Taming MySQL 8.4 Background Histogram Lock Wait Timeouts on Amazon RDS
Umesh Shastry, MySQL :: under_the_hood

TL;DR - RDS's monitoring agent holds a near-constant write lock on mysql.rds_heartbeat2, so 8.4's background histogram collection keeps timing out on it. The fix is to drop the histograms on that one table, not to change global settings.

How fast is vector search inside MySQL?
Alkin Tezuysal, AskDBA

TL;DR - Benchmarks the MyVector HNSW plugin on three MySQL versions against MariaDB's native vector type. lefred's reply argues that MariaDB comes out ahead once you compare at matching recall.

Announcing VillageSQL Server 0.0.7
VillageSQL Blog

TL;DR - Builds now track MySQL 8.4.11, 9.7.2 and Percona Server 8.4.11-11. Extensions can now define custom indexes, including nearest-neighbor search, which lays the groundwork for vector search.

“MySQL Is Still the Backbone”: Zongzhi Chen on Hot Rows, PolarDB, and the Agent Era
OurSQL Foundation Blog

TL;DR - An interview with a longtime MySQL contributor and early PolarDB engineer about why China runs on MySQL, what Singles Day does to hot rows, and where MySQL goes next.

MySQL Online DDL "Duplicate Key" Optimization
Libing Song

TL;DR - Since 5.6, an online DDL can fail with a false Duplicate Entry error when concurrent DML fails on a unique key and leaves entries in the row log. This explains the cause and the fix shipped in AliSQL.

InnoDB for the fast. DuckDB for the heavy. Same connection for both.
Daniel Guzmán Burgos, dbtrail Blog

TL;DR - dbtrail (Apache 2.0) reads the binlog into Parquet copies and routes each SELECT to InnoDB or DuckDB. 13 of 14 reports ran about 10x faster, while point lookups got slower. A companion post uses it for row-level recovery.

How a Routine Linux fstrim Job Took Down MySQL: An InnoDB Crash Analysis
Vinoth Kanna, Mydbops Blog

TL;DR - A weekly fstrim sent about 5.2 TiB of discards to thick-provisioned disks and stalled I/O until InnoDB's 600-second semaphore watchdog killed mysqld. The fix is to disable fstrim.timer on such storage.

MySQL Workbench 26.7
Dave Stokes, Dave's MySQL Stuff

TL;DR - The new Workbench is a rewrite, not an update of the version retired in January. It lacks relationship maps and reverse engineering, and servers 8.0 and older are no longer supported.

PGO or not PGO this is the dilemma. Step 1
Marco Tusa, Percona Blog

TL;DR - A three-part look at profile-guided builds of MySQL 9.7.2. Steps 2 and 3 show why a build trained only on sysbench loses at high concurrency and recommend mixing MTR runs with moderate-concurrency training.

Database News

Neki update: $50 Metal, read-only replicas, Azure, and more
Nick Van Wiggeren, PlanetScale Blog

TL;DR - A month into preview, PlanetScale's sharded Postgres has added read-only replicas, Metal instances from $50/month, Azure regions, and Terraform and CLI controls.

Make pg_stat_monitor stable again!
Percona Community Blog

TL;DR - pg_stat_monitor 2.4.0 fixes memory leaks that could end in an OOM kill, along with buffer overflows, a use-after-free, a lock-ordering bug and incorrect statistics.

CedarDB Cloud Is Now Generally Available
CedarDB Blog

TL;DR - CedarDB, a Postgres-compatible database for both transactional and analytical workloads, is now available as a fully managed service.

What gets TOASTed in Postgres
Ahmed Darwich, PlanetScale Blog

TL;DR - Postgres 19 makes LZ4 the default TOAST compression. LZ4 is usually the right choice, but pglz still compresses repetitive JSON smaller and reads large-value previews faster.

In MariaDB 13.0, new-style optimizer hints go beyond MySQL’s
Sergei Petrunia

TL;DR - MariaDB 13.0 is stable, and its optimizer hints now go beyond MySQL's syntax, starting with path-based query-block addressing in QB_NAME.

Load terabytes of data in minutes into Lakebase Postgres
Databricks Blog

TL;DR - Spark builds valid Postgres heap and index pages in parallel and writes them to storage, then the primary publishes them with one compact WAL record. Databricks reports loading 1 TB in under 5 minutes.

Upcoming Database Events

PGConf.EU
October 20–23, 2026 (PostgreSQL Europe)
Valencia, Spain

MySQL Public Discussion #6
October 22, 2026, 17:00–18:00 UTC
Online

Oracle AI World
October 25–28, 2026
Las Vegas, NV

PG Down Under 2026
October 30, 2026
Surry Hills
Sydney, Australia

MySQL Contributor Summit
November 4-5, 2026
Virtual

KubeCon + CloudNativeCon North America (VillageSQL is a sponsor)
November 9-12, 2026
Salt Lake City, Utah

PASS Data Community Summit
November 9-11, 2026
Hyatt Regency Seattle
Seattle, WA

DuckDB × MySQL Day
November 17, 2026
Online

PGConf.Asia 2026
November 17-18, 2026
Hong Kong

PGConf.PL 2026
November 24, 2026
ARCHE Dwór Uphagena
Gdańsk, Poland

AWS re:Invent 2026
November 30 – December 4, 2026
Las Vegas, NV

Open Source Summit Japan
December 7-9, 2026
Tokyo, Japan

CIDR 2027
January 24-27, 2027
Mövenpick Hotel Amsterdam City Centre
Amsterdam, The Netherlands

MySQL Belgian Days 2027
January 28-29, 2027
ICAB Incubator
Brussels, Belgium

FOSDEM PGDay 2027
January 28-29, 2027
Brussels Marriott Hotel Grand Place
Brussels, Belgium

FOSDEM 2027
January 30-31, 2027
ULB Solbosch Campus
Brussels, Belgium

CERN PGDay 2027
February 12, 2027
CERN Council Chamber
Geneva, Switzerland

PGConf India 2027
March 2-5, 2027
Sheraton Grand Hotel at Brigade Gateway
Bengaluru, India

FOSSASIA PGDay 2027
March 9, 2027
Bangkok, Thailand

KubeCon + CloudNativeCon Europe 2027
March 15-18, 2027
Barcelona, Spain

Nordic PGDay 2027
March 16, 2027
Courtyard Kungsholmen
Stockholm, Sweden

SCaLE 24x
April 1-4, 2027
Pasadena Convention Center
Pasadena, CA

PgDay Boston 2027
April 10, 2027
Tufts University Joyce Cummings Center
Boston, MA

PGConf.dev 2027
May 11-14, 2027
Plaza Centre-Ville
Montréal, QC, Canada

]]> Wed, 21 Oct 2026 19:52:23 +0000 VillageSQL MySQL October 2026 GA Releases Now Available 4d572f7104ea86e92da7ac3cfbf6b13a https://blogs.oracle.com/mysql/mysql-october-2026-ga-releases-now-available The October 2026 MySQL releases are now available, including: MySQL 26.10 brings additional capabilities to Community Edition, new options for building REST applications, and improvements to database operations and developer tools. MySQL 9.7.4 and MySQL 8.4.13 continue maintenance updates for the supported LTS release lines. October MySQL Server releases MySQL 26.10 Innovation delivers new functionality […] Wed, 21 Oct 2026 19:10:48 +0000 Oracle MySQL Group MySQL MySQL Community MySQL Enterprise News mysql mysqlcommunity Introducing the MySQL REST Service Component: Now Generally Available 7cd64365aa2c9e72a381c1cead93efac https://blogs.oracle.com/mysql/introducing-the-mysql-rest-service-component-now-generally-available With MySQL 26.10, we’re announcing the general availability of the MySQL REST Service (MRS) Component, an installable component that runs MRS directly inside MySQL Server. MRS: the MySQL REST Service Many modern applications, well, not just applications, but also tools, CLIs, CI pipelines, and automation, use HTTP and JSON to communicate with other systems. For […] Wed, 21 Oct 2026 10:02:39 +0000 Oracle MySQL Group MySQL MySQL Community News components JSON MySQL 26.10 MySQL REST Service MySQL Server REST API MySQL 26.10 Is Here: New Capabilities for Developers and DBAs a415ef7d26b753ed46b5277a5a69ad3b https://blogs.oracle.com/mysql/mysql-26-10-is-here MySQL 26.10 is here! This release brings MySQL REST Service into the server, expands Community Edition capabilities, and adds more control for applications using JSON Relational Duality Views. Here’s a look at the highlights. MySQL REST Service joins the server MySQL REST Service (MRS) is now available as a MySQL Server component, enabling HTTPS endpoints […] Tue, 20 Oct 2026 22:31:29 +0000 Oracle MySQL Group MySQL MySQL Community mysql MySQL 26.10 mysqlcommunity MySQL JSON Duality Views: Now with granular write and ETAG controls 1ea30820382a65862e65841b77121977 https://blogs.oracle.com/mysql/mysql-json-duality-views-now-with-granular-write-and-etag-controls JSON Duality Views (JDVs) let applications read and write JSON documents backed by relational tables (see our introductory JDV blog). MySQL handles the document-to-table mapping, reducing application code while preserving relational integrity. We still need to decide which parts of a document applications can change. Earlier JDVs offered limited choices: table writes were generally all-or-nothing, and […] Tue, 20 Oct 2026 12:59:00 +0000 Oracle MySQL Group MySQL MySQL Basics/How-To MySQL Community JSON Duality mysql mysqlcommunity OCI Troubleshooting MySQL with OpenTelemetry Metrics ed5189806daa7bd6f1b67b0863c877a2 https://blogs.oracle.com/mysql/troubleshooting-mysql-with-opentelemetry-metrics In the first post, we exported native MySQL telemetry to Prometheus. In the second, we organized those metrics around DBA questions. In the third, we built a Grafana dashboard that makes those questions visible. This final post shows how to use that dashboard when an application slows down or a database signal changes. The goal […] Mon, 19 Oct 2026 23:48:23 +0000 Oracle MySQL Group Monitoring MySQL MySQL Enterprise Upgrading MySQL HeatWave High Availability from MySQL 8.0.46 to MySQL 8.4 LTS 277c5ccffa8a3fb074d561b46825ee5c https://blogs.oracle.com/mysql/upgrading-mysql-heatwave-high-availability-from-mysql-8-0-46-to-mysql-8-4-lts Planning, compatibility testing, performance validation, application readiness, and rollback strategies for a large production deployment MySQL 8.0 reached its official End of Life (EOL) in April 2026 alongside the final release of version 8.0.46, but specific support extensions are available depending on your hosting environment.  Oracle extended support for MySQL 8.0 instances in MySQL HeatWave […] Mon, 19 Oct 2026 19:50:31 +0000 Oracle MySQL Group MySQL MySQL Basics/How-To MySQL HeatWave mysql OCI Devart ODBC Drivers update brings SQL Server 2025, MySQL 9.7, and new Snowflake authentication methods https://www.devart.com/blog/?p=184660 https://www.devart.com/blog/devart-odbc-drivers-update-brings-sql-server-2025-mysql-9-7-and-new-snowflake-authentication-methods.html The latest Devart ODBC Drivers release is here, adding support for SQL Server 2025, MySQL 9.7.x, and NexusDB 4.75.20. The update also introduces new authentication options, improved data protection, and broader compatibility with development tools, operating systems, and other data sources. The post Devart ODBC Drivers update brings SQL Server 2025, MySQL 9.7, and new Snowflake authentication methods appeared first on Devart Blog. The latest Devart ODBC Drivers release is here, adding support for SQL Server 2025, MySQL 9.7.x, and NexusDB 4.75.20. The update also introduces new authentication options, improved data protection, and broader compatibility with development tools, operating systems, and other data sources.

The post Devart ODBC Drivers update brings SQL Server 2025, MySQL 9.7, and new Snowflake authentication methods appeared first on Devart Blog.

]]> Fri, 16 Oct 2026 15:28:13 +0000 Alena Subotina ODBC Drivers What’s New Devart ODBC Driver for SQL Server odbc odbc driver odbc drivers When a Small Change Writes Too Much – How A Linux Kernel Improvement Can Strike You in PXC/Galera Workloads https://www.percona.com/?p=54716 https://www.percona.com/blog/when-a-small-change-writes-too-much-how-a-linux-kernel-improvement-can-strike-you-in-pxc-galera-workloads/ We’ve recently been dealing with an interesting customer case where a PXC version upgrade from 5.7 to 8.0 seemed to trigger a completely unexpected problem. Even though the workload and MySQL settings did not change, all PXC members began exhibiting sparse but regular bursts of massive disk writes. Yet the MySQL side workload statistics that we gathered during those bursts did not explain the volume of them at all! Analyzing these bursts further with pidstat and iostat confirmed that the mysqld process was indeed the sole source of excessive disk writes, and digging further showed the responsible threads to be the Galera receiver thread responsible for subscribing to the replication stream, as well as the main connection threads on the writer node, or Galera applier threads on the secondary members. And what’s most important is that there was a huge disparity between InnoDB engine threads writes (redo, undo, tablespace, etc) and binary log writes versus what was accounted to the Galera threads. At first, it looked like during the burst, several GB of write-sets were received and applied, yet InnoDB barely changed any pages. All InnoDB-related write activity was completely flat, so they did not participate in these high IO load spikes at all! An example mysqld per-thread breakdown for a few-second burst looked like this: Thread group Accounted Writes Share applier (Galera write-set appliers, many threads) 13.0 GB 61% galera_recv-0 (Galera receiver thread) 5.2 GB 24% ib_pg_flush* (InnoDB page cleaners) 2.0 GB 9% ib_log_writer (redo log) 0.76 GB 4% Galera_srvc (Galera service thread) 0.29 GB 1% The quick thought on the small redo log vs potentially huge writeset volume was that maybe transactions changing a single column would bloat the binlog events with other large blob/text/json columns, due to binlog_row_image=full, as the ROW image format applies to both Galera writesets and binary logs. But, analyzing the binary logs (with binlogsum) showed moderate and completely flat volume, nothing related to those bursts. Looking for clues, I asked myself, so what are those Galera threads exactly writing? The receiver thread copies all replicated writesets into the Galera cache file(s), but also reclaims the space when the ring wraps (the ring-buffer allocator walks forward from the oldest buffers and discards released ones). Then, the applier or connection threads also update the cache according to the transaction’s state. Still, when there are no spikes with big writesets (MySQL write load stays flat), why would we see such massive IO write bursts only accounted to the Galera threads? And why would that weird behavior start to happen after the PXC upgrade? We could not find any clues in the source code; there were no changes to the Galera plugin or wsrep extensions that would explain these bursts. Later, we found out additional information, which turned out to be crucial – not only the PXC version, but also the OS was upgraded! Now, the previous one: Debian 11, uses Linux 5.10 LTS kernel series, while the upgraded version 12 uses the Linux 6.1 LTS kernel series. And as the main suspect at this point was the Gcache store, how would the new kernel change the way this file is handled, though? It turns out that it does indeed, and it is related to the fact that the Galera plugin uses the mmap() path to handle the cache! Page Folios in Modern Linux Kernels In pursuit of handling large data on machines with big RAM and disk space more efficiently in Linux, in addition to huge pages, something called page folios was introduced in 5.16. What is relevant to this case is that large folios were implemented to also handle the page cache, initially only for the XFS filesystem in kernel 5.18, and yes, the affected cluster kept Galera cache on the XFS partition. Then, large folios support is what makes the kernel 6.1 different from the previous 5.10. OK, but why would “folios” cause bursts, and why do they not affect InnoDB writes, but only the Galera threads? Now, unlike any other data or log files in MySQL (except TempTable storage) or MariaDB, Galera cache files do not use the write() or pwrite() path, but mmap rather. This difference pointed me to some reports complaining about the systemd journal’s excessive IO problems, which turns out to be using mmap() path as well! Funny enough, the problem is not very well known, and the root cause is still being confused to this day, even in the most affected systemd community. Some reports describe the journal IO pressure being higher than expected, but never define the real culprit. The systemd issue that currently covers this problem best is: https://github.com/systemd/systemd/issues/40262 Related kernel thread discussions: https://lwn.net/Articles/1092530/ https://lists.openwall.net/linux-kernel/2026/08/20/1169 And despite many good blog posts about optimizing journal IO, I found only this one that points to the folio impact: https://www.cnblogs.com/32bin/p/22929557 The problem even reached the general tech press, as still unsolved in 2026: https://northeasttimes.com/2026/08/14/a-single-log-line-eats-49kb-of-disk-and-linux-admins-are-furious/ In short: on a large folios-enabled kernel and filesystem, page cache entries are grouped in folios. In early kernel implementations (5.18–6.5), write()/pwrite() paths were also affected, but InnoDB typically uses O_DIRECT for table spaces (so not using page cache at all), and redo and binlog use small appends, which create order-0 (4 kB) small folios. Then in Linux 6.6, a patch was implemented that added per-block dirty tracking, as now iomap records the overlapping filesystem blocks (filesystem-block granularity): see RFCv3 for details. With mmap writes, any update only tracks a whole folio. Thus, writeback page cache iteration, in the mmap case, will sync the whole folio (up to 2MB) instead of the exact fs block (4kB). In other words, the mmap() path cannot benefit from the improved I/O tracking loops in 6.6+. So, yes, the difference may be potentially huge, and small random writes to a mmapped file can induce huge IO bursts when the related page cache gets flushed to disk. There are important variables here: Readahead size via optimal_io_size → read_ahead_kb, per block device, influences the sizes of large folios allocated through it (from 128kB up to even 2MB) Bursts are specifically periodic due to how writeback of dirty data gets invoked – every dirty_writeback_centisecs (default 5 seconds) the kernel checks if the dirty data (tracked per inode) age is old enough via dirty_expire_centisecs (default 30 seconds) – the observed roughly 33-second bursts on the affected PXC nodes, are consistent with this mechanism (although these settings do not guarantee a fixed flush period) when the object, here the Gcache inode, gets flushed when receiving regular writes. Workload characteristics matter a lot – writes accumulated within a single folio will result in way smaller write amplifications as compared to a file being updated in many random places, hence touching many different large folios. The whole concept is damn complex, so let me share a more technical deep dive schematic in a separate document. Bear in mind, I am not a Linux kernel expert, so I hope there are not too many errors. The document is available here: https://przemekmalkowski.github.io/folios-mmap/ Galera Cache Impact Galera’s writeset cache has been historically a mmapped file from the very beginning. And there are a couple of factors that make the large folio-induced write amplifications particularly severe here: The cache file is by default 128MB, but very often a much bigger size is used to provide long enough IST coverage. In addition to sequential appends of new writesets, there are many small and random updates that change their status to RELEASED when the transaction is already applied and ready to be overwritten. Quite often, MySQL data, together with the galera.cache file, is placed on a large storage, like RAID arrays. Specifically for software RAID, the Linux kernel automatically increases the readahead size proportionally to the array geometry and the advertised optimal I/O size. For hardware RAID and dm/LVM stacks, it may be the case too. This leads to even larger folios, and that’s exactly what happened for the said customer case. During the PXC member startup, the whole gcache file is scanned sequentially during the recovery process (enabled by default), which can immediately populate the large page cache areas. Having said that, every PXC/Galera cluster may be affected by extra write amplifications for the galera.cache (and gcache.page.X), if kernel and filesystem versions support large folios. But the impact may be small enough to slip through unnoticed when the default readahead of 128kB is set for the disk block device. The extra IO write pressure may be wrongly assumed to be related to regular (and complex enough) database writes. But when the storage uses bigger readahead, and large folios get to even 2MB in size, the amplification is huge and really difficult to miss. Possible workarounds As currently there is no Linux kernel side solution implemented for the mmap() path, there are a couple of workarounds possible, and they apply to any applications using mmapped files, not only PXC/Galera: Depending on the kernel version, using the EXT4 filesystem may be an option, as folio support was introduced there much later (in 6.16) than in XFS (5.18) – if applicable, this allows us to avoid the said amplifications altogether. It helped in this customer’s case! Lower the readahead size for the disk device – for the price of potentially limited performance, the maximum size of large folios will be impacted by it, making the amplifications smaller. Modify the application mmap calls to use MADV_RANDOM via madvise() hints. It will basically disable readahead via enforced random access. Stop using mmap() path in favor of pwrite() on Linux 6.6+. Finally, use a customized modern kernel, compiled without THP support via the disabled CONFIG_TRANSPARENT_HUGEPAGE option – large folios are not created on recent kernels when THP is not available. (Note that disabling THP in GRUB is not enough to disable large folios!) For PXC, I reported the problem here: https://perconadev.atlassian.net/browse/PXC-5340 How to check if you’re affected Kernel and filesystem: XFS on 5.18+, or ext4 on 6.16+. Readahead size: cat /sys/block/<dev>/queue/{read_ahead_kb,optimal_io_size}. Large mappings: FilePmdMapped in /proc/meminfo or smaps for the gcache mapping, and thp_file_mapped climbing in /proc/vmstat. Bursts: about every 30–35 s (unless different timers in vm.dirty_expire_centisecs and vm.dirty_writeback_centisecs) in iostat, with per-thread pidstat -d charges in up to 2 MB multiples. More on Large Folios Impact Cases During research, I found another interesting story. It turns out that mixing active write workloads using both page cache and bypassing it with O_DIRECT may lead to additional unexpected stalls. This time it was visible with RocksDB, as described here: https://blog.wegenz.in/posts/page-cache-reclaim-foliotune-direct-io-stalemate The article was created by a human. The post When a Small Change Writes Too Much – How A Linux Kernel Improvement Can Strike You in PXC/Galera Workloads appeared first on Percona. We’ve recently been dealing with an interesting customer case where a PXC version upgrade from 5.7 to 8.0 seemed to trigger a completely unexpected problem. Even though the workload and MySQL settings did not change, all PXC members began exhibiting sparse but regular bursts of massive disk writes. Yet the MySQL side workload statistics that we gathered during those bursts did not explain the volume of them at all!

Analyzing these bursts further with pidstat and iostat confirmed that the mysqld process was indeed the sole source of excessive disk writes, and digging further showed the responsible threads to be the Galera receiver thread responsible for subscribing to the replication stream, as well as the main connection threads on the writer node, or Galera applier threads on the secondary members. And what’s most important is that there was a huge disparity between InnoDB engine threads writes (redo, undo, tablespace, etc) and binary log writes versus what was accounted to the Galera threads.

At first, it looked like during the burst, several GB of write-sets were received and applied, yet InnoDB barely changed any pages. All InnoDB-related write activity was completely flat, so they did not participate in these high IO load spikes at all! An example mysqld per-thread breakdown for a few-second burst looked like this:

Thread group Accounted Writes Share
applier (Galera write-set appliers, many threads) 13.0 GB 61%
galera_recv-0 (Galera receiver thread) 5.2 GB 24%
ib_pg_flush* (InnoDB page cleaners) 2.0 GB 9%
ib_log_writer (redo log) 0.76 GB 4%
Galera_srvc (Galera service thread) 0.29 GB 1%

The quick thought on the small redo log vs potentially huge writeset volume was that maybe transactions changing a single column would bloat the binlog events with other large blob/text/json columns, due to binlog_row_image=full, as the ROW image format applies to both Galera writesets and binary logs. But, analyzing the binary logs (with binlogsum) showed moderate and completely flat volume, nothing related to those bursts.

Looking for clues, I asked myself, so what are those Galera threads exactly writing? The receiver thread copies all replicated writesets into the Galera cache file(s), but also reclaims the space when the ring wraps (the ring-buffer allocator walks forward from the oldest buffers and discards released ones).

Then, the applier or connection threads also update the cache according to the transaction’s state. Still, when there are no spikes with big writesets (MySQL write load stays flat), why would we see such massive IO write bursts only accounted to the Galera threads?

And why would that weird behavior start to happen after the PXC upgrade? We could not find any clues in the source code; there were no changes to the Galera plugin or wsrep extensions that would explain these bursts.

Later, we found out additional information, which turned out to be crucial – not only the PXC version, but also the OS was upgraded! Now, the previous one: Debian 11, uses Linux 5.10 LTS kernel series, while the upgraded version 12 uses the Linux 6.1 LTS kernel series. And as the main suspect at this point was the Gcache store, how would the new kernel change the way this file is handled, though? It turns out that it does indeed, and it is related to the fact that the Galera plugin uses the mmap() path to handle the cache!

Page Folios in Modern Linux Kernels

In pursuit of handling large data on machines with big RAM and disk space more efficiently in Linux, in addition to huge pages, something called page folios was introduced in 5.16. What is relevant to this case is that large folios were implemented to also handle the page cache, initially only for the XFS filesystem in kernel 5.18, and yes, the affected cluster kept Galera cache on the XFS partition. Then, large folios support is what makes the kernel 6.1 different from the previous 5.10.

OK, but why would “folios” cause bursts, and why do they not affect InnoDB writes, but only the Galera threads? Now, unlike any other data or log files in MySQL (except TempTable storage) or MariaDB, Galera cache files do not use the write() or pwrite() path, but mmap rather.

This difference pointed me to some reports complaining about the systemd journal’s excessive IO problems, which turns out to be using mmap() path as well!

Funny enough, the problem is not very well known, and the root cause is still being confused to this day, even in the most affected systemd community. Some reports describe the journal IO pressure being higher than expected, but never define the real culprit. The systemd issue that currently covers this problem best is:
https://github.com/systemd/systemd/issues/40262
Related kernel thread discussions:
https://lwn.net/Articles/1092530/
https://lists.openwall.net/linux-kernel/2026/08/20/1169
And despite many good blog posts about optimizing journal IO, I found only this one that points to the folio impact:
https://www.cnblogs.com/32bin/p/22929557
The problem even reached the general tech press, as still unsolved in 2026:
https://northeasttimes.com/2026/08/14/a-single-log-line-eats-49kb-of-disk-and-linux-admins-are-furious/

In short: on a large folios-enabled kernel and filesystem, page cache entries are grouped in folios. In early kernel implementations (5.18–6.5), write()/pwrite() paths were also affected, but InnoDB typically uses O_DIRECT for table spaces (so not using page cache at all), and redo and binlog use small appends, which create order-0 (4 kB) small folios. Then in Linux 6.6, a patch was implemented that added per-block dirty tracking, as now iomap records the overlapping filesystem blocks (filesystem-block granularity): see RFCv3 for details. With mmap writes, any update only tracks a whole folio. Thus, writeback page cache iteration, in the mmap case, will sync the whole folio (up to 2MB) instead of the exact fs block (4kB). In other words, the mmap() path cannot benefit from the improved I/O tracking loops in 6.6+.

So, yes, the difference may be potentially huge, and small random writes to a mmapped file can induce huge IO bursts when the related page cache gets flushed to disk.

There are important variables here:

    • Readahead size via optimal_io_size → read_ahead_kb, per block device, influences the sizes of large folios allocated through it (from 128kB up to even 2MB)
    • Bursts are specifically periodic due to how writeback of dirty data gets invoked – every dirty_writeback_centisecs (default 5 seconds) the kernel checks if the dirty data (tracked per inode) age is old enough via dirty_expire_centisecs (default 30 seconds) – the observed roughly 33-second bursts on the affected PXC nodes, are consistent with this mechanism (although these settings do not guarantee a fixed flush period) when the object, here the Gcache inode, gets flushed when receiving regular writes.
    • Workload characteristics matter a lot – writes accumulated within a single folio will result in way smaller write amplifications as compared to a file being updated in many random places, hence touching many different large folios.

The whole concept is damn complex, so let me share a more technical deep dive schematic in a separate document. Bear in mind, I am not a Linux kernel expert, so I hope there are not too many errors. The document is available here: https://przemekmalkowski.github.io/folios-mmap/

Galera Cache Impact

Galera’s writeset cache has been historically a mmapped file from the very beginning. And there are a couple of factors that make the large folio-induced write amplifications particularly severe here:

  • The cache file is by default 128MB, but very often a much bigger size is used to provide long enough IST coverage.
  • In addition to sequential appends of new writesets, there are many small and random updates that change their status to RELEASED when the transaction is already applied and ready to be overwritten.
  • Quite often, MySQL data, together with the galera.cache file, is placed on a large storage, like RAID arrays. Specifically for software RAID, the Linux kernel automatically increases the readahead size proportionally to the array geometry and the advertised optimal I/O size. For hardware RAID and dm/LVM stacks, it may be the case too. This leads to even larger folios, and that’s exactly what happened for the said customer case.
  • During the PXC member startup, the whole gcache file is scanned sequentially during the recovery process (enabled by default), which can immediately populate the large page cache areas.

Having said that, every PXC/Galera cluster may be affected by extra write amplifications for the galera.cache (and gcache.page.X), if kernel and filesystem versions support large folios. But the impact may be small enough to slip through unnoticed when the default readahead of 128kB is set for the disk block device. The extra IO write pressure may be wrongly assumed to be related to regular (and complex enough) database writes. But when the storage uses bigger readahead, and large folios get to even 2MB in size, the amplification is huge and really difficult to miss.

Possible workarounds

As currently there is no Linux kernel side solution implemented for the mmap() path, there are a couple of workarounds possible, and they apply to any applications using mmapped files, not only PXC/Galera:

  • Depending on the kernel version, using the EXT4 filesystem may be an option, as folio support was introduced there much later (in 6.16) than in XFS (5.18) – if applicable, this allows us to avoid the said amplifications altogether. It helped in this customer’s case!
  • Lower the readahead size for the disk device – for the price of potentially limited performance, the maximum size of large folios will be impacted by it, making the amplifications smaller.
  • Modify the application mmap calls to use MADV_RANDOM via madvise() hints. It will basically disable readahead via enforced random access.
  • Stop using mmap() path in favor of pwrite() on Linux 6.6+.
  • Finally, use a customized modern kernel, compiled without THP support via the disabled CONFIG_TRANSPARENT_HUGEPAGE option – large folios are not created on recent kernels when THP is not available. (Note that disabling THP in GRUB is not enough to disable large folios!)

For PXC, I reported the problem here: https://perconadev.atlassian.net/browse/PXC-5340

How to check if you’re affected

  • Kernel and filesystem: XFS on 5.18+, or ext4 on 6.16+.
  • Readahead size: cat /sys/block/<dev>/queue/{read_ahead_kb,optimal_io_size}.
  • Large mappings: FilePmdMapped in /proc/meminfo or smaps for the gcache mapping, and thp_file_mapped climbing in /proc/vmstat.
  • Bursts: about every 30–35 s (unless different timers in vm.dirty_expire_centisecs and vm.dirty_writeback_centisecs) in iostat, with per-thread pidstat -d charges in up to 2 MB multiples.

More on Large Folios Impact Cases

During research, I found another interesting story. It turns out that mixing active write workloads using both page cache and bypassing it with O_DIRECT may lead to additional unexpected stalls. This time it was visible with RocksDB, as described here:

https://blog.wegenz.in/posts/page-cache-reclaim-foliotune-direct-io-stalemate

The article was created by a human.

The post When a Small Change Writes Too Much – How A Linux Kernel Improvement Can Strike You in PXC/Galera Workloads appeared first on Percona.

]]> Fri, 16 Oct 2026 12:58:08 +0000 MySQL Performance Blog Hardware and Storage Insight for DBAs MariaDB MySQL Uncategorized XtraDB Cluster (PXC) galera large folios MMAP page cache Just Because You Can SELECT Everything Doesn’t Mean You Should 8d747a8e361a94a5dcd3ef653d0e4e0c https://blogs.oracle.com/mysql/mysql-select-star-best-practices Last week, I shared a meme in the MySQL Group on LinkedIn about using SELECT * in MySQL queries. Today, I’m backing up the punchline with something every joke desperately needs: a technical explanation. We’ve all written queries like this: It’s quick. It’s convenient. It lets you postpone remembering whether the column is called customer_name, […] Thu, 15 Oct 2026 15:19:36 +0000 Oracle MySQL Group MySQL MySQL Community mysql MySQL Best Practices mysqlcommunity JavaScript in MySQL: Extending SQL, Not Moving the Application https://www.percona.com/?p=54696 https://www.percona.com/blog/javascript-in-mysql-extending-sql-not-moving-the-application/ At Percona Live Europe 2026, Dmitry Lenev and I presented JavaScript in My Database? Over My Dead Query, the story of component_js_lang, the V8-based JavaScript engine for Percona Server for MySQL 8.4. That experience told us few things: The presentation was seen as “for developers” What most DBA or DevOps heard was “JavaScript in the database” = “application logic in the database” This let most of the people pictured developers shipping their business logic into stored procedures. Order workflows, pricing rules, REST calls, all running inside mysqld.  That picture is wrong, and in this post I want to replace it with the right one, using the same four demos we showed on stage. The fear is reasonable, the target is wrong DBAs have earned their skepticism. Many of us have inherited a schema where thousands of lines of stored procedures were the application: impossible to test, impossible to version, impossible to scale out, and owned by nobody who still works there. So when a new language arrives inside the server, the reflex is to protect the database from becoming an application server again.  We share that reflex. The point of this post is that JavaScript routines, used as designed, do not move us in that direction. They move the opposite way: they make SQL stronger so that less logic needs to be pulled out of the database and re-implemented somewhere else. The engine itself also enforces the boundary. Inside component_js_lang a routine has: No network I/O. No fetch(), no sockets. A routine cannot call your payment provider or another microservice. No file system access. It cannot read configs, keys or files. No Node.js or DOM APIs. No require(), no process, no npm ecosystem. No useful async. Everything runs synchronously inside the statement. You cannot build an application with that toolbox. However you can build very good functions! What JavaScript routines are actually for A JavaScript routine extends the SQL vocabulary. It is a new function you call from SELECT, JOIN, WHERE, GROUP BY or a trigger, exactly like LOWER() or JSON_EXTRACT(). The only new syntax is LANGUAGE JS:CREATE FUNCTION norm_keys(d JSON) RETURNS JSON DETERMINISTIC NO SQL LANGUAGE JS AS $$ const walk = (j) =&gt; { if (j === null || typeof j !== 'object') return j if (Array.isArray(j)) return j.map(walk) const out = {} for (const [k, v] of Object.entries(j)) out[k.toLowerCase()] = walk(v) return out } return walk(typeof d === 'string' ? JSON.parse(d) : d) $$;  The rule we kept repeating on stage is simple: SQL owns the set, JavaScript owns the value. SQL is unbeatable at filtering, joining, ordering, windowing and aggregating. Where SQL and its procedural language stop is a short, very specific list: Recursion over a shape not known in advance. JSON_TABLE needs every path as a literal. Arrays. SQL has no array type, so any algorithm that fills a grid has nowhere to live. Big integers. BIGINT stops at 19 digits; an IBAN checksum needs 29. Real iteration with aggregates. A recursive CTE cannot contain an aggregate or window function (ERROR 3575), and a WHILE loop in PL pays an executor round trip on every statement. Rich string handling. UPPER() and LOWER() do not know that McDonald has a capital D. None of these is business logic. They are missing building blocks. JavaScript supplies them and hands the result straight back to SQL. The JavaScript routine lives inside the SQL layer, not beside the application: a query calls it, and it returns a value to that query. Four demos, one pattern Each demo in the pleu_2026 repository follows the same shape: SQL does the set work, a small JavaScript function does the one step SQL cannot phrase, and the decision that matters to the business stays in the application. Demo What SQL could not do The JavaScript piece Result What stays in the application Normalize JSON keys Walk a JSON tree of unknown shape; four partners spell customer.id four ways norm_keys(), 9 lines One report went from 51 lines with four UNION ALL branches to 19 lines with no partner names Which partners to onboard and what the report means Validate input at the door Proper-case names, normalise phones, run an IBAN mod-97 on a 29-digit number proper_name(), clean_email(), clean_phone(), valid_iban(), clean_contact() 460 of 2,000 rows caught, each with a list of what was wrong What to do with a rejected contact: fix, call the customer, quarantine Deduplicate customers Levenshtein distance needs a grid, and SQL has no arrays lev_distance() inside a JOIN predicate Name equality found 21.7% of duplicates; blocking in SQL plus two distances found about 80% at 99.7% precision Whether to merge two records, and how Where should Federico live? Iterate to a geometric median; recursive CTEs refuse aggregates Weiszfeld procedure 11.00 s in procedural SQL vs 0.52 s in JavaScript; 327,033 statements vs 6 Whether Federico actually moves A few details are worth calling out, because they are exactly where DBAs and developers should meet. Dedupe: SQL narrows, JavaScript decides the distance. Comparing 3,000 customers with 2,000 signups is six million pairs. We never let JavaScript see them all. Ordinary join predicates on city and initials cut the work to 21,352 pairs, and only those reach lev_distance(). The function returns a number. Merging two customers is a business decision, and it stays in the application. Validation: the function reports, the policy decides. clean_contact() returns the cleaned values and an array of errors. A trigger uses it to guarantee that nothing dirty lands in contacts, whoever does the insert: web app, mobile app, nightly import, or someone at a mysql prompt at 2am. What happens next to a rejected row is still the application’s job. Federico: three steps, one of them JavaScript. Turning 44,347 raw stops into 16,232 weighted endpoints is two window functions, and nothing beats SQL there. Turning the resulting coordinate into a district is plain joins. JavaScript only runs the inner loop in the middle. Note that this demo uses session.runSql(), so today it runs on MySQL Enterprise; SQL from JavaScript is the top roadmap item for Percona Server. Where we draw the line A JavaScript routine belongs in the database when it makes a data operation possible or cheaper. It belongs in the application when it decides what the business does. In practice we use five questions before accepting a routine: Does it work on values the database already holds? If it needs an external service, a file or a user session, it is application code. Is it deterministic, with no side effects outside the transaction? DETERMINISTIC NO SQL is not only a binlog requirement; it is a good design filter. Is it called from SQL? A good routine sits inside a SELECT, a JOIN, a JSON_TABLE or a trigger. If nothing in SQL calls it, ask why it is in the database. Is the result smaller than the input, or does it save a round trip? Two numbers out instead of 5,000 names, emails and cities is the kind of win we want. Would every client need the same rule? Validation that the web app, the mobile app and the importer all must apply is a data rule. Pricing for a campaign that changes next week is business policy. Belongs in a JS routine Stays in the application Normalising and validating values (keys, names, phones, IBAN) Workflows, approvals, notifications Distance, similarity, checksum and parsing functions Decisions based on those scores: merge, reject, refund The inner loop of a numeric algorithm over rows SQL selected Orchestration across services and external APIs Integrity rules every writer must obey UI, sessions, authentication flows Reshaping JSON so SQL can query it Business rules owned by product and changed often If a routine starts to need configuration tables, feature flags or calls to another system, that is our signal that it has crossed the line. Why developers should ask for it For developers this is not about giving up control of the application. It is about no longer having to pull data out of MySQL just because SQL could not express one step. It is the language you already write. ECMAScript on V8, the same runtime as Chrome and Node.js. JSON columns arrive as native JS objects, no JSON.parse() needed. Fewer round trips. In the commute demo one CALL read 16,232 rows and wrote 2,000 answers with zero rows crossing the network. The speed-up from 11.00 s to 0.52 s came from not crossing the executor boundary 327,033 times, not from JavaScript being a faster language. Less personal data in motion. The dedupe query returns two numbers instead of shipping 5,000 names, emails and cities to an app server, and into its logs. Your GDPR scope stays in the database. One rule, written once. Validation that lives in five clients becomes four out-of-date copies. A function and a trigger are one copy nobody can bypass. Simpler queries in your code. norm_keys() turned a 51-line, partner-specific report into 19 lines. Partner five becomes a data change, not a code change. Pure functions are easy to test. Deterministic, no I/O, no hidden state: the same unit-testing habits you already have. How we suggest developers and DBAs work together: The developer proposes the function, with the SQL that will call it. The DBA reviews it against the five questions above, and checks if the call site uses blocking or filtering so the function runs on as few rows as possible. Grant CREATE_JS_ROUTINE deliberately; it is required on top of CREATE ROUTINE, so who can write JavaScript is an explicit choice. Keep the routine in version control next to the schema migrations. SQL did not get worse, it got a companion JavaScript in MySQL is not an invitation to rebuild the application inside the server.  It is a set of sharp tools for the few places where SQL and PL run out: recursion, arrays, big numbers and tight loops. The set work stays in SQL, the business logic stays in the application, and JavaScript fills the gap between them. If you want to try it, component_js_lang is available as a tech preview in Percona Server for MySQL 8.4.5-5 and later, free and open source, while the equivalent feature from Oracle requires MySQL Enterprise.  You can easily use the code available here Demo code from Percona Live Europe 2026 to perform some tests, just doing cut and paste.  Tell us what you think on the Percona Community Forum or in Jira with the label PS-JS-LANG. Your feedback decides what goes into GA, starting with SQL execution from JavaScript routines. Resources Percona Server js_lang documentation Demo code from Percona Live Europe 2026 Source code on GitHub The post JavaScript in MySQL: Extending SQL, Not Moving the Application appeared first on Percona. At Percona Live Europe 2026, Dmitry Lenev and I presented JavaScript in My Database? Over My Dead Query, the story of component_js_lang, the V8-based JavaScript engine for Percona Server for MySQL 8.4. That experience told us few things:

  • The presentation was seen as “for developers”
  • What most DBA or DevOps heard was “JavaScript in the database” = “application logic in the database”

This let most of the people pictured developers shipping their business logic into stored procedures. Order workflows, pricing rules, REST calls, all running inside mysqld. 

That picture is wrong, and in this post I want to replace it with the right one, using the same four demos we showed on stage.

The fear is reasonable, the target is wrong

DBAs have earned their skepticism. Many of us have inherited a schema where thousands of lines of stored procedures were the application: impossible to test, impossible to version, impossible to scale out, and owned by nobody who still works there.

So when a new language arrives inside the server, the reflex is to protect the database from becoming an application server again. 

We share that reflex.

The point of this post is that JavaScript routines, used as designed, do not move us in that direction.
They move the opposite way: they make SQL stronger so that less logic needs to be pulled out of the database and re-implemented somewhere else.

The engine itself also enforces the boundary. Inside component_js_lang a routine has:

  • No network I/O. No fetch(), no sockets. A routine cannot call your payment provider or another microservice.
  • No file system access. It cannot read configs, keys or files.
  • No Node.js or DOM APIs. No require(), no process, no npm ecosystem.
  • No useful async. Everything runs synchronously inside the statement.

You cannot build an application with that toolbox. However you can build very good functions!

What JavaScript routines are actually for

A JavaScript routine extends the SQL vocabulary. It is a new function you call from SELECT, JOIN, WHERE, GROUP BY or a trigger, exactly like LOWER() or JSON_EXTRACT(). The only new syntax is LANGUAGE JS:

CREATE FUNCTION norm_keys(d JSON) RETURNS JSON
DETERMINISTIC NO SQL LANGUAGE JS AS $$
  const walk = (j) => {
    if (j === null || typeof j !== 'object') return j
    if (Array.isArray(j)) return j.map(walk)
    const out = {}
    for (const [k, v] of Object.entries(j))
      out[k.toLowerCase()] = walk(v)
    return out
  }
  return walk(typeof d === 'string' ? JSON.parse(d) : d)
$$;

 

The rule we kept repeating on stage is simple: SQL owns the set, JavaScript owns the value. SQL is unbeatable at filtering, joining, ordering, windowing and aggregating. Where SQL and its procedural language stop is a short, very specific list:

  • Recursion over a shape not known in advance. JSON_TABLE needs every path as a literal.
  • Arrays. SQL has no array type, so any algorithm that fills a grid has nowhere to live.
  • Big integers. BIGINT stops at 19 digits; an IBAN checksum needs 29.
  • Real iteration with aggregates. A recursive CTE cannot contain an aggregate or window function (ERROR 3575), and a WHILE loop in PL pays an executor round trip on every statement.
  • Rich string handling. UPPER() and LOWER() do not know that McDonald has a capital D.

None of these is business logic. They are missing building blocks. JavaScript supplies them and hands the result straight back to SQL.

The JavaScript routine lives inside the SQL layer, not beside the application: a query calls it, and it returns a value to that query.

Four demos, one pattern

Each demo in the pleu_2026 repository follows the same shape: SQL does the set work, a small JavaScript function does the one step SQL cannot phrase, and the decision that matters to the business stays in the application.

Demo What SQL could not do The JavaScript piece Result What stays in the application
Normalize JSON keys Walk a JSON tree of unknown shape; four partners spell customer.id four ways norm_keys(), 9 lines One report went from 51 lines with four UNION ALL branches to 19 lines with no partner names Which partners to onboard and what the report means
Validate input at the door Proper-case names, normalise phones, run an IBAN mod-97 on a 29-digit number proper_name(), clean_email(), clean_phone(), valid_iban(), clean_contact() 460 of 2,000 rows caught, each with a list of what was wrong What to do with a rejected contact: fix, call the customer, quarantine
Deduplicate customers Levenshtein distance needs a grid, and SQL has no arrays lev_distance() inside a JOIN predicate Name equality found 21.7% of duplicates; blocking in SQL plus two distances found about 80% at 99.7% precision Whether to merge two records, and how
Where should Federico live? Iterate to a geometric median; recursive CTEs refuse aggregates Weiszfeld procedure 11.00 s in procedural SQL vs 0.52 s in JavaScript; 327,033 statements vs 6 Whether Federico actually moves

A few details are worth calling out, because they are exactly where DBAs and developers should meet.

Dedupe: SQL narrows, JavaScript decides the distance. Comparing 3,000 customers with 2,000 signups is six million pairs. We never let JavaScript see them all. Ordinary join predicates on city and initials cut the work to 21,352 pairs, and only those reach lev_distance(). The function returns a number. Merging two customers is a business decision, and it stays in the application.

Validation: the function reports, the policy decides. clean_contact() returns the cleaned values and an array of errors. A trigger uses it to guarantee that nothing dirty lands in contacts, whoever does the insert: web app, mobile app, nightly import, or someone at a mysql prompt at 2am. What happens next to a rejected row is still the application’s job.

Federico: three steps, one of them JavaScript. Turning 44,347 raw stops into 16,232 weighted endpoints is two window functions, and nothing beats SQL there. Turning the resulting coordinate into a district is plain joins. JavaScript only runs the inner loop in the middle. Note that this demo uses session.runSql(), so today it runs on MySQL Enterprise; SQL from JavaScript is the top roadmap item for Percona Server.

Where we draw the line

A JavaScript routine belongs in the database when it makes a data operation possible or cheaper. It belongs in the application when it decides what the business does. In practice we use five questions before accepting a routine:

  1. Does it work on values the database already holds? If it needs an external service, a file or a user session, it is application code.
  2. Is it deterministic, with no side effects outside the transaction? DETERMINISTIC NO SQL is not only a binlog requirement; it is a good design filter.
  3. Is it called from SQL? A good routine sits inside a SELECT, a JOIN, a JSON_TABLE or a trigger. If nothing in SQL calls it, ask why it is in the database.
  4. Is the result smaller than the input, or does it save a round trip? Two numbers out instead of 5,000 names, emails and cities is the kind of win we want.
  5. Would every client need the same rule? Validation that the web app, the mobile app and the importer all must apply is a data rule. Pricing for a campaign that changes next week is business policy.
Belongs in a JS routine Stays in the application
Normalising and validating values (keys, names, phones, IBAN) Workflows, approvals, notifications
Distance, similarity, checksum and parsing functions Decisions based on those scores: merge, reject, refund
The inner loop of a numeric algorithm over rows SQL selected Orchestration across services and external APIs
Integrity rules every writer must obey UI, sessions, authentication flows
Reshaping JSON so SQL can query it Business rules owned by product and changed often

If a routine starts to need configuration tables, feature flags or calls to another system, that is our signal that it has crossed the line.

Why developers should ask for it

For developers this is not about giving up control of the application. It is about no longer having to pull data out of MySQL just because SQL could not express one step.

  • It is the language you already write. ECMAScript on V8, the same runtime as Chrome and Node.js. JSON columns arrive as native JS objects, no JSON.parse() needed.
  • Fewer round trips. In the commute demo one CALL read 16,232 rows and wrote 2,000 answers with zero rows crossing the network. The speed-up from 11.00 s to 0.52 s came from not crossing the executor boundary 327,033 times, not from JavaScript being a faster language.
  • Less personal data in motion. The dedupe query returns two numbers instead of shipping 5,000 names, emails and cities to an app server, and into its logs. Your GDPR scope stays in the database.
  • One rule, written once. Validation that lives in five clients becomes four out-of-date copies. A function and a trigger are one copy nobody can bypass.
  • Simpler queries in your code. norm_keys() turned a 51-line, partner-specific report into 19 lines. Partner five becomes a data change, not a code change.
  • Pure functions are easy to test. Deterministic, no I/O, no hidden state: the same unit-testing habits you already have.

How we suggest developers and DBAs work together:

  1. The developer proposes the function, with the SQL that will call it.
  2. The DBA reviews it against the five questions above, and checks if the call site uses blocking or filtering so the function runs on as few rows as possible.
  3. Grant CREATE_JS_ROUTINE deliberately; it is required on top of CREATE ROUTINE, so who can write JavaScript is an explicit choice.
  4. Keep the routine in version control next to the schema migrations.

SQL did not get worse, it got a companion

JavaScript in MySQL is not an invitation to rebuild the application inside the server. 

It is a set of sharp tools for the few places where SQL and PL run out: recursion, arrays, big numbers and tight loops.
The set work stays in SQL, the business logic stays in the application, and JavaScript fills the gap between them.

If you want to try it, component_js_lang is available as a tech preview in Percona Server for MySQL 8.4.5-5 and later, free and open source, while the equivalent feature from Oracle requires MySQL Enterprise. 

You can easily use the code available here Demo code from Percona Live Europe 2026 to perform some tests, just doing cut and paste. 

Tell us what you think on the Percona Community Forum or in Jira with the label PS-JS-LANG. Your feedback decides what goes into GA, starting with SQL execution from JavaScript routines.

Resources

The post JavaScript in MySQL: Extending SQL, Not Moving the Application appeared first on Percona.

]]> Thu, 15 Oct 2026 15:12:30 +0000 Marco Tusa Database Trends Insight for DBAs Insight for Developers MySQL Open Source SQL Uncategorized Application Logic vs Database Logic component_js_lang Data Validation Database Administration (DBA) Database Architecture Javascript JSON Processing Percona Live E JavaScript in MySQL: Extending SQL, Not Moving the Application http://www.tusacentral.com/joomla/index.php/mysql-blogs/272-javascript-in-mysql-extending-sql-not-moving-the-application http://www.tusacentral.com/joomla/index.php/mysql-blogs/272-javascript-in-mysql-extending-sql-not-moving-the-application At Percona Live Europe 2026, Dmitry Lenev and I presented JavaScript in My Database? Over My Dead Query, the story of component_js_lang, the V8-based JavaScript engine for Percona Server for MySQL 8.4. That experience told us few things: The presentation was seen as “for developers” What most DBA or DevOps heard was "JavaScript in the database" = "application logic in the database" This let most of the people pictured developers shipping their business logic into stored procedures. Order workflows, pricing rules, REST calls, all running inside mysqld.  That picture is wrong, and in this post I want to replace it with the right one, using the same four demos we showed on stage. The fear is reasonable, the target is wrong DBAs have earned their skepticism. Many of us have inherited a schema where thousands of lines of stored procedures were the application: impossible to test, impossible to version, impossible to scale out, and owned by nobody who still works there. So when a new language arrives inside the server, the reflex is to protect the database from becoming an application server again.    We share that reflex. The point of this post is that JavaScript routines, used as designed, do not move us in that direction. They move the opposite way: they make SQL stronger so that less logic needs to be pulled out of the database and re-implemented somewhere else. The engine itself also enforces the boundary. Inside component_js_lang a routine has: No network I/O. No fetch(), no sockets. A routine cannot call your payment provider or another microservice. No file system access. It cannot read configs, keys or files. No Node.js or DOM APIs. No require(), no process, no npm ecosystem. No useful async. Everything runs synchronously inside the statement. You cannot build an application with that toolbox. However you can build very good functions! What JavaScript routines are actually for A JavaScript routine extends the SQL vocabulary. It is a new function you call from SELECT, JOIN, WHERE, GROUP BY or a trigger, exactly like LOWER() or JSON_EXTRACT(). The only new syntax is LANGUAGE JS: CREATE FUNCTION norm_keys(d JSON) RETURNS JSON DETERMINISTIC NO SQL LANGUAGE JS AS $$ const walk = (j) =&gt; { if (j === null || typeof j !== 'object') return j if (Array.isArray(j)) return j.map(walk) const out = {} for (const [k, v] of Object.entries(j)) out[k.toLowerCase()] = walk(v) return out } return walk(typeof d === 'string' ? JSON.parse(d) : d) $$;   The rule we kept repeating on stage is simple: SQL owns the set, JavaScript owns the value. SQL is unbeatable at filtering, joining, ordering, windowing and aggregating. Where SQL and its procedural language stop is a short, very specific list: Recursion over a shape not known in advance. JSON_TABLE needs every path as a literal. Arrays. SQL has no array type, so any algorithm that fills a grid has nowhere to live. Big integers. BIGINT stops at 19 digits; an IBAN checksum needs 29. Real iteration with aggregates. A recursive CTE cannot contain an aggregate or window function (ERROR 3575), and a WHILE loop in PL pays an executor round trip on every statement. Rich string handling. UPPER() and LOWER() do not know that McDonald has a capital D. None of these is business logic. They are missing building blocks. JavaScript supplies them and hands the result straight back to SQL. The JavaScript routine lives inside the SQL layer, not beside the application: a query calls it, and it returns a value to that query. Four demos, one pattern Each demo in the pleu_2026 repository follows the same shape: SQL does the set work, a small JavaScript function does the one step SQL cannot phrase, and the decision that matters to the business stays in the application. Demo What SQL could not do The JavaScript piece Result What stays in the application Normalize JSON keys Walk a JSON tree of unknown shape; four partners spell customer.id four ways norm_keys(), 9 lines One report went from 51 lines with four UNION ALL branches to 19 lines with no partner names Which partners to onboard and what the report means Validate input at the door Proper-case names, normalise phones, run an IBAN mod-97 on a 29-digit number proper_name(), clean_email(), clean_phone(), valid_iban(), clean_contact() 460 of 2,000 rows caught, each with a list of what was wrong What to do with a rejected contact: fix, call the customer, quarantine Deduplicate customers Levenshtein distance needs a grid, and SQL has no arrays lev_distance() inside a JOIN predicate Name equality found 21.7% of duplicates; blocking in SQL plus two distances found about 80% at 99.7% precision Whether to merge two records, and how Where should Federico live? Iterate to a geometric median; recursive CTEs refuse aggregates Weiszfeld procedure 11.00 s in procedural SQL vs 0.52 s in JavaScript; 327,033 statements vs 6 Whether Federico actually moves A few details are worth calling out, because they are exactly where DBAs and developers should meet. Dedupe: SQL narrows, JavaScript decides the distance. Comparing 3,000 customers with 2,000 signups is six million pairs. We never let JavaScript see them all. Ordinary join predicates on city and initials cut the work to 21,352 pairs, and only those reach lev_distance(). The function returns a number. Merging two customers is a business decision, and it stays in the application. Validation: the function reports, the policy decides. clean_contact() returns the cleaned values and an array of errors. A trigger uses it to guarantee that nothing dirty lands in contacts, whoever does the insert: web app, mobile app, nightly import, or someone at a mysql prompt at 2am. What happens next to a rejected row is still the application's job. Federico: three steps, one of them JavaScript. Turning 44,347 raw stops into 16,232 weighted endpoints is two window functions, and nothing beats SQL there. Turning the resulting coordinate into a district is plain joins. JavaScript only runs the inner loop in the middle. Note that this demo uses session.runSql(), so today it runs on MySQL Enterprise; SQL from JavaScript is the top roadmap item for Percona Server. Where we draw the line A JavaScript routine belongs in the database when it makes a data operation possible or cheaper. It belongs in the application when it decides what the business does. In practice we use five questions before accepting a routine: Does it work on values the database already holds? If it needs an external service, a file or a user session, it is application code. Is it deterministic, with no side effects outside the transaction? DETERMINISTIC NO SQL is not only a binlog requirement; it is a good design filter. Is it called from SQL? A good routine sits inside a SELECT, a JOIN, a JSON_TABLE or a trigger. If nothing in SQL calls it, ask why it is in the database. Is the result smaller than the input, or does it save a round trip? Two numbers out instead of 5,000 names, emails and cities is the kind of win we want. Would every client need the same rule? Validation that the web app, the mobile app and the importer all must apply is a data rule. Pricing for a campaign that changes next week is business policy. Belongs in a JS routine Stays in the application Normalising and validating values (keys, names, phones, IBAN) Workflows, approvals, notifications Distance, similarity, checksum and parsing functions Decisions based on those scores: merge, reject, refund The inner loop of a numeric algorithm over rows SQL selected Orchestration across services and external APIs Integrity rules every writer must obey UI, sessions, authentication flows Reshaping JSON so SQL can query it Business rules owned by product and changed often If a routine starts to need configuration tables, feature flags or calls to another system, that is our signal that it has crossed the line. Why developers should ask for it For developers this is not about giving up control of the application. It is about no longer having to pull data out of MySQL just because SQL could not express one step. It is the language you already write. ECMAScript on V8, the same runtime as Chrome and Node.js. JSON columns arrive as native JS objects, no JSON.parse() needed. Fewer round trips. In the commute demo one CALL read 16,232 rows and wrote 2,000 answers with zero rows crossing the network. The speed-up from 11.00 s to 0.52 s came from not crossing the executor boundary 327,033 times, not from JavaScript being a faster language. Less personal data in motion. The dedupe query returns two numbers instead of shipping 5,000 names, emails and cities to an app server, and into its logs. Your GDPR scope stays in the database. One rule, written once. Validation that lives in five clients becomes four out-of-date copies. A function and a trigger are one copy nobody can bypass. Simpler queries in your code. norm_keys() turned a 51-line, partner-specific report into 19 lines. Partner five becomes a data change, not a code change. Pure functions are easy to test. Deterministic, no I/O, no hidden state: the same unit-testing habits you already have. How we suggest developers and DBAs work together: The developer proposes the function, with the SQL that will call it. The DBA reviews it against the five questions above, and checks if the call site uses blocking or filtering so the function runs on as few rows as possible. Grant CREATE_JS_ROUTINE deliberately; it is required on top of CREATE ROUTINE, so who can write JavaScript is an explicit choice. Keep the routine in version control next to the schema migrations. SQL did not get worse, it got a companion JavaScript in MySQL is not an invitation to rebuild the application inside the server.  It is a set of sharp tools for the few places where SQL and PL run out: recursion, arrays, big numbers and tight loops. The set work stays in SQL, the business logic stays in the application, and JavaScript fills the gap between them. If you want to try it, component_js_lang is available as a tech preview in Percona Server for MySQL 8.4.5-5 and later, free and open source, while the equivalent feature from Oracle requires MySQL Enterprise.  You can easily use the code available here Demo code from Percona Live Europe 2026 to perform some tests, just doing cut and paste.  Tell us what you think on the Percona Community Forum or in Jira with the label PS-JS-LANG. Your feedback decides what goes into GA, starting with SQL execution from JavaScript routines. Resources Percona Server js_lang documentation Demo code from Percona Live Europe 2026 Source code on GitHub scaled goatAt Percona Live Europe 2026, Dmitry Lenev and I presented JavaScript in My Database? Over My Dead Query, the story of component_js_lang, the V8-based JavaScript engine for Percona Server for MySQL 8.4. That experience told us few things:

  • The presentation was seen as “for developers”
  • What most DBA or DevOps heard was "JavaScript in the database" = "application logic in the database"

This let most of the people pictured developers shipping their business logic into stored procedures. Order workflows, pricing rules, REST calls, all running inside mysqld. 

That picture is wrong, and in this post I want to replace it with the right one, using the same four demos we showed on stage.

The fear is reasonable, the target is wrong

DBAs have earned their skepticism. Many of us have inherited a schema where thousands of lines of stored procedures were the application: impossible to test, impossible to version, impossible to scale out, and owned by nobody who still works there.

So when a new language arrives inside the server, the reflex is to protect the database from becoming an application server again. 

 

We share that reflex.

The point of this post is that JavaScript routines, used as designed, do not move us in that direction. They move the opposite way: they make SQL stronger so that less logic needs to be pulled out of the database and re-implemented somewhere else.

The engine itself also enforces the boundary. Inside component_js_lang a routine has:

  • No network I/O. No fetch(), no sockets. A routine cannot call your payment provider or another microservice.
  • No file system access. It cannot read configs, keys or files.
  • No Node.js or DOM APIs. No require(), no process, no npm ecosystem.
  • No useful async. Everything runs synchronously inside the statement.

You cannot build an application with that toolbox. However you can build very good functions!

What JavaScript routines are actually for

A JavaScript routine extends the SQL vocabulary. It is a new function you call from SELECT, JOIN, WHERE, GROUP BY or a trigger, exactly like LOWER() or JSON_EXTRACT(). The only new syntax is LANGUAGE JS:

CREATE FUNCTION norm_keys(d JSON) RETURNS JSON
DETERMINISTIC NO SQL LANGUAGE JS AS $$
  const walk = (j) => {
    if (j === null || typeof j !== 'object') return j
    if (Array.isArray(j)) return j.map(walk)
    const out = {}
    for (const [k, v] of Object.entries(j))
      out[k.toLowerCase()] = walk(v)
    return out
  }
  return walk(typeof d === 'string' ? JSON.parse(d) : d)
$$;

 

The rule we kept repeating on stage is simple: SQL owns the set, JavaScript owns the value. SQL is unbeatable at filtering, joining, ordering, windowing and aggregating. Where SQL and its procedural language stop is a short, very specific list:

  • Recursion over a shape not known in advance. JSON_TABLE needs every path as a literal.
  • Arrays. SQL has no array type, so any algorithm that fills a grid has nowhere to live.
  • Big integers. BIGINT stops at 19 digits; an IBAN checksum needs 29.
  • Real iteration with aggregates. A recursive CTE cannot contain an aggregate or window function (ERROR 3575), and a WHILE loop in PL pays an executor round trip on every statement.
  • Rich string handling. UPPER() and LOWER() do not know that McDonald has a capital D.

None of these is business logic. They are missing building blocks. JavaScript supplies them and hands the result straight back to SQL.

schema

The JavaScript routine lives inside the SQL layer, not beside the application: a query calls it, and it returns a value to that query.

Four demos, one pattern

slide1

Each demo in the pleu_2026 repository follows the same shape: SQL does the set work, a small JavaScript function does the one step SQL cannot phrase, and the decision that matters to the business stays in the application.

Demo What SQL could not do The JavaScript piece Result What stays in the application
Normalize JSON keys Walk a JSON tree of unknown shape; four partners spell customer.id four ways norm_keys(), 9 lines One report went from 51 lines with four UNION ALL branches to 19 lines with no partner names Which partners to onboard and what the report means
Validate input at the door Proper-case names, normalise phones, run an IBAN mod-97 on a 29-digit number proper_name(), clean_email(), clean_phone(), valid_iban(), clean_contact() 460 of 2,000 rows caught, each with a list of what was wrong What to do with a rejected contact: fix, call the customer, quarantine
Deduplicate customers Levenshtein distance needs a grid, and SQL has no arrays lev_distance() inside a JOIN predicate Name equality found 21.7% of duplicates; blocking in SQL plus two distances found about 80% at 99.7% precision Whether to merge two records, and how
Where should Federico live? Iterate to a geometric median; recursive CTEs refuse aggregates Weiszfeld procedure 11.00 s in procedural SQL vs 0.52 s in JavaScript; 327,033 statements vs 6 Whether Federico actually moves

A few details are worth calling out, because they are exactly where DBAs and developers should meet.

Dedupe: SQL narrows, JavaScript decides the distance. Comparing 3,000 customers with 2,000 signups is six million pairs. We never let JavaScript see them all. Ordinary join predicates on city and initials cut the work to 21,352 pairs, and only those reach lev_distance(). The function returns a number. Merging two customers is a business decision, and it stays in the application.

Validation: the function reports, the policy decides. clean_contact() returns the cleaned values and an array of errors. A trigger uses it to guarantee that nothing dirty lands in contacts, whoever does the insert: web app, mobile app, nightly import, or someone at a mysql prompt at 2am. What happens next to a rejected row is still the application's job.

Federico: three steps, one of them JavaScript. Turning 44,347 raw stops into 16,232 weighted endpoints is two window functions, and nothing beats SQL there. Turning the resulting coordinate into a district is plain joins. JavaScript only runs the inner loop in the middle. Note that this demo uses session.runSql(), so today it runs on MySQL Enterprise; SQL from JavaScript is the top roadmap item for Percona Server.

Where we draw the line

A JavaScript routine belongs in the database when it makes a data operation possible or cheaper. It belongs in the application when it decides what the business does. In practice we use five questions before accepting a routine:

  1. Does it work on values the database already holds? If it needs an external service, a file or a user session, it is application code.
  2. Is it deterministic, with no side effects outside the transaction? DETERMINISTIC NO SQL is not only a binlog requirement; it is a good design filter.
  3. Is it called from SQL? A good routine sits inside a SELECT, a JOIN, a JSON_TABLE or a trigger. If nothing in SQL calls it, ask why it is in the database.
  4. Is the result smaller than the input, or does it save a round trip? Two numbers out instead of 5,000 names, emails and cities is the kind of win we want.
  5. Would every client need the same rule? Validation that the web app, the mobile app and the importer all must apply is a data rule. Pricing for a campaign that changes next week is business policy.
Belongs in a JS routine Stays in the application
Normalising and validating values (keys, names, phones, IBAN) Workflows, approvals, notifications
Distance, similarity, checksum and parsing functions Decisions based on those scores: merge, reject, refund
The inner loop of a numeric algorithm over rows SQL selected Orchestration across services and external APIs
Integrity rules every writer must obey UI, sessions, authentication flows
Reshaping JSON so SQL can query it Business rules owned by product and changed often

If a routine starts to need configuration tables, feature flags or calls to another system, that is our signal that it has crossed the line.

Why developers should ask for it

For developers this is not about giving up control of the application. It is about no longer having to pull data out of MySQL just because SQL could not express one step.

  • It is the language you already write. ECMAScript on V8, the same runtime as Chrome and Node.js. JSON columns arrive as native JS objects, no JSON.parse() needed.
  • Fewer round trips. In the commute demo one CALL read 16,232 rows and wrote 2,000 answers with zero rows crossing the network. The speed-up from 11.00 s to 0.52 s came from not crossing the executor boundary 327,033 times, not from JavaScript being a faster language.
  • Less personal data in motion. The dedupe query returns two numbers instead of shipping 5,000 names, emails and cities to an app server, and into its logs. Your GDPR scope stays in the database.
  • One rule, written once. Validation that lives in five clients becomes four out-of-date copies. A function and a trigger are one copy nobody can bypass.
  • Simpler queries in your code. norm_keys() turned a 51-line, partner-specific report into 19 lines. Partner five becomes a data change, not a code change.
  • Pure functions are easy to test. Deterministic, no I/O, no hidden state: the same unit-testing habits you already have.

How we suggest developers and DBAs work together:

  1. The developer proposes the function, with the SQL that will call it.
  2. The DBA reviews it against the five questions above, and checks if the call site uses blocking or filtering so the function runs on as few rows as possible.
  3. Grant CREATE_JS_ROUTINE deliberately; it is required on top of CREATE ROUTINE, so who can write JavaScript is an explicit choice.
  4. Keep the routine in version control next to the schema migrations.

SQL did not get worse, it got a companion

JavaScript in MySQL is not an invitation to rebuild the application inside the server. 

It is a set of sharp tools for the few places where SQL and PL run out: recursion, arrays, big numbers and tight loops. The set work stays in SQL, the business logic stays in the application, and JavaScript fills the gap between them.

If you want to try it, component_js_lang is available as a tech preview in Percona Server for MySQL 8.4.5-5 and later, free and open source, while the equivalent feature from Oracle requires MySQL Enterprise. 

You can easily use the code available here Demo code from Percona Live Europe 2026 to perform some tests, just doing cut and paste. 

Tell us what you think on the Percona Community Forum or in Jira with the label PS-JS-LANG. Your feedback decides what goes into GA, starting with SQL execution from JavaScript routines.

Resources

]]> Thu, 15 Oct 2026 15:04:29 +0000 Marco Tusa MySQL Visualizing MySQL Telemetry with Grafana 6ed583627c0febbc5ca7f6bdaa813a36 https://blogs.oracle.com/mysql/visualizing-mysql-telemetry-with-grafana In the first post, we exported MySQL OpenTelemetry metrics directly to Prometheus. In the second, we used DBA questions to decide which metrics matter. This post turns those questions into a Grafana dashboard. The objective is not to display every available metric. It is to make the next investigation obvious. When workload changes, a DBA […] Thu, 15 Oct 2026 04:10:08 +0000 Oracle MySQL Group Monitoring MySQL MySQL Enterprise Announcing VillageSQL Server 0.0.7 6abfdfed4f4d7f0001b09df8 https://villagesql.com/blog/007/ We are pleased to announce the launch of VillageSQL Server 0.0.7. You now have a License to Query (please note this post contains lots of James Bond 007 database puns). In this release, the mysql-8.4 build tracks MySQL Server 8.4.11, the mysql-9.7 build tracks MySQL Server 9.7.2, and the percona-8.4 build carries Percona Server 8.4.11-11. Highlights of this release include the following: The server can now describe a custom index, store its data, report it, and run a nearest-neighbor search through it. This is the foundation of custom indexing and vector search we demoed at Percona Live Amsterdam 2026. CLONE now checks that the recipient has your extensions before it copies anything. An extension can give every connection its own copy of a setting. Every bundled extension, and the server itself, runs under Valgrind and ThreadSanitizer every night. Custom indexes defined by extensions MySQL gives you B-tree, R-tree, hash, and full-text indexes, and that list is fixed. If you want a vector index, you need to wait for a future MySQL release that ships one, and MySQL Community Server doesn't ship one in any version. VillageSQL Server custom indexes let you define an index yourself as an extension. The extension supplies the index type, the storage, the build, and the search, and the server treats the result as a first-class index. This release lays the foundation for that (the Quantum of SQL). SHOW CREATE TABLE renders a custom index as DDL, with its type, the distance profile for each column (hnsw_cosine here), and the WITH parameters the extension defined: KEY `idx_title` (`title_vec` `vsql_vector`.`hnsw_cosine`) USING EXTENDED(`vsql_vector`.`hnsw`) WITH (`ef_construction` = 64, `m` = 8), KEY `idx_body` (`body_vec` `vsql_vector`.`hnsw_l1`) USING EXTENDED(`vsql_vector`.`hnsw`), The server doesn't interpret m or ef_construction on its own. It stores them because the extension asked it to, and prints them back as DDL you can replay. SHOW INDEX and INFORMATION_SCHEMA.STATISTICS now report the extension's index type name. The server also stores the data a custom index needs. A vector column lives outside the row, in storage the extension manages, and InnoDB now indexes it through a reference, so you can index a column far wider than the usual key-length limit. The server can also plan a search through a custom index. The query shape is a distance function over the indexed column in ORDER BY, with a LIMIT. The server routes that query to the extension's index instead of sorting the whole table. The classic optimizer reports the access type as index_distance, and EXPLAIN FORMAT=TREE prints Custom index distance scan on idx_title. The hypergraph optimizer picks the same plan. The server runs the whole scan: it picks the index, asks the extension for the nearest entries, and fetches each row by the primary key the index stored. That means the index has to store the primary key. The vsql-vector-pk extension does, and its HNSW index answers these queries end to end on this release. It is a variant of vsql-vector whose index stores the primary key alongside each entry. The remaining work on custom indexing is index maintenance. On a table with a custom index, the server currently refuses an UPDATE that changes the primary key or a column in any index, ordinary indexes included, and an ALTER TABLE that changes the primary key. It also refuses a DELETE (DELETE Another Day) against an index that identifies its entries by its own handles rather than by row key, because the server can't yet tell the index which entry to drop. The HNSW indexes in vsql-vector and vsql-vector-pk both work this way, so neither extension supports DELETE yet. Other ALTER TABLE operations, including a full table rebuild, keep the index. At Percona Live in Amsterdam on September 9 we demoed vector search in MySQL on a development build. An HNSW index defined entirely by the vsql-vector extension answered a nearest-neighbor query from the graph instead of scanning the table, and we measured recall against the same rows. The metadata, the storage, the DDL, and the server side of that scan all ship in this release. CLONE now checks for your extensions CLONE INSTANCE copies a running server's data into another server. We extended its compatibility checks to cover extensions, alongside the existing checks for plugins, character sets, and configuration. Extension bundles (.veb files) live outside the data directory, so a clone doesn't copy them, but the system table that records which extensions are installed is copied. We check the donor's extension list during the handshake, before any data moves. Every extension installed on the donor has to be present in the recipient's veb_dir with a matching version and a matching .veb checksum, which is the same bar the server applies to its own extensions at startup. This check matters beyond clone. An extension can define a column type whose stored bytes mean nothing without the extension that wrote them. The extension is part of the server's state, so every operational path that copies, moves, or upgrades a server has to account for it. Clone is the first path we covered. Backup and restore, upgrades across codebases, and dependencies between extensions come next. Extension functions can compute their own return type The vsql_tvector example extension defines TVECTOR(N), a fixed-width vector type. A function that joins a TVECTOR(3) and a TVECTOR(4) end to end should return a TVECTOR(7). The server's built-in type rules can't produce that answer: they copy one argument's parameters onto the return value, and they reject a call whose arguments disagree. An extension can now register a function with bind_and_check_types, and the server calls that hook while it resolves the statement: make_func&lt;&amp;tvector_concat&gt;("tvector_concat") .bind_and_check_types&lt;&amp;tvector_concat_bind&gt;() The hook receives each argument's kind, the type parameters already resolved for it, and the literal text when an argument is a string constant. It returns the type parameters for the result, optionally new parameters for each argument, or an error. The hook replaces the built-in rules for that function, so a call the server would have rejected is allowed and the return type is the one the extension computed. vsql_tvector gains 7 functions that use it, covering a computed return dimension, 2 that fix the dimension and inherit the element type, 1 that inherits the dimension and fixes the element type, 2 reductions to a scalar, and an accessor. You can use these as examples when building your own extension. Per-session extension settings An extension can declare per-session variables through the new vsql::session_var capability. Every connection holds its own value, SET SESSION changes it, and the value the extension declared is the global default. INT and STR are supported. Extension system variables, which hold one value for the whole server, work as they did before. The vector work is why this exists. HNSW search has a dial that trades recall against speed, and that choice belongs to the query, so vsql-vector declares the dial per session: SET SESSION vsql_vector.ef_search = 100; For vsql-vector, the default is 40, and a value outside 1 to 65536 is clamped to the nearest bound. Turn it up and the search reads more of the graph, returns better answers, and takes longer. One connection can run an analytics query at 400 while every other connection stays at the default. A function reads the calling session's value through int_var(). The capability is preview, so it needs vsql_allow_preview_extensions=ON. System variables with a fixed list of values For Your Enums Only. A vsql::sys_var variable can now be an enum: a fixed list of names, stored as an index. The server validates the name at SET time and rejects anything off the list, so the extension no longer has to check a free-form string after the fact. Statement events identify the client The Spy Who Logged Me. The vsql::preview::statement_event capability now reports the client process id, the connector library, and the application name, each NULL when the client didn't send it. If you're trying to be The Man with the Golden Query and writing per-query telemetry, that's what attributes a slow query to the service that ran it. Breaking changes DDL that the server used to accept can now be rejected. The server validates type parameters before it hands them to an extension, and for the TYPE, TYPE(N), and TYPE('k=v,...') forms it rejects a duplicate parameter name, a nameless parameter, and a parameter with no value. After the extension's resolve_params callback returns, the server also rejects a rewrite that dropped a parameter. Adding parameters and changing values is still legal. Check your DDL and your extensions against these rules before you upgrade. The preview authentication capability changed too. auto_grant is gone, replaced by roles_mode, which tells the server whether to activate, grant, or sync the roles an identity provider supplies. An authentication extension built against the 0.0.6 preview needs a source change to compile. vsql-oauth2 already has, and its own vsql_oauth2.auto_grant setting still exists. Strengthening testing Tomorrow Never Deadlocks. Every night we run the unit tests, the full VillageSQL test suite, and every bundled extension under both Valgrind memcheck and ThreadSanitizer, so the server and the extensions you get with a release have been through both. We also used gcov to find code the suites never reached and wrote tests for it, and moving our CI runners to Ubuntu 26.04 turned up undefined behavior in our type-parameter parsing along with an upstream bug in the Group Replication plugin. New extensions Three extensions joined this cycle: vsql-duckdb (From Amsterdam with Select): query Parquet, CSV, and JSON files from an ordinary MySQL connection, without exporting the data first or standing up a second query engine. It embeds DuckDB in the server process and reads the files where they already live, in an s3:// bucket or behind an https:// URL. Reading files off the server's own disk works too, once you turn on vsql_duckdb.allow_local_files. It's a port of the PostgreSQL extension pg_duckdb, narrowed to what the extension framework supports today. It uses preview capabilities, so it needs vsql_allow_preview_extensions=ON, and you build it from source. It also needs a server newer than 0.0.6, so this is the first release it runs on. vsql-vector (Dr. No-Scan): the SVECTOR vector type, stored as packed floats in InnoDB pages through the column storage interface, the distance functions to go with it, and the HNSW index type the demo ran on. Its index does not store a row reference, so on this release the server builds and keeps it current on INSERT but cannot answer a query through it. It needs vsql_allow_preview_extensions=ON, and you build it from source. vsql-vector-pk (The Vector Who Loved Me): the same SVECTOR type and HNSW index, with the primary key stored alongside each entry. That is what the nearest-neighbor scan described above needs, so this is the variant that answers queries on this release. A server runs either vsql-vector or vsql-vector-pk, never both. It needs vsql_allow_preview_extensions=ON, and you build it from source. What's ahead The next release is more hardening and production readiness. We're testing VillageSQL against popular ORMs, the JDBC and ODBC connectors, the Windows client, and backup and restore. We're building performance tests for a server running extensions, so that when someone asks what an extension costs in terms of performance we have a number instead of an opinion. Custom indexes carry on in parallel, on the query side and the write side, until an extension's index runs end to end on a released server without the preview flag. Special thanks The native RPM and DEB packaging in this release came from Evgeniy Patlan at Percona. The packages take over an existing MySQL install (On Her Majesty's Secret Server) rather than sitting beside one, sharing its binary names, data directory, systemd unit, and port, and the build dependencies resolve on EL8, EL9, EL10, Ubuntu 24.04, Ubuntu 26.04, Debian 12, and Debian 13. No workflow publishes them yet, so the release artifacts are unchanged for now, but the sources and the build scripts are in the tree. There's more in this release than fits here. The release notes carry the per-PR detail. Our roadmap is published on GitHub, and we'd like to hear what you're building and what we should work on next, either on GitHub or on Discord. If you want to support VillageSQL, please star the repo.

We are pleased to announce the launch of VillageSQL Server 0.0.7. You now have a License to Query (please note this post contains lots of James Bond 007 database puns). In this release, the mysql-8.4 build tracks MySQL Server 8.4.11, the mysql-9.7 build tracks MySQL Server 9.7.2, and the percona-8.4 build carries Percona Server 8.4.11-11.

Highlights of this release include the following:

  • The server can now describe a custom index, store its data, report it, and run a nearest-neighbor search through it. This is the foundation of custom indexing and vector search we demoed at Percona Live Amsterdam 2026.
  • CLONE now checks that the recipient has your extensions before it copies anything.
  • An extension can give every connection its own copy of a setting.
  • Every bundled extension, and the server itself, runs under Valgrind and ThreadSanitizer every night.

Custom indexes defined by extensions

MySQL gives you B-tree, R-tree, hash, and full-text indexes, and that list is fixed. If you want a vector index, you need to wait for a future MySQL release that ships one, and MySQL Community Server doesn't ship one in any version. VillageSQL Server custom indexes let you define an index yourself as an extension. The extension supplies the index type, the storage, the build, and the search, and the server treats the result as a first-class index. This release lays the foundation for that (the Quantum of SQL).

SHOW CREATE TABLE renders a custom index as DDL, with its type, the distance profile for each column (hnsw_cosine here), and the WITH parameters the extension defined:

  KEY `idx_title` (`title_vec` `vsql_vector`.`hnsw_cosine`) USING EXTENDED(`vsql_vector`.`hnsw`) WITH (`ef_construction` = 64, `m` = 8),
  KEY `idx_body` (`body_vec` `vsql_vector`.`hnsw_l1`) USING EXTENDED(`vsql_vector`.`hnsw`),

The server doesn't interpret m or ef_construction on its own. It stores them because the extension asked it to, and prints them back as DDL you can replay. SHOW INDEX and INFORMATION_SCHEMA.STATISTICS now report the extension's index type name. The server also stores the data a custom index needs. A vector column lives outside the row, in storage the extension manages, and InnoDB now indexes it through a reference, so you can index a column far wider than the usual key-length limit.

The server can also plan a search through a custom index. The query shape is a distance function over the indexed column in ORDER BY, with a LIMIT. The server routes that query to the extension's index instead of sorting the whole table. The classic optimizer reports the access type as index_distance, and EXPLAIN FORMAT=TREE prints Custom index distance scan on idx_title. The hypergraph optimizer picks the same plan.

The server runs the whole scan: it picks the index, asks the extension for the nearest entries, and fetches each row by the primary key the index stored. That means the index has to store the primary key. The vsql-vector-pk extension does, and its HNSW index answers these queries end to end on this release. It is a variant of vsql-vector whose index stores the primary key alongside each entry.

The remaining work on custom indexing is index maintenance. On a table with a custom index, the server currently refuses an UPDATE that changes the primary key or a column in any index, ordinary indexes included, and an ALTER TABLE that changes the primary key. It also refuses a DELETE (DELETE Another Day) against an index that identifies its entries by its own handles rather than by row key, because the server can't yet tell the index which entry to drop. The HNSW indexes in vsql-vector and vsql-vector-pk both work this way, so neither extension supports DELETE yet. Other ALTER TABLE operations, including a full table rebuild, keep the index.

At Percona Live in Amsterdam on September 9 we demoed vector search in MySQL on a development build. An HNSW index defined entirely by the vsql-vector extension answered a nearest-neighbor query from the graph instead of scanning the table, and we measured recall against the same rows. The metadata, the storage, the DDL, and the server side of that scan all ship in this release.

CLONE now checks for your extensions

CLONE INSTANCE copies a running server's data into another server. We extended its compatibility checks to cover extensions, alongside the existing checks for plugins, character sets, and configuration. Extension bundles (.veb files) live outside the data directory, so a clone doesn't copy them, but the system table that records which extensions are installed is copied.

We check the donor's extension list during the handshake, before any data moves. Every extension installed on the donor has to be present in the recipient's veb_dir with a matching version and a matching .veb checksum, which is the same bar the server applies to its own extensions at startup.

This check matters beyond clone. An extension can define a column type whose stored bytes mean nothing without the extension that wrote them. The extension is part of the server's state, so every operational path that copies, moves, or upgrades a server has to account for it. Clone is the first path we covered. Backup and restore, upgrades across codebases, and dependencies between extensions come next.

Extension functions can compute their own return type

The vsql_tvector example extension defines TVECTOR(N), a fixed-width vector type. A function that joins a TVECTOR(3) and a TVECTOR(4) end to end should return a TVECTOR(7). The server's built-in type rules can't produce that answer: they copy one argument's parameters onto the return value, and they reject a call whose arguments disagree.

An extension can now register a function with bind_and_check_types, and the server calls that hook while it resolves the statement:

make_func<&tvector_concat>("tvector_concat")
    .bind_and_check_types<&tvector_concat_bind>()

The hook receives each argument's kind, the type parameters already resolved for it, and the literal text when an argument is a string constant. It returns the type parameters for the result, optionally new parameters for each argument, or an error. The hook replaces the built-in rules for that function, so a call the server would have rejected is allowed and the return type is the one the extension computed.

vsql_tvector gains 7 functions that use it, covering a computed return dimension, 2 that fix the dimension and inherit the element type, 1 that inherits the dimension and fixes the element type, 2 reductions to a scalar, and an accessor. You can use these as examples when building your own extension.

Per-session extension settings

An extension can declare per-session variables through the new vsql::session_var capability. Every connection holds its own value, SET SESSION changes it, and the value the extension declared is the global default. INT and STR are supported. Extension system variables, which hold one value for the whole server, work as they did before.

The vector work is why this exists. HNSW search has a dial that trades recall against speed, and that choice belongs to the query, so vsql-vector declares the dial per session:

SET SESSION vsql_vector.ef_search = 100;

For vsql-vector, the default is 40, and a value outside 1 to 65536 is clamped to the nearest bound. Turn it up and the search reads more of the graph, returns better answers, and takes longer. One connection can run an analytics query at 400 while every other connection stays at the default. A function reads the calling session's value through int_var(). The capability is preview, so it needs vsql_allow_preview_extensions=ON.

System variables with a fixed list of values

For Your Enums Only. A vsql::sys_var variable can now be an enum: a fixed list of names, stored as an index. The server validates the name at SET time and rejects anything off the list, so the extension no longer has to check a free-form string after the fact.

Statement events identify the client

The Spy Who Logged Me. The vsql::preview::statement_event capability now reports the client process id, the connector library, and the application name, each NULL when the client didn't send it. If you're trying to be The Man with the Golden Query and writing per-query telemetry, that's what attributes a slow query to the service that ran it.

Breaking changes

DDL that the server used to accept can now be rejected. The server validates type parameters before it hands them to an extension, and for the TYPE, TYPE(N), and TYPE('k=v,...') forms it rejects a duplicate parameter name, a nameless parameter, and a parameter with no value. After the extension's resolve_params callback returns, the server also rejects a rewrite that dropped a parameter. Adding parameters and changing values is still legal. Check your DDL and your extensions against these rules before you upgrade.

The preview authentication capability changed too. auto_grant is gone, replaced by roles_mode, which tells the server whether to activate, grant, or sync the roles an identity provider supplies. An authentication extension built against the 0.0.6 preview needs a source change to compile. vsql-oauth2 already has, and its own vsql_oauth2.auto_grant setting still exists.

Strengthening testing

Tomorrow Never Deadlocks. Every night we run the unit tests, the full VillageSQL test suite, and every bundled extension under both Valgrind memcheck and ThreadSanitizer, so the server and the extensions you get with a release have been through both. We also used gcov to find code the suites never reached and wrote tests for it, and moving our CI runners to Ubuntu 26.04 turned up undefined behavior in our type-parameter parsing along with an upstream bug in the Group Replication plugin.

New extensions

Three extensions joined this cycle:

  • vsql-duckdb (From Amsterdam with Select): query Parquet, CSV, and JSON files from an ordinary MySQL connection, without exporting the data first or standing up a second query engine. It embeds DuckDB in the server process and reads the files where they already live, in an s3:// bucket or behind an https:// URL. Reading files off the server's own disk works too, once you turn on vsql_duckdb.allow_local_files. It's a port of the PostgreSQL extension pg_duckdb, narrowed to what the extension framework supports today. It uses preview capabilities, so it needs vsql_allow_preview_extensions=ON, and you build it from source. It also needs a server newer than 0.0.6, so this is the first release it runs on.
  • vsql-vector (Dr. No-Scan): the SVECTOR vector type, stored as packed floats in InnoDB pages through the column storage interface, the distance functions to go with it, and the HNSW index type the demo ran on. Its index does not store a row reference, so on this release the server builds and keeps it current on INSERT but cannot answer a query through it. It needs vsql_allow_preview_extensions=ON, and you build it from source.
  • vsql-vector-pk (The Vector Who Loved Me): the same SVECTOR type and HNSW index, with the primary key stored alongside each entry. That is what the nearest-neighbor scan described above needs, so this is the variant that answers queries on this release. A server runs either vsql-vector or vsql-vector-pk, never both. It needs vsql_allow_preview_extensions=ON, and you build it from source.

What's ahead

The next release is more hardening and production readiness. We're testing VillageSQL against popular ORMs, the JDBC and ODBC connectors, the Windows client, and backup and restore. We're building performance tests for a server running extensions, so that when someone asks what an extension costs in terms of performance we have a number instead of an opinion. Custom indexes carry on in parallel, on the query side and the write side, until an extension's index runs end to end on a released server without the preview flag.

Special thanks

The native RPM and DEB packaging in this release came from Evgeniy Patlan at Percona. The packages take over an existing MySQL install (On Her Majesty's Secret Server) rather than sitting beside one, sharing its binary names, data directory, systemd unit, and port, and the build dependencies resolve on EL8, EL9, EL10, Ubuntu 24.04, Ubuntu 26.04, Debian 12, and Debian 13. No workflow publishes them yet, so the release artifacts are unchanged for now, but the sources and the build scripts are in the tree.

There's more in this release than fits here. The release notes carry the per-PR detail. Our roadmap is published on GitHub, and we'd like to hear what you're building and what we should work on next, either on GitHub or on Discord. If you want to support VillageSQL, please star the repo.

]]> Wed, 14 Oct 2026 13:00:00 +0000 VillageSQL MySQL VillageSQL Open Source PHP và MySQL: 7 câu hỏi dev mới hay hỏi nhất https://dbahire.com/php-va-mysql-7-cau-hoi-dev-moi-hay-hoi-nhat https://dbahire.com/php-va-mysql-7-cau-hoi-dev-moi-hay-hoi-nhat Khi bắt đầu làm một website có tính năng động, gần như lúc nào bạn cũng sẽ phải đụng đến PHP và MySQL. Đây là phần lập trình xử lý logic và phần lưu trữ dữ liệu. Bài viết này tổng hợp top 7 câu hỏi thường gặp về PHP và MySQL mà người mới hay hỏi nhất. PHP là phần xử lý logic chạy trên máy chủ. MySQL là nơi lưu trữ dữ liệu dưới dạng các bảng có liên kết với nhau. Chức năng đăng nhập, giỏ hàng, danh sách bài viết hay sản phẩm phần lớn là kết quả của việc hai thành phần này phối hợp. Chúng tôi nhận thấy người mới thường hỏi gần như cùng một nhóm câu hỏi. Dưới đây là phần giải đáp cụ thể. Nếu bạn đang tự học, làm đồ án, hoặc là chủ shop nhỏ đang tự dựng website, phần này sẽ giúp bạn có nền tảng vững hơn. PHP và MySQL khác nhau thế nào, có cần học cả hai không? PHP là ngôn ngữ lập trình, dùng để viết các quyết định trong chương trình. Ví dụ: kiểm tra mật khẩu có đúng hay không, tính tổng tiền giỏ hàng, hay quyết định hiển thị nội dung nào cho người truy cập. MySQL là hệ quản trị cơ sở dữ liệu, chuyên giữ dữ liệu và trả về đúng thứ bạn yêu cầu. Nếu chỉ làm trang giới thiệu tĩnh, ít thay đổi, bạn chưa cần MySQL. Khi có người đăng nhập hoặc bài viết cần thêm, sửa, xóa, bạn cần chỗ lưu dữ liệu đó. Với phần lớn dự án, học cả hai là cần thiết. Nên chọn phiên bản PHP nào cho dự án hiện tại? Tiêu chí quan trọng nhất là phiên bản đó còn được nhà phát triển vá lỗi bảo mật hay không. Khi phiên bản ngừng hỗ trợ, lỗ hổng mới sẽ không còn bản sửa chính thức. Tiêu chí thứ hai là thư viện bạn định dùng có tương thích với phiên bản đó không. Danh sách phiên bản được hỗ trợ thay đổi theo thời gian, nên hãy kiểm tra trên trang chính thức của PHP trước khi quyết định. Đừng chỉ tin một bài viết cũ, kể cả bài này chỉ là điểm khởi đầu để bạn đối chiếu. Đừng quên kiểm tra hosting. Một số gói giá rẻ chỉ hỗ trợ phiên bản khá thấp. Kiểm tra trước khi cài đặt sẽ tránh phiền phức khi nâng cấp về sau. Kết nối PHP với MySQL sao cho an toàn? Có hai cách kết nối phổ biến: PDO và MySQLi. PDO làm việc được với nhiều loại cơ sở dữ liệu, còn MySQLi chỉ dành cho MySQL. Với người mới, chúng tôi thường gợi ý PDO vì kiến thức này dễ mang sang dự án khác. Quan trọng hơn cả là không ghép trực tiếp dữ liệu người dùng nhập vào câu lệnh SQL. Hãy dùng câu lệnh chuẩn bị sẵn (prepared statement) để tách lệnh khỏi dữ liệu. Thông tin kết nối như tên đăng nhập và mật khẩu cơ sở dữ liệu nên đặt trong tệp cấu hình nằm ngoài thư mục công khai. Vì sao SQL Injection nguy hiểm và phòng tránh ra sao? SQL Injection là kiểu tấn công mà kẻ xấu nhập mã SQL vào các ô nhập liệu như ô tìm kiếm hoặc form đăng nhập. Nếu chương trình ghép thẳng chuỗi đó vào truy vấn mà không kiểm tra, dữ liệu có thể bị đọc, sửa hoặc xóa. Cách xử lý cốt lõi là dùng prepared statement và kiểm tra kiểu dữ liệu đầu vào. Ô nhập số lượng chỉ nhận số, còn ô email cần kiểm tra đúng định dạng. Bên cạnh đó, tài khoản kết nối cơ sở dữ liệu chỉ nên có những quyền thật sự cần thiết. Thiết kế bảng MySQL cho shop nhỏ thế nào cho gọn? Với shop nhỏ, bạn nên tách riêng bảng người dùng, bảng sản phẩm và bảng đơn hàng. Các bảng liên kết với nhau qua khóa ngoại, ví dụ mỗi đơn hàng trỏ về đúng một người mua. Cách này giúp tránh lặp thông tin và giúp việc sửa đổi chỉ cần làm ở một chỗ. Đừng chuẩn hóa quá mức khi quy mô còn nhỏ. Thiết kế quá phức tạp sẽ làm việc viết truy vấn khó hơn. Chuẩn hóa vừa phải là đủ, và khi cửa hàng mở rộng bạn có thể điều chỉnh tiếp. Website chậm vì truy vấn MySQL, nên xử lý từ đâu? Điều đầu tiên là đừng đoán mò. Hãy bật slow query log để ghi lại các truy vấn chạy chậm, rồi xem xét từng câu. Thường thì thêm chỉ mục (index) cho các cột xuất hiện trong mệnh đề WHERE hoặc JOIN sẽ giúp cải thiện. Một lỗi phổ biến là dùng SELECT * khi chỉ cần vài cột. Chỉ chọn đúng cột cần thiết cũng góp phần cải thiện tốc độ. Hãy đo lại sau mỗi lần chỉnh sửa để biết hiệu quả thực tế, vì kết quả phụ thuộc vào lượng dữ liệu và cấu hình máy chủ. So sánh PDO, MySQLi và mysql_ cho người mới Khi tìm tài liệu cũ, bạn sẽ gặp cả ba cách này. Dưới đây là tóm tắt nhanh: mysql_: cách cũ, đã ngừng sử dụng và không hỗ trợ prepared statement. Không nên dùng cho dự án mới. MySQLi: hỗ trợ prepared statement, phù hợp khi dự án chỉ dùng MySQL. PDO: hỗ trợ prepared statement tốt và dùng được với nhiều hệ quản trị, thường là lựa chọn dễ mang theo kiến thức nhất cho người mới. Quan trọng hơn là chọn một cách, dùng đúng và giữ nhất quán trong toàn bộ dự án. Câu hỏi thường gặp PHP và MySQL có miễn phí cho dự án thương mại không? Cả hai đều là mã nguồn mở, bạn có thể dùng cho dự án kinh doanh. Tuy nhiên, công cụ đi kèm hoặc dịch vụ hosting có thể tính phí, nên hãy đọc kỹ giấy phép từng thành phần. Hosting giá rẻ có chạy tốt PHP và MySQL không? Đa số hosting phổ thông đều hỗ trợ PHP và MySQL. Cần để ý phiên bản PHP và giới hạn tài nguyên. Với website nhỏ thường đủ dùng, khi lượng truy cập tăng bạn sẽ cần nâng cấp gói. Nên học MySQL trước hay PHP trước? Người hoàn toàn mới nên học PHP căn bản trước để thấy dữ liệu được lấy ra và hiển thị như thế nào, từ đó hiểu MySQL nhanh hơn. Nếu chưa biết bắt đầu từ đâu, hãy chọn bài toán nhỏ như trang đăng nhập hoặc danh sách sản phẩm rồi tự làm thử. Để làm quen công cụ quản trị, bạn có thể đọc tổng quan về hệ quản trị cơ sở dữ liệu MySQL và MySQL Workbench là gì và vì sao nên cài đặt. Nếu cần đơn vị tư vấn để dựng website đúng chuẩn, bạn có thể tham khảo dotnetguru.org. The post PHP và MySQL: 7 câu hỏi dev mới hay hỏi nhất appeared first on DBAhire. Khi bắt đầu làm một website có tính năng động, gần như lúc nào bạn cũng sẽ phải đụng đến PHP và MySQL. Đây là phần lập trình xử lý logic và phần lưu trữ dữ liệu. Bài viết này tổng hợp top 7 câu hỏi thường gặp về PHP và MySQL mà người mới hay hỏi nhất.

PHP là phần xử lý logic chạy trên máy chủ. MySQL là nơi lưu trữ dữ liệu dưới dạng các bảng có liên kết với nhau. Chức năng đăng nhập, giỏ hàng, danh sách bài viết hay sản phẩm phần lớn là kết quả của việc hai thành phần này phối hợp.

Chúng tôi nhận thấy người mới thường hỏi gần như cùng một nhóm câu hỏi. Dưới đây là phần giải đáp cụ thể. Nếu bạn đang tự học, làm đồ án, hoặc là chủ shop nhỏ đang tự dựng website, phần này sẽ giúp bạn có nền tảng vững hơn.

PHP và MySQL khác nhau thế nào, có cần học cả hai không?

PHP là ngôn ngữ lập trình, dùng để viết các quyết định trong chương trình. Ví dụ: kiểm tra mật khẩu có đúng hay không, tính tổng tiền giỏ hàng, hay quyết định hiển thị nội dung nào cho người truy cập.

MySQL là hệ quản trị cơ sở dữ liệu, chuyên giữ dữ liệu và trả về đúng thứ bạn yêu cầu. Nếu chỉ làm trang giới thiệu tĩnh, ít thay đổi, bạn chưa cần MySQL. Khi có người đăng nhập hoặc bài viết cần thêm, sửa, xóa, bạn cần chỗ lưu dữ liệu đó. Với phần lớn dự án, học cả hai là cần thiết.

Nên chọn phiên bản PHP nào cho dự án hiện tại?

Tiêu chí quan trọng nhất là phiên bản đó còn được nhà phát triển vá lỗi bảo mật hay không. Khi phiên bản ngừng hỗ trợ, lỗ hổng mới sẽ không còn bản sửa chính thức. Tiêu chí thứ hai là thư viện bạn định dùng có tương thích với phiên bản đó không.

Danh sách phiên bản được hỗ trợ thay đổi theo thời gian, nên hãy kiểm tra trên trang chính thức của PHP trước khi quyết định. Đừng chỉ tin một bài viết cũ, kể cả bài này chỉ là điểm khởi đầu để bạn đối chiếu.

Đừng quên kiểm tra hosting. Một số gói giá rẻ chỉ hỗ trợ phiên bản khá thấp. Kiểm tra trước khi cài đặt sẽ tránh phiền phức khi nâng cấp về sau.

Kết nối PHP với MySQL sao cho an toàn?

Có hai cách kết nối phổ biến: PDO và MySQLi. PDO làm việc được với nhiều loại cơ sở dữ liệu, còn MySQLi chỉ dành cho MySQL. Với người mới, chúng tôi thường gợi ý PDO vì kiến thức này dễ mang sang dự án khác.

Quan trọng hơn cả là không ghép trực tiếp dữ liệu người dùng nhập vào câu lệnh SQL. Hãy dùng câu lệnh chuẩn bị sẵn (prepared statement) để tách lệnh khỏi dữ liệu. Thông tin kết nối như tên đăng nhập và mật khẩu cơ sở dữ liệu nên đặt trong tệp cấu hình nằm ngoài thư mục công khai.

Vì sao SQL Injection nguy hiểm và phòng tránh ra sao?

SQL Injection là kiểu tấn công mà kẻ xấu nhập mã SQL vào các ô nhập liệu như ô tìm kiếm hoặc form đăng nhập. Nếu chương trình ghép thẳng chuỗi đó vào truy vấn mà không kiểm tra, dữ liệu có thể bị đọc, sửa hoặc xóa.

Cách xử lý cốt lõi là dùng prepared statement và kiểm tra kiểu dữ liệu đầu vào. Ô nhập số lượng chỉ nhận số, còn ô email cần kiểm tra đúng định dạng. Bên cạnh đó, tài khoản kết nối cơ sở dữ liệu chỉ nên có những quyền thật sự cần thiết.

Thiết kế bảng MySQL cho shop nhỏ thế nào cho gọn?

Với shop nhỏ, bạn nên tách riêng bảng người dùng, bảng sản phẩm và bảng đơn hàng. Các bảng liên kết với nhau qua khóa ngoại, ví dụ mỗi đơn hàng trỏ về đúng một người mua. Cách này giúp tránh lặp thông tin và giúp việc sửa đổi chỉ cần làm ở một chỗ.

Đừng chuẩn hóa quá mức khi quy mô còn nhỏ. Thiết kế quá phức tạp sẽ làm việc viết truy vấn khó hơn. Chuẩn hóa vừa phải là đủ, và khi cửa hàng mở rộng bạn có thể điều chỉnh tiếp.

Website chậm vì truy vấn MySQL, nên xử lý từ đâu?

Điều đầu tiên là đừng đoán mò. Hãy bật slow query log để ghi lại các truy vấn chạy chậm, rồi xem xét từng câu. Thường thì thêm chỉ mục (index) cho các cột xuất hiện trong mệnh đề WHERE hoặc JOIN sẽ giúp cải thiện.

Một lỗi phổ biến là dùng SELECT * khi chỉ cần vài cột. Chỉ chọn đúng cột cần thiết cũng góp phần cải thiện tốc độ. Hãy đo lại sau mỗi lần chỉnh sửa để biết hiệu quả thực tế, vì kết quả phụ thuộc vào lượng dữ liệu và cấu hình máy chủ.

So sánh PDO, MySQLi và mysql_ cho người mới

Khi tìm tài liệu cũ, bạn sẽ gặp cả ba cách này. Dưới đây là tóm tắt nhanh:

  • mysql_: cách cũ, đã ngừng sử dụng và không hỗ trợ prepared statement. Không nên dùng cho dự án mới.
  • MySQLi: hỗ trợ prepared statement, phù hợp khi dự án chỉ dùng MySQL.
  • PDO: hỗ trợ prepared statement tốt và dùng được với nhiều hệ quản trị, thường là lựa chọn dễ mang theo kiến thức nhất cho người mới.

Quan trọng hơn là chọn một cách, dùng đúng và giữ nhất quán trong toàn bộ dự án.

Câu hỏi thường gặp

PHP và MySQL có miễn phí cho dự án thương mại không?

Cả hai đều là mã nguồn mở, bạn có thể dùng cho dự án kinh doanh. Tuy nhiên, công cụ đi kèm hoặc dịch vụ hosting có thể tính phí, nên hãy đọc kỹ giấy phép từng thành phần.

Hosting giá rẻ có chạy tốt PHP và MySQL không?

Đa số hosting phổ thông đều hỗ trợ PHP và MySQL. Cần để ý phiên bản PHP và giới hạn tài nguyên. Với website nhỏ thường đủ dùng, khi lượng truy cập tăng bạn sẽ cần nâng cấp gói.

Nên học MySQL trước hay PHP trước?

Người hoàn toàn mới nên học PHP căn bản trước để thấy dữ liệu được lấy ra và hiển thị như thế nào, từ đó hiểu MySQL nhanh hơn.

Nếu chưa biết bắt đầu từ đâu, hãy chọn bài toán nhỏ như trang đăng nhập hoặc danh sách sản phẩm rồi tự làm thử. Để làm quen công cụ quản trị, bạn có thể đọc tổng quan về hệ quản trị cơ sở dữ liệu MySQL và MySQL Workbench là gì và vì sao nên cài đặt. Nếu cần đơn vị tư vấn để dựng website đúng chuẩn, bạn có thể tham khảo dotnetguru.org.

The post PHP và MySQL: 7 câu hỏi dev mới hay hỏi nhất appeared first on DBAhire.

]]> Wed, 14 Oct 2026 09:20:32 +0000 Jaime Crespo How fast is vector search inside MySQL? https://askdba.net/?p=3009 https://askdba.net/how-fast-is-vector-search-inside-mysql/ Benchmarks · HNSW vector search MyVector adds approximate nearest-neighbor search to stock MySQL — no separate vector database. We put it on identical hardware across three MySQL versions, added MariaDB’s native vector type as a reference, and measured build time, throughput, latency, and recall on the same data. askdba/myvector · live dashboard at bench.myvector.online 548peak ANN queries/sec (MySQL 9.7), ~1.8 ms median 0.95best recall@10 (MariaDB 12.3) once tuned 4×engines, one 50k×128 workload, identical 4 vCPU / 24 GB slices Why benchmark this at all Vector search usually means standing up a dedicated database next to your primary store, then keeping the two in sync. MyVector takes the other path: it lives inside MySQL as UDFs and stored procedures, builds an HNSW index over a column of vectors, and answers nearest-neighbor queries in SQL. The obvious question is whether that convenience costs you speed or accuracy. To answer it honestly we needed a level field. Every engine runs in its own container on one Ampere A1 host, each pinned to the same 4 vCPU / 24 GB slice, loading the same 50,000 synthetic 128-dimensional vectors (seeded, so every engine indexes identical data), and builds an HNSW index with the same parameters (M=16, ef_construction=200, L2 distance). Three of the four are MySQL running MyVector — versions 8.4, 9.7 and 26.7. The fourth is MariaDB 12.3, which ships its own native VECTOR type, included as a cross-engine reference point. What each number means Five measurements, each isolating one thing: Index build — wall-clock seconds to build the HNSW graph over all 50k rows. Insert throughput — rows/sec written into a live, indexed table. ANN QPS &amp; latency — approximate nearest-neighbor queries per second through the index, and their median response time. Recall@10 — of the true 10 nearest neighbors (found by exact brute-force search), how many the approximate search actually returns, averaged over held-out queries that were never inserted. 1.0 is perfect; 0.82 means it finds ~8 of the real 10. ef_search sweep — the accuracy/throughput trade-off: searching more of the graph raises recall but lowers QPS. Results The headline is throughput. On identical A1 hardware, all three MySQL versions clear ~530–550 approximate queries per second at ~1.8 ms median — roughly 500× faster than the exact brute-force scan (~1 QPS) they fall back to without an index. 8.4, 9.7 and 26.7 land within a few percent of each other, so moving to a newer MySQL costs nothing here. Approximate nearest-neighbor throughput at the default operating point (ef_search=200). Higher is better. All four engines on identical Ampere A1 hardware. Recall@10 at the same operating point. The three MyVector builds land at 0.82; MariaDB 12.3 reaches 0.948 — it trades some speed for accuracy in how it walks the graph. The real picture: accuracy vs throughput A single QPS number hides the trade-off. The ef_search knob lets each query decide how hard to look: low values are fast but miss neighbors, high values are accurate but slow. Plotting recall against QPS across that sweep is the honest comparison — up and to the right is better. Each point is one ef_search setting (10 → 200). MariaDB’s curve sits higher on recall at a given speed; the MySQL/MyVector curves reach higher raw QPS. All share the same synthetic ground truth. EngineBuild (s)Insert QPSANN QPSANN p50 (ms)Recall@10 Exact brute-force KNN measured ~1 QPS (~700 ms) on every engine — the baseline the index is beating. The MariaDB surprise The first MariaDB run looked grim: 29 ANN QPS, a 583-second build, recall aside. Taken at face value it was 15× behind MySQL. That gap was too large to be a real engine difference, so we dug in. The cause was one default. MariaDB caches its HNSW graph in memory up to mhnsw_max_cache_size, which ships at 16 MB — but the index for this dataset is ~27 MB. The graph didn’t fit, so every search and the whole online build re-read nodes from disk. MyVector sidesteps this by loading its entire index into memory, so the default pitted a disk-bound MariaDB against an in-memory MyVector. Sizing the cache above the index changed everything: MariaDB 12.3, same data and hardware, cache at its 16 MB default vs sized above the index. ANN throughput rose ~9×; build dropped from 583 s to 92 s; recall improved from 0.85 to 0.95. (Search alone, measured in isolation, was ~15× faster.) The lesson, not the leaderboard Properly configured, MariaDB’s native vector search is competitive with MyVector on speed and ahead on recall. The takeaway isn’t a winner — it’s that a single cache default can make one engine look an order of magnitude slower than it is. Benchmarks are only as fair as their configuration, which is why the setup here is in the open. Takeaways Native MySQL vector search is fast. ~530–550 QPS at single-digit-millisecond latency on 4 modest cores, ~500× over brute force — without a second database to run. Newer MySQL isn’t slower. 8.4, 9.7 and 26.7 land within a few percent of each other; there’s no regression moving forward. MariaDB 12.3 holds its own once its HNSW cache fits the index — higher recall, comparable throughput. Configuration is part of the result. The most interesting finding wasn’t a speed number; it was a default that quietly handicapped an engine 15×. Every number here is reproducible. The live, interactive dashboard is at bench.myvector.online, and the benchmark harness and raw results live in github.com/askdba/myvector under scripts/bench-dashboard. Workload: 50,000 × 128-d vectors, L2 distance, HNSW M=16 / ef_construction=200, held-out queries for recall, each engine pinned to 4 vCPU / 24 GB on Ampere. Bar chart showing MySQL vector search benchmark: 548 QPS for MySQL 9.7, 257 QPS for MariaDB 12.3

Benchmarks · HNSW vector search

MyVector adds approximate nearest-neighbor search to stock MySQL — no separate vector database. We put it on identical hardware across three MySQL versions, added MariaDB’s native vector type as a reference, and measured build time, throughput, latency, and recall on the same data.

askdba/myvector · live dashboard at bench.myvector.online

548
peak ANN queries/sec (MySQL 9.7), ~1.8 ms median
0.95
best recall@10 (MariaDB 12.3) once tuned
4×
engines, one 50k×128 workload, identical 4 vCPU / 24 GB slices

Why benchmark this at all

Vector search usually means standing up a dedicated database next to your primary store, then keeping the two in sync. MyVector takes the other path: it lives inside MySQL as UDFs and stored procedures, builds an HNSW index over a column of vectors, and answers nearest-neighbor queries in SQL. The obvious question is whether that convenience costs you speed or accuracy.

To answer it honestly we needed a level field. Every engine runs in its own container on one Ampere A1 host, each pinned to the same 4 vCPU / 24 GB slice, loading the same 50,000 synthetic 128-dimensional vectors (seeded, so every engine indexes identical data), and builds an HNSW index with the same parameters (M=16, ef_construction=200, L2 distance). Three of the four are MySQL running MyVector — versions 8.4, 9.7 and 26.7. The fourth is MariaDB 12.3, which ships its own native VECTOR type, included as a cross-engine reference point.

What each number means

Five measurements, each isolating one thing:

  • Index build — wall-clock seconds to build the HNSW graph over all 50k rows.
  • Insert throughput — rows/sec written into a live, indexed table.
  • ANN QPS & latency — approximate nearest-neighbor queries per second through the index, and their median response time.
  • Recall@10 — of the true 10 nearest neighbors (found by exact brute-force search), how many the approximate search actually returns, averaged over held-out queries that were never inserted. 1.0 is perfect; 0.82 means it finds ~8 of the real 10.
  • ef_search sweep — the accuracy/throughput trade-off: searching more of the graph raises recall but lowers QPS.

Results

The headline is throughput. On identical A1 hardware, all three MySQL versions clear ~530–550 approximate queries per second at ~1.8 ms median — roughly 500× faster than the exact brute-force scan (~1 QPS) they fall back to without an index. 8.4, 9.7 and 26.7 land within a few percent of each other, so moving to a newer MySQL costs nothing here.

Approximate nearest-neighbor throughput at the default operating point (ef_search=200). Higher is better. All four engines on identical Ampere A1 hardware.
Recall@10 at the same operating point. The three MyVector builds land at 0.82; MariaDB 12.3 reaches 0.948 — it trades some speed for accuracy in how it walks the graph.

The real picture: accuracy vs throughput

A single QPS number hides the trade-off. The ef_search knob lets each query decide how hard to look: low values are fast but miss neighbors, high values are accurate but slow. Plotting recall against QPS across that sweep is the honest comparison — up and to the right is better.

Each point is one ef_search setting (10 → 200). MariaDB’s curve sits higher on recall at a given speed; the MySQL/MyVector curves reach higher raw QPS. All share the same synthetic ground truth.
EngineBuild (s)Insert QPSANN QPSANN p50 (ms)Recall@10

Exact brute-force KNN measured ~1 QPS (~700 ms) on every engine — the baseline the index is beating.

The MariaDB surprise

The first MariaDB run looked grim: 29 ANN QPS, a 583-second build, recall aside. Taken at face value it was 15× behind MySQL. That gap was too large to be a real engine difference, so we dug in.

The cause was one default. MariaDB caches its HNSW graph in memory up to mhnsw_max_cache_size, which ships at 16 MB — but the index for this dataset is ~27 MB. The graph didn’t fit, so every search and the whole online build re-read nodes from disk. MyVector sidesteps this by loading its entire index into memory, so the default pitted a disk-bound MariaDB against an in-memory MyVector.

Sizing the cache above the index changed everything:

MariaDB 12.3, same data and hardware, cache at its 16 MB default vs sized above the index. ANN throughput rose ~9×; build dropped from 583 s to 92 s; recall improved from 0.85 to 0.95. (Search alone, measured in isolation, was ~15× faster.)

The lesson, not the leaderboard

Properly configured, MariaDB’s native vector search is competitive with MyVector on speed and ahead on recall. The takeaway isn’t a winner — it’s that a single cache default can make one engine look an order of magnitude slower than it is. Benchmarks are only as fair as their configuration, which is why the setup here is in the open.

Takeaways

  • Native MySQL vector search is fast. ~530–550 QPS at single-digit-millisecond latency on 4 modest cores, ~500× over brute force — without a second database to run.
  • Newer MySQL isn’t slower. 8.4, 9.7 and 26.7 land within a few percent of each other; there’s no regression moving forward.
  • MariaDB 12.3 holds its own once its HNSW cache fits the index — higher recall, comparable throughput.
  • Configuration is part of the result. The most interesting finding wasn’t a speed number; it was a default that quietly handicapped an engine 15×.

Every number here is reproducible. The live, interactive dashboard is at bench.myvector.online, and the benchmark harness and raw results live in github.com/askdba/myvector under scripts/bench-dashboard. Workload: 50,000 × 128-d vectors, L2 distance, HNSW M=16 / ef_construction=200, held-out queries for recall, each engine pinned to 4 vCPU / 24 GB on Ampere.

]]> Wed, 14 Oct 2026 05:20:14 +0000 Alkin Tezuysal mysql Technical mariadb myvector search vector MySQL Online DDL "Duplicate Key" Optimization https://songlibing.github.io/posts/mysql-ddl-duplicate-key-problem-en/ https://songlibing.github.io/posts/mysql-ddl-duplicate-key-problem-en/ When you rebuild a table with MySQL’s Online DDL, the operation can fail partway through with a Duplicate Entry error: 1 2 mysql&gt; alter table tt add c3 int, algorithm=inplace; ERROR 1062 (23000): Duplicate entry '1' for key 'tt.uk_c2' This is a long-standing, well-known problem — it has existed ever since Online DDL was introduced in MySQL 5.6, and it appears across several bug reports: BUG#76895 — Adding new column OR Drop column causes duplicate PK error BUG#77572 — The bogus duplicate key error in online ddl with incorrect key name BUG#98600 — Optimize table fails with duplicate entry on UNIQUE KEY BUG#104626 — Remove failure of Online ALTER because concurrent Duplicate entry BUG#76895 was the first to report it. It called the problem a Duplicate PRIMARY KEY error, but that label was itself a bug: a separate 5.6 issue, BUG#77572, printed PRIMARY KEY in the message when it should have said UNIQUE KEY. Once BUG#77572 was fixed (in 5.6.28 and 5.7.10), the message correctly read Duplicate UNIQUE KEY — and BUG#98600 was filed later for what was really the same underlying problem. MySQL closed BUG#76895 as Not a Bug, treating the error as an unavoidable side effect of how Online DDL works. But “not a bug” is of little consolation to a DBA: a failed DDL derails a carefully planned change and often pushes it to the next maintenance window. That inconvenience is what eventually produced BUG#104626 — this time filed as a feature request asking for a real fix. So what actually causes this error, why does MySQL consider it working as intended, and can you do anything about it? Answering all three means looking at how Online DDL works internally. How Online DDL Works To keep a table writable while it’s being altered, Online DDL splits the job in two: a full copy of the rows that already exist, and an incremental replay of everything that changes while that copy runs. It copies the existing rows into the new table (when you’re adding an index there’s no new table — the rows go straight into the new index). The copy takes time, and DML keeps running throughout, so every change made along the way is captured to be applied afterward. Two things make this work: A precise cut-off in time, so the full copy takes only the rows that existed before it and nothing after. An incremental log that captures every change made after the cut-off, so replaying it brings the new table fully up to date. Distinguishing Full Data from Incremental Changes The full data lives in the clustered B+Tree, and Online DDL reads it by scanning that tree. Since DML runs during the scan, the tree also holds changes made after the cut-off, so Online DDL needs a way to separate the original rows from those later changes. It does this with InnoDB’s MVCC (multi-version concurrency control). MVCC lets a transaction read from a consistent snapshot of the data — one that isn’t affected by changes other transactions make while it runs. The mechanism behind it is the read view: when one is created, it records the IDs of all currently active transactions, then uses that list to decide which version of each row the transaction may see. If a row is modified and committed after the read view was created, the view can still reach the earlier version through the undo log, even once the new version is sitting in the clustered index. That is exactly what Online DDL relies on to set the boundary of the full copy: it opens a read view at one precise moment, and the copy reads only what that view can see. Whatever concurrent DML changes afterward — no matter when it commits — stays invisible to the copy. The Three Phases of Online DDL InnoDB runs an Online DDL in three phases, each holding a different level of Metadata Lock (MDL) to control precisely what concurrent DML is allowed to do. Prepare Prepare runs under MDL_EXCLUSIVE, which blocks every read and write on the table. That brief “writes-stopped” window is the cut-off between full and incremental data. During it, InnoDB: Creates a consistent read view via trx_assign_read_view(). Sets up the row log (the incremental log) on the target index and marks the index ONLINE_INDEX_CREATION. From then on, each insert, update, or delete on that index’s B+tree checks the flag to decide whether it also needs to go into the row log. Both steps happen under MDL_EXCLUSIVE, which guarantees two things: The read view and the row log line up perfectly, so together they cover the existing rows and every later change with no gap. It shuts out any in-flight transaction. Without the exclusive lock, a transaction could have modified the table but not yet committed at the moment of the cut-off. The row log isn’t installed yet, so its change goes unlogged; and because it’s still uncommitted, the read view marks it active and the full copy can’t see it either. The change would then be in neither the copy nor the log, and would be lost. MDL_EXCLUSIVE guarantees no such transaction is running, so the read view and row log meet at a clean boundary: everything committed before it belongs to the full copy; everything after it is captured by the row log. Once Prepare finishes, the lock is downgraded from MDL_EXCLUSIVE to the shared MDL_SHARED_UPGRADABLE. Execute During Execute, Online DDL holds MDL_SHARED_UPGRADABLE, so concurrent DML runs normally. This is where the full copy happens, along with the first pass of incremental replay — and it is by far the longest phase, the one throughout which the table stays writable. Every DML on the table is recorded in the row log. Commit For Commit, the DDL upgrades the lock from MDL_SHARED_UPGRADABLE back to MDL_EXCLUSIVE, blocking all reads and writes again. Under that lock it replays whatever row log is left and commits the DDL transaction. Because Execute already replayed most of the log, there is little left to do here, so the exclusive lock is held only briefly. The exclusive lock in this phase also guarantees two things: Every DML from the Execute phase that touched the table has committed. No new DML can start, so no new row log is produced. In short, Online DDL is not fully online — it blocks DML during Prepare and Commit. And because a metadata lock is held for the life of a transaction, a long-running transaction that is in progress when the DDL tries to acquire MDL_EXCLUSIVE will block it — and while the DDL waits, it blocks other DML too, leaving the table effectively unavailable. The Full Copy The full copy runs during Execute, under MDL_SHARED_UPGRADABLE, with concurrent DML allowed. It uses row_merge_read_clustered_index() to scan the old table’s clustered index in ascending PRIMARY KEY order, checking each row’s visibility with MVCC as it goes. Only rows visible to the read view are copied into the new table; delete-marked rows are skipped. Thanks to MVCC, even when concurrent DML rewrites a row mid-scan, the copy still sees the version as of the read view — reconstructed from the undo log if necessary. What it copies is always one consistent snapshot, unaffected by concurrent activity. Recording the Incremental Log The row log records page-level changes to a B+tree, and it behaves differently in the two kinds of DDL: Rebuilding the table: it logs only changes to the clustered B+tree — INSERT, UPDATE, and DELETE — and on replay each entry is applied to every index. Adding an index: it logs only changes to the new index’s B+tree — INSERT and DELETE, never UPDATE. Since the index is still being built, these operations are only written to the row log and never actually applied to the B+tree — the opposite of a table rebuild, where every existing index on the old table is maintained in real time. One detail here is worth paying attention to: when an entry is logged. Because the row log tracks B+tree changes, an entry is written only after the B+tree operation succeeds. As the figure shows, for a table rebuild the entry is logged after the row is inserted into the clustered B+tree; for an added index it is logged at the point the row would go into the secondary index's B+tree — except that, for an added index, nothing is actually inserted, and only the log entry is written. What Happens on a DML Rollback A DML operates on the clustered B+tree first, then the secondary indexes. So what happens if the secondary-index step fails? The row-log entry remains — it isn’t cleaned up. It records one B+tree operation, and that operation really did happen. The statement then rolls back. Rolling back replays the change in reverse against the B+tree (driven by the undo log), and that reverse operation is logged too. So a failed INSERT leaves two row-log entries — a ROW_T_INSERT followed by a ROW_T_DELETE (below). Applied in order, the two cancel out, exactly as a rollback should. A failed statement is just one way to reach this state; every rollback works the same way, including an explicit ROLLBACK from the user. Replaying the Incremental Log The row log is replayed twice: The first pass runs in Execute, right after the full copy. The lock is still MDL_SHARED_UPGRADABLE, so DML keeps running and keeps adding to the log. The purpose of this pass is to replay as much of the log as possible now, so the exclusive lock in Commit is held for as short a time as possible. The second pass runs in Commit, under MDL_EXCLUSIVE, with no new log being produced. It handles the small remainder, and once it is done the new table matches the old one exactly. Both passes read the log in write order, one entry at a time. For a table rebuild, each entry is applied to every index on the new table; for an added index, only to the index being built. What Actually Causes the Duplicate Entry Recall the rollback case: even when an INSERT encounters a Duplicate Entry on a unique index, it still leaves two row-log entries behind (below). Replaying the first entry, ROW_T_INSERT, does exactly what the original INSERT did: insert a row into the clustered B+tree, then add an entry to each index. When it reaches the unique index, it encounters the same Duplicate Entry — this time during replay, which is what fails the whole DDL. Reproducing It 1 2 3 4 5 CREATE TABLE t1 ( c1 INT AUTO_INCREMENT PRIMARY KEY, c2 INT, UNIQUE KEY (c2) ) ENGINE=InnoDB; Run these statements in three separate sessions: 1 2 3 4 5 6 7 8 9 # Session 1 BEGIN; INSERT INTO t1 VALUES(NULL, 1); # Session 2 ALTER TABLE t1 ENGINE = InnoDB; # Session 3 INSERT INTO t1 VALUES(NULL, 1); Now commit Session 1, and both Session 2’s ALTER and Session 3’s INSERT fail with Duplicate Entry: 1 2 3 4 5 6 7 # Session 2 mysql&gt; ALTER TABLE t1 ENGINE = InnoDB; ERROR 1062 (23000): Duplicate entry '1' for key 't1.c2' # Session 3 mysql&gt; INSERT INTO t1 VALUES(2, 1); ERROR 1062 (23000): Duplicate entry '1' for key 't1.c2' The example relies on the MDL behavior described above: Session 1’s INSERT takes an MDL_SHARED_WRITE lock on t1 and holds it until it commits. Session 2’s ALTER needs MDL_EXCLUSIVE for its prepare phase, so it is blocked by Session 1. Session 3’s INSERT wants MDL_SHARED_WRITE, but is now blocked by Session 2. You can observe this in performance_schema.metadata_locks — the first row below is Session 1’s lock. When Session 1 commits, Session 2 acquires MDL_EXCLUSIVE, finishes Prepare, and downgrades to MDL_SHARED_UPGRADABLE. That lock doesn’t conflict with MDL_SHARED_WRITE, so Session 3 runs — and since c2 = 1 already exists, it fails with a Duplicate key error. That failure leaves a row-log entry, and replaying it is what makes Session 2’s ALTER fail in turn. How to Avoid It Once the mechanism is clear, I would call this a bug — which is why we fixed it in AliSQL. On community MySQL, the only option is to prevent DML from encountering Duplicate Entry while a DDL is running. A common trigger is a table with an auto-increment primary key that has also gained a unique index. The application inserts a row, lets MySQL assign the auto-increment key, and — if the first INSERT is slow — retries the same statement from another session. That is perfectly sensible on its own: the unique index guarantees only one of the two can succeed. But a single failing INSERT is enough to make the DDL fail. So while a DDL is in progress, increasing the retry timeout is a practical way to avoid the problem. The Fix in AliSQL AliSQL takes a straightforward approach: when replay encounters a Duplicate Entry, ignore it. Real Duplicates vs. False Ones There is one caveat: not every Duplicate Entry is safe to ignore. Online DDL has a genuine real duplicate case: the DDL introduces a new uniqueness constraint (a new or changed primary key, or a new UNIQUE index) while the existing data already contains duplicates. That new constraint isn’t enforced during the DDL, so concurrent DML can keep adding duplicates — and here the DDL must fail. Ignoring it would silently break the guarantee the user asked for. But if the Duplicate Entry occurs on a unique index that already existed on the table, it is always a false duplicate and safe to skip. That is the common case, and the one AliSQL optimizes. Cleaning Up the Rollback Entry As we saw, a failed DML leaves two row-log entries. For an INSERT: 1 2 &lt;ROW_T_INSERT, pk1, ...&gt; &lt;ROW_T_DELETE, pk1&gt; On replay, the insert into the unique index fails and we ignore it — but the insert into the primary-key B+tree did succeed. When the second entry (ROW_T_DELETE) is replayed, deleting from the primary key is fine; deleting from the unique index is not, because nothing was ever inserted there, so the record can’t be found and the replay fails. So whenever we skip a Duplicate Entry on a pre-existing unique index, we have to record it and skip the matching operation when the rollback’s row log arrives later, as shown below: UPDATE is more complex. When an UPDATE changes a secondary-index column, the secondary index performs two operations: Delete the old entry. Insert the new one. The replay fails at step 2, so on the rollback we cannot simply skip the whole index — we skip step 1 but still perform step 2: With this handling, AliSQL eliminates the Duplicate Entry failures that Online DDL should never have raised in the first place. Conclusion Online DDL can fail with a Duplicate Entry, and because it is nondeterministic — it depends on the timing of concurrent DML — it is hard to rule out entirely, and each failure can disrupt a planned change. The failure happens in just a few steps: a concurrent DML encounters a Duplicate Entry, that error is carried through Online DDL’s row log, and replaying the log fails the DDL. AliSQL breaks this chain by ignoring Duplicate Entry errors that occur on an already-existing unique index, so Online DDL is no longer aborted by an error that did not originate from the DDL itself. When you rebuild a table with MySQL’s Online DDL, the operation can fail partway through with a Duplicate Entry error:

1
2

mysql> alter table tt add c3 int, algorithm=inplace;
ERROR 1062 (23000): Duplicate entry '1' for key 'tt.uk_c2'

This is a long-standing, well-known problem — it has existed ever since Online DDL was introduced in MySQL 5.6, and it appears across several bug reports:

  • BUG#76895 — Adding new column OR Drop column causes duplicate PK error
  • BUG#77572 — The bogus duplicate key error in online ddl with incorrect key name
  • BUG#98600 — Optimize table fails with duplicate entry on UNIQUE KEY
  • BUG#104626 — Remove failure of Online ALTER because concurrent Duplicate entry

BUG#76895 was the first to report it. It called the problem a Duplicate PRIMARY KEY error, but that label was itself a bug: a separate 5.6 issue, BUG#77572, printed PRIMARY KEY in the message when it should have said UNIQUE KEY. Once BUG#77572 was fixed (in 5.6.28 and 5.7.10), the message correctly read Duplicate UNIQUE KEY — and BUG#98600 was filed later for what was really the same underlying problem.

MySQL closed BUG#76895 as Not a Bug, treating the error as an unavoidable side effect of how Online DDL works. But “not a bug” is of little consolation to a DBA: a failed DDL derails a carefully planned change and often pushes it to the next maintenance window. That inconvenience is what eventually produced BUG#104626 — this time filed as a feature request asking for a real fix.

So what actually causes this error, why does MySQL consider it working as intended, and can you do anything about it? Answering all three means looking at how Online DDL works internally.

How Online DDL Works

To keep a table writable while it’s being altered, Online DDL splits the job in two: a full copy of the rows that already exist, and an incremental replay of everything that changes while that copy runs. It copies the existing rows into the new table (when you’re adding an index there’s no new table — the rows go straight into the new index). The copy takes time, and DML keeps running throughout, so every change made along the way is captured to be applied afterward. Two things make this work:

  • A precise cut-off in time, so the full copy takes only the rows that existed before it and nothing after.
  • An incremental log that captures every change made after the cut-off, so replaying it brings the new table fully up to date.

Distinguishing Full Data from Incremental Changes

The full data lives in the clustered B+Tree, and Online DDL reads it by scanning that tree. Since DML runs during the scan, the tree also holds changes made after the cut-off, so Online DDL needs a way to separate the original rows from those later changes. It does this with InnoDB’s MVCC (multi-version concurrency control).

MVCC lets a transaction read from a consistent snapshot of the data — one that isn’t affected by changes other transactions make while it runs. The mechanism behind it is the read view: when one is created, it records the IDs of all currently active transactions, then uses that list to decide which version of each row the transaction may see. If a row is modified and committed after the read view was created, the view can still reach the earlier version through the undo log, even once the new version is sitting in the clustered index.

That is exactly what Online DDL relies on to set the boundary of the full copy: it opens a read view at one precise moment, and the copy reads only what that view can see. Whatever concurrent DML changes afterward — no matter when it commits — stays invisible to the copy.

The Three Phases of Online DDL

InnoDB runs an Online DDL in three phases, each holding a different level of Metadata Lock (MDL) to control precisely what concurrent DML is allowed to do.

Prepare

Prepare runs under MDL_EXCLUSIVE, which blocks every read and write on the table. That brief “writes-stopped” window is the cut-off between full and incremental data. During it, InnoDB:

  1. Creates a consistent read view via trx_assign_read_view().
  2. Sets up the row log (the incremental log) on the target index and marks the index ONLINE_INDEX_CREATION. From then on, each insert, update, or delete on that index’s B+tree checks the flag to decide whether it also needs to go into the row log.

Both steps happen under MDL_EXCLUSIVE, which guarantees two things:

  1. The read view and the row log line up perfectly, so together they cover the existing rows and every later change with no gap.
  2. It shuts out any in-flight transaction. Without the exclusive lock, a transaction could have modified the table but not yet committed at the moment of the cut-off. The row log isn’t installed yet, so its change goes unlogged; and because it’s still uncommitted, the read view marks it active and the full copy can’t see it either. The change would then be in neither the copy nor the log, and would be lost. MDL_EXCLUSIVE guarantees no such transaction is running, so the read view and row log meet at a clean boundary: everything committed before it belongs to the full copy; everything after it is captured by the row log.

Once Prepare finishes, the lock is downgraded from MDL_EXCLUSIVE to the shared MDL_SHARED_UPGRADABLE.

Execute

During Execute, Online DDL holds MDL_SHARED_UPGRADABLE, so concurrent DML runs normally. This is where the full copy happens, along with the first pass of incremental replay — and it is by far the longest phase, the one throughout which the table stays writable. Every DML on the table is recorded in the row log.

Commit

For Commit, the DDL upgrades the lock from MDL_SHARED_UPGRADABLE back to MDL_EXCLUSIVE, blocking all reads and writes again. Under that lock it replays whatever row log is left and commits the DDL transaction. Because Execute already replayed most of the log, there is little left to do here, so the exclusive lock is held only briefly.

The exclusive lock in this phase also guarantees two things:

  • Every DML from the Execute phase that touched the table has committed.
  • No new DML can start, so no new row log is produced.

In short, Online DDL is not fully online — it blocks DML during Prepare and Commit. And because a metadata lock is held for the life of a transaction, a long-running transaction that is in progress when the DDL tries to acquire MDL_EXCLUSIVE will block it — and while the DDL waits, it blocks other DML too, leaving the table effectively unavailable.

The Full Copy

The full copy runs during Execute, under MDL_SHARED_UPGRADABLE, with concurrent DML allowed.

It uses row_merge_read_clustered_index() to scan the old table’s clustered index in ascending PRIMARY KEY order, checking each row’s visibility with MVCC as it goes. Only rows visible to the read view are copied into the new table; delete-marked rows are skipped.

Thanks to MVCC, even when concurrent DML rewrites a row mid-scan, the copy still sees the version as of the read view — reconstructed from the undo log if necessary. What it copies is always one consistent snapshot, unaffected by concurrent activity.

Recording the Incremental Log

The row log records page-level changes to a B+tree, and it behaves differently in the two kinds of DDL:

  • Rebuilding the table: it logs only changes to the clustered B+tree — INSERT, UPDATE, and DELETE — and on replay each entry is applied to every index.
  • Adding an index: it logs only changes to the new index’s B+tree — INSERT and DELETE, never UPDATE. Since the index is still being built, these operations are only written to the row log and never actually applied to the B+tree — the opposite of a table rebuild, where every existing index on the old table is maintained in real time.

One detail here is worth paying attention to: when an entry is logged. Because the row log tracks B+tree changes, an entry is written only after the B+tree operation succeeds. As the figure shows, for a table rebuild the entry is logged after the row is inserted into the clustered B+tree; for an added index it is logged at the point the row would go into the secondary index's B+tree — except that, for an added index, nothing is actually inserted, and only the log entry is written.

What Happens on a DML Rollback

A DML operates on the clustered B+tree first, then the secondary indexes. So what happens if the secondary-index step fails?

  • The row-log entry remains — it isn’t cleaned up. It records one B+tree operation, and that operation really did happen.
  • The statement then rolls back. Rolling back replays the change in reverse against the B+tree (driven by the undo log), and that reverse operation is logged too. So a failed INSERT leaves two row-log entries — a ROW_T_INSERT followed by a ROW_T_DELETE (below). Applied in order, the two cancel out, exactly as a rollback should.

A failed statement is just one way to reach this state; every rollback works the same way, including an explicit ROLLBACK from the user.

Replaying the Incremental Log

The row log is replayed twice:

  • The first pass runs in Execute, right after the full copy. The lock is still MDL_SHARED_UPGRADABLE, so DML keeps running and keeps adding to the log. The purpose of this pass is to replay as much of the log as possible now, so the exclusive lock in Commit is held for as short a time as possible.
  • The second pass runs in Commit, under MDL_EXCLUSIVE, with no new log being produced. It handles the small remainder, and once it is done the new table matches the old one exactly.

Both passes read the log in write order, one entry at a time. For a table rebuild, each entry is applied to every index on the new table; for an added index, only to the index being built.

What Actually Causes the Duplicate Entry

Recall the rollback case: even when an INSERT encounters a Duplicate Entry on a unique index, it still leaves two row-log entries behind (below).

Replaying the first entry, ROW_T_INSERT, does exactly what the original INSERT did: insert a row into the clustered B+tree, then add an entry to each index. When it reaches the unique index, it encounters the same Duplicate Entry — this time during replay, which is what fails the whole DDL.

Reproducing It

1
2
3
4
5

CREATE TABLE t1 (
  c1 INT AUTO_INCREMENT PRIMARY KEY,
  c2 INT,
  UNIQUE KEY (c2)
) ENGINE=InnoDB;

Run these statements in three separate sessions:

1
2
3
4
5
6
7
8
9

# Session 1
BEGIN;
INSERT INTO t1 VALUES(NULL, 1);

# Session 2
ALTER TABLE t1 ENGINE = InnoDB;

# Session 3
INSERT INTO t1 VALUES(NULL, 1);

Now commit Session 1, and both Session 2’s ALTER and Session 3’s INSERT fail with Duplicate Entry:

1
2
3
4
5
6
7

# Session 2
mysql> ALTER TABLE t1 ENGINE = InnoDB;
ERROR 1062 (23000): Duplicate entry '1' for key 't1.c2'

# Session 3
mysql> INSERT INTO t1 VALUES(2, 1);
ERROR 1062 (23000): Duplicate entry '1' for key 't1.c2'

The example relies on the MDL behavior described above:

  • Session 1’s INSERT takes an MDL_SHARED_WRITE lock on t1 and holds it until it commits.
  • Session 2’s ALTER needs MDL_EXCLUSIVE for its prepare phase, so it is blocked by Session 1.
  • Session 3’s INSERT wants MDL_SHARED_WRITE, but is now blocked by Session 2.

You can observe this in performance_schema.metadata_locks — the first row below is Session 1’s lock.

When Session 1 commits, Session 2 acquires MDL_EXCLUSIVE, finishes Prepare, and downgrades to MDL_SHARED_UPGRADABLE. That lock doesn’t conflict with MDL_SHARED_WRITE, so Session 3 runs — and since c2 = 1 already exists, it fails with a Duplicate key error. That failure leaves a row-log entry, and replaying it is what makes Session 2’s ALTER fail in turn.

How to Avoid It

Once the mechanism is clear, I would call this a bug — which is why we fixed it in AliSQL. On community MySQL, the only option is to prevent DML from encountering Duplicate Entry while a DDL is running.

A common trigger is a table with an auto-increment primary key that has also gained a unique index. The application inserts a row, lets MySQL assign the auto-increment key, and — if the first INSERT is slow — retries the same statement from another session. That is perfectly sensible on its own: the unique index guarantees only one of the two can succeed. But a single failing INSERT is enough to make the DDL fail. So while a DDL is in progress, increasing the retry timeout is a practical way to avoid the problem.

The Fix in AliSQL

AliSQL takes a straightforward approach: when replay encounters a Duplicate Entry, ignore it.

Real Duplicates vs. False Ones

There is one caveat: not every Duplicate Entry is safe to ignore.

Online DDL has a genuine real duplicate case: the DDL introduces a new uniqueness constraint (a new or changed primary key, or a new UNIQUE index) while the existing data already contains duplicates. That new constraint isn’t enforced during the DDL, so concurrent DML can keep adding duplicates — and here the DDL must fail. Ignoring it would silently break the guarantee the user asked for.

But if the Duplicate Entry occurs on a unique index that already existed on the table, it is always a false duplicate and safe to skip. That is the common case, and the one AliSQL optimizes.

Cleaning Up the Rollback Entry

As we saw, a failed DML leaves two row-log entries. For an INSERT:

1
2

<ROW_T_INSERT, pk1, ...>
<ROW_T_DELETE, pk1>

On replay, the insert into the unique index fails and we ignore it — but the insert into the primary-key B+tree did succeed. When the second entry (ROW_T_DELETE) is replayed, deleting from the primary key is fine; deleting from the unique index is not, because nothing was ever inserted there, so the record can’t be found and the replay fails.

So whenever we skip a Duplicate Entry on a pre-existing unique index, we have to record it and skip the matching operation when the rollback’s row log arrives later, as shown below:

UPDATE is more complex. When an UPDATE changes a secondary-index column, the secondary index performs two operations:

  1. Delete the old entry.
  2. Insert the new one.

The replay fails at step 2, so on the rollback we cannot simply skip the whole index — we skip step 1 but still perform step 2:

With this handling, AliSQL eliminates the Duplicate Entry failures that Online DDL should never have raised in the first place.

Conclusion

Online DDL can fail with a Duplicate Entry, and because it is nondeterministic — it depends on the timing of concurrent DML — it is hard to rule out entirely, and each failure can disrupt a planned change. The failure happens in just a few steps: a concurrent DML encounters a Duplicate Entry, that error is carried through Online DDL’s row log, and replaying the log fails the DDL. AliSQL breaks this chain by ignoring Duplicate Entry errors that occur on an already-existing unique index, so Online DDL is no longer aborted by an error that did not originate from the DDL itself.

]]> Tue, 13 Oct 2026 08:30:00 +0000 Libing Song MySQL DDL Online DDL InnoDB Thread Pool in Percona Server and MySQL (Part 2) https://www.percona.com/?p=54397 https://www.percona.com/blog/thread-pool-in-percona-server-and-mysql-part-2/ Thread Pool in Percona Server and MySQL (Part 2) The previous Part 1 focused on explaining the idea of thread pool, configuration options and analysis of thread pool efficiency in Percona Server for MySQL 9.7.1-1 against the base line, which is the same server without thread pooling. We recommend reading the post with Part 1 before proceeding with reading Part 2 in the current post. MySQL 26.7.0 with and without thread pool: For better understanding of Percona Server 9.7.1-1 and MySQL 26.7.0 / 9.7.2 thread pool performance against each other we will start with a brief look at MySQL on its own. The test runs did not find any significant differences in the performance of the thread pool between MySQL Server 26.7.0 and 9.7.2:  Graph 1 – Comparison of MySQL 26.7.0 and 9.7.2 [ INTERACTIVE GRAPH ][ TABLE ] As can be seen on the Graph 1 – the variation in performance between the versions is minimal. Configurations without thread pool go extremely close to each other, same as with the thread pool. Lines that correspond to similar configurations converge to the same values across all server buffer tiers (2G / 12G / 32G) and across thread pool parameter ranges (thread_pool_size=[10/20/40/80/120/160] and thread_pool_oversubscribe=[2/3/4]). NOTE: MySQL 26.7.0 is very slightly faster than 9.7.2 in the high connection threads count. Therefore, to avoid the crowded graphs the data and performance comparisons will be presented for MySQL Version 26.7.0 only. The results for MySQL 9.7.2 can be viewed on the interactive graph page on GitHub.  Graph 2 – MySQL with 2G buffer  [ INTERACTIVE GRAPH ][ TABLE ] The thread pool implementation behaves differently in MySQL Server 26.7.0. Although it does not bring a huge performance boost as it was in Percona Server, the buffer pool for a high number of client connection threads (640 / 1280 / 2560) helps to keep TPS performance better than without the thread pool. Especially with 2560 connections the pooled TPS is three times the number of non-pooled TPS: 371 and 122 respectively. The selection of the optimal value for thread_pool_size is important. The value itself (10) looks the same as it was with Percona Server. In the same way, higher values for thread_pool_size (20 / 40 / 80 / 120 / 160) do less TPS improvements. Unlike Percona Server the oversubscription variations do not impact TPS much when innodb_buffer_pool_size=2G. However, thread_pool_oversubscribe=2 does slightly better than 3 or 4.  Graph 3 – MySQL with 12G buffer  [ INTERACTIVE GRAPH ][ TABLE ] Increasing buffer pool (innodb_buffer_pool_size=12G) brings the overall TPS performance up, but thread pool makes less impact at the highest number of connections: 1273 with thread pool and 718 without (TPS is improved by 77%). With the fully buffered data set (innodb_buffer_pool_size=32G) MySQL thread pool configuration managed to achieve better TPS performance than non-pooled configuration.  Graph 4 – MySQL with 32G buffer  [ INTERACTIVE GRAPH ][ TABLE ] The pooled configuration yielded 18,975 TPS against 17,011 for non-pooled server configuration, which is 11.6% win for the thread pool. The p95 latency in MySQL had a very similar characteristic to Percona Server with thread pool reducing the latency from 759ms down to 153 ms. Comparison of Percona Server 9.7.1-1 for MySQL and MySQL Server 26.7.0: Finally, we arrived at the direct TPS comparison. Let’s start with innodb_buffer_pool_size=2G: For the comparison we will pick the best performing thread pool configurations for Percona Server and MySQL Server along with baseline (thread pool disabled) configurations. It was established before that thread_pool_size=10 works best for both servers (TP 10 on the graph), but the oversubscription 4 (os4 on the graph) works better for Percona Server while the optimal oversubscription for MySQL Server is 2 (os2 on the graph).  Graph 5 – Percona Server and MySQL with 2G buffer  [ INTERACTIVE GRAPH ][ TABLE ] MySQL Server starts well, winning with 40 connections, but we do not consider thread pool efficiency to be good with low number of connections. At 2560 connections mark the Percona Server implementation of the thread pool helps it to produce more TPS than MySQL server by a factor of 6.2 (2,304 against 371 TPS).  Graph 6 – Percona Server and MySQL with 12G buffer  [ INTERACTIVE GRAPH ][ TABLE ] Again, at a lower number of connections (40 and 80) MySQL Server goes faster than Percona Server. Moving to the high number of connections (2560) with innodb_buffer_pool_size=12G the Percona thread pool performs better than MySQL. On the next graph corresponding to innodb_buffer_pool_size=32G, the best performing configurations are different for Percona Server (thread_pool_size=160/thread_pool_oversubscribe=2) and MySQL Server (thread_pool_size=10/thread_pool_oversubscribe=2). It looks that in Percona Server increasing innodb_buffer_pool_size requires increasing thread_pool_size for the thread pool to work optimally.  Graph 7 – Percona Server and MySQL with 32G buffer  [ INTERACTIVE GRAPH ][ TABLE ] In the tier of fully buffered data (innodb_buffer_pool_size=32G) both servers performance was close, but MySQL Server was slightly faster. When it comes to the thread pool, MySQL is a clear winner with 19,054 TPS against 15,519 TPS from Percona Server (22% difference). With regard to p95 the Percona Server showed slightly better (lower) latency than MySQL server in 2G and 12G buffer tiers. However, with 32G the MySQL Server is ahead of Percona Server:  Graph 8 – p95 latency in Percona Server and MySQL with 32G buffer  [ INTERACTIVE GRAPH ][ TABLE ] With disabled thread pool the latencies for both Percona Server and MySQL land close to each other with MySQL being slightly better (shorter). When thread pool is enabled the p95 latency in MySQL is notably better (by ~25%) with 177ms against 235ms in Percona server. Conclusion Despite sharing the base server code, the thread pool implementations in Percona Server for MySQL and MySQL are very different. When the data set is too big to be fully buffered the Percona Server thread pool shows better performance compared to MySQL Server. However, MySQL Server and its thread pool wins with the configurations when data is fully buffered. Also, in the fully buffered scenario the p95 latency in MySQL Server is slightly shorter than in Percona Server. Another important observation is that the thread pool inside Percona Server for MySQL is very sensitive to the number of thread groups (thread_pool_size) and the number of active threads in the same group (thread_pool_oversubscribe). These options had the biggest impact on the performance and setting optimal values is important to reach the full potential of the thread pool. With MySQL Server change of these parameters has less dramatic effect. Also, the noticeable benefit of thread pooling in case of MySQL is visible only on high connection numbers such as 1280/2560 while Percona Server implementation does it much earlier with 80+ connections. The post Thread Pool in Percona Server and MySQL (Part 2) appeared first on Percona. Thread Pool in Percona Server and MySQL (Part 2)

The previous Part 1 focused on explaining the idea of thread pool, configuration options and analysis of thread pool efficiency in Percona Server for MySQL 9.7.1-1 against the base line, which is the same server without thread pooling.

We recommend reading the post with Part 1 before proceeding with reading Part 2 in the current post.

MySQL 26.7.0 with and without thread pool:

For better understanding of Percona Server 9.7.1-1 and MySQL 26.7.0 / 9.7.2 thread pool performance against each other we will start with a brief look at MySQL on its own.

The test runs did not find any significant differences in the performance of the thread pool between MySQL Server 26.7.0 and 9.7.2:

 Graph 1 – Comparison of MySQL 26.7.0 and 9.7.2 [ INTERACTIVE GRAPH ][ TABLE ]Graph 1 - Comparison of MySQL 26.7.0 and 9.7.2

As can be seen on the Graph 1 – the variation in performance between the versions is minimal. Configurations without thread pool go extremely close to each other, same as with the thread pool.

Lines that correspond to similar configurations converge to the same values across all server buffer tiers (2G / 12G / 32G) and across thread pool parameter ranges (thread_pool_size=[10/20/40/80/120/160] and thread_pool_oversubscribe=[2/3/4]).

NOTE: MySQL 26.7.0 is very slightly faster than 9.7.2 in the high connection threads count.

Therefore, to avoid the crowded graphs the data and performance comparisons will be presented for MySQL Version 26.7.0 only. The results for MySQL 9.7.2 can be viewed on the interactive graph page on GitHub.

 Graph 2 – MySQL with 2G buffer  [ INTERACTIVE GRAPH ][ TABLE ]Graph 2 - MySQL with 2G buffer

The thread pool implementation behaves differently in MySQL Server 26.7.0. Although it does not bring a huge performance boost as it was in Percona Server, the buffer pool for a high number of client connection threads (640 / 1280 / 2560) helps to keep TPS performance better than without the thread pool. Especially with 2560 connections the pooled TPS is three times the number of non-pooled TPS: 371 and 122 respectively.

The selection of the optimal value for thread_pool_size is important. The value itself (10) looks the same as it was with Percona Server. In the same way, higher values for thread_pool_size (20 / 40 / 80 / 120 / 160) do less TPS improvements.

Unlike Percona Server the oversubscription variations do not impact TPS much when innodb_buffer_pool_size=2G. However, thread_pool_oversubscribe=2 does slightly better than 3 or 4.

 Graph 3 – MySQL with 12G buffer  [ INTERACTIVE GRAPH ][ TABLE ]Graph 3 - MySQL with 12G buffer

Increasing buffer pool (innodb_buffer_pool_size=12G) brings the overall TPS performance up, but thread pool makes less impact at the highest number of connections: 1273 with thread pool and 718 without (TPS is improved by 77%).

With the fully buffered data set (innodb_buffer_pool_size=32G) MySQL thread pool configuration managed to achieve better TPS performance than non-pooled configuration.

 Graph 4 – MySQL with 32G buffer  [ INTERACTIVE GRAPH ][ TABLE ]Graph 4 - MySQL with 32G buffer

The pooled configuration yielded 18,975 TPS against 17,011 for non-pooled server configuration, which is 11.6% win for the thread pool.

The p95 latency in MySQL had a very similar characteristic to Percona Server with thread pool reducing the latency from 759ms down to 153 ms.

Comparison of Percona Server 9.7.1-1 for MySQL and MySQL Server 26.7.0:

Finally, we arrived at the direct TPS comparison.
Let’s start with innodb_buffer_pool_size=2G:

For the comparison we will pick the best performing thread pool configurations for Percona Server and MySQL Server along with baseline (thread pool disabled) configurations. It was established before that thread_pool_size=10 works best for both servers (TP 10 on the graph), but the oversubscription 4 (os4 on the graph) works better for Percona Server while the optimal oversubscription for MySQL Server is 2 (os2 on the graph).

 Graph 5 – Percona Server and MySQL with 2G buffer  [ INTERACTIVE GRAPH ][ TABLE ]Graph 5 - Percona Server and MySQL with 2G buffer

MySQL Server starts well, winning with 40 connections, but we do not consider thread pool efficiency to be good with low number of connections.

At 2560 connections mark the Percona Server implementation of the thread pool helps it to produce more TPS than MySQL server by a factor of 6.2 (2,304 against 371 TPS).

 Graph 6 – Percona Server and MySQL with 12G buffer  [ INTERACTIVE GRAPH ][ TABLE ]Graph 6 - Percona Server and MySQL with 12G buffer

Again, at a lower number of connections (40 and 80) MySQL Server goes faster than Percona Server.

Moving to the high number of connections (2560) with innodb_buffer_pool_size=12G the Percona thread pool performs better than MySQL.

On the next graph corresponding to innodb_buffer_pool_size=32G, the best performing configurations are different for Percona Server (thread_pool_size=160/thread_pool_oversubscribe=2) and MySQL Server (thread_pool_size=10/thread_pool_oversubscribe=2). It looks that in Percona Server increasing innodb_buffer_pool_size requires increasing thread_pool_size for the thread pool to work optimally.

 Graph 7 – Percona Server and MySQL with 32G buffer  [ INTERACTIVE GRAPH ][ TABLE ]Graph 7 - Percona Server and MySQL with 32G buffer

In the tier of fully buffered data (innodb_buffer_pool_size=32G) both servers performance was close, but MySQL Server was slightly faster. When it comes to the thread pool, MySQL is a clear winner with 19,054 TPS against 15,519 TPS from Percona Server (22% difference).

With regard to p95 the Percona Server showed slightly better (lower) latency than MySQL server in 2G and 12G buffer tiers. However, with 32G the MySQL Server is ahead of Percona Server:

 Graph 8 – p95 latency in Percona Server and MySQL with 32G buffer  [ INTERACTIVE GRAPH ][ TABLE ]
Graph 8 - p95 latency in Percona Server and MySQL with 32G buffer

With disabled thread pool the latencies for both Percona Server and MySQL land close to each other with MySQL being slightly better (shorter).

When thread pool is enabled the p95 latency in MySQL is notably better (by ~25%) with 177ms against 235ms in Percona server.

Conclusion

Despite sharing the base server code, the thread pool implementations in Percona Server for MySQL and MySQL are very different. When the data set is too big to be fully buffered the Percona Server thread pool shows better performance compared to MySQL Server. However, MySQL Server and its thread pool wins with the configurations when data is fully buffered. Also, in the fully buffered scenario the p95 latency in MySQL Server is slightly shorter than in Percona Server.

Another important observation is that the thread pool inside Percona Server for MySQL is very sensitive to the number of thread groups (thread_pool_size) and the number of active threads in the same group (thread_pool_oversubscribe). These options had the biggest impact on the performance and setting optimal values is important to reach the full potential of the thread pool.

With MySQL Server change of these parameters has less dramatic effect. Also, the noticeable benefit of thread pooling in case of MySQL is visible only on high connection numbers such as 1280/2560 while Percona Server implementation does it much earlier with 80+ connections.

The post Thread Pool in Percona Server and MySQL (Part 2) appeared first on Percona.

]]> Mon, 12 Oct 2026 11:14:07 +0000 MySQL Performance Blog Benchmarks MySQL benchmark Percona Server for MySQL Thread pool FOSS 2026 conference in Chernihiv: how it went https://percona.community/blog/2026/10/12/foss-2026-conference-in-chernihiv-how-it-went/ https://percona.community/blog/2026/10/12/foss-2026-conference-in-chernihiv-how-it-went/ On October 1-2, Chernihiv Polytechnic National University held the II International scientific and practical conference “Free/Open Source Software in Education, Science &amp; Business”. The conference was online and free for everyone. The idea is to bring together people from universities, from industry and from open source communities and talk about how free software is used in education, science and business. This year Percona was well represented. There were six speakers from Percona, and two more keynote speakers also worked at Percona in the past. Day 1: keynotes The first day started with welcome words from Volodymyr Bazylevych, Chair of the Organizing Committee, and Anatolii Prystupa, Vice-Rector for Research. After that there were five keynote talks. Yevhen Patlan (Percona) spoke about embedding DuckDB into MySQL. MySQL works very well for transactions, but analytical queries over hundreds of millions of rows can take a long time on InnoDB. Usually people solve this by copying data to a separate column store, so they need to support two systems and a pipeline between them. Yevhen showed another way: run DuckDB inside MySQL. He talked about Percona’s experimental DuckDB engine for MySQL 9.7: how query pushdown works, how results stay the same as in InnoDB, and the TPC-H results on 500 GB. He also told about a two-phase commit bug that made a replica lose data without any error. Frédéric Descamps (MariaDB Foundation), known in the community as “lefred”, worked many years at Percona and then in the MySQL Community Team. This year he joined the MariaDB Foundation. In his talk he showed MariaDB Server features that people often don’t know about: system-versioned tables, flashback, sequences, invisible columns and others. For every feature he explained which real problem it solves. Sergii Shchaslyvyi (AWS) talked about virtualization. After the licensing changes, many companies started to look for alternatives to their current virtualization platform. Sergii explained what migration options exist and where open source technologies make sense. Oleksiy Lukin (Percona) is not a new person for this conference. He worked at Chernihiv Polytechnic for more than 20 years and has been one of the organizers since 2009. Oleksiy presented kmipclient and kmipcore, an open source C++ library for KMIP (Key Management Interoperability Protocol). It supports KMIP 1.4 and 2.0, it is tested with five different KMIP servers, and Percona Server for MySQL already uses it. Vinicius Grippa (Readyset, Oracle ACE Director) finished the first day. Before Readyset he worked at Percona for eight years. He presented myflames, an open source tool that builds flame graphs, treemaps and Visual Explain style diagrams from MySQL execution plans. With these pictures it is much easier to understand where a query spends time. Day 2: track talks On the second day there were 13 talks from researchers, teachers and students. A team from Chernihiv Polytechnic presented a browser-based simulator for STM32 firmware and electrical circuits. Other talks were about microgrid optimization with CVXPY, local AI on x86 NPUs, digital education in crisis conditions, cyber hygiene for students and teaching steganography with free software. Four more Percona engineers spoke on this day: Adnan Šupić - zero-downtime migration with Percona ClusterSync for MongoDB Puneet Kala - a practical open source stack for running MongoDB Fabio Silva - how PMM hosts another application inside an iframe Briana Swift - how to manage public messaging for seven database technologies without a PMO, using a public GitHub repository and treating text like code Some numbers 82 registered participants 51 submitted theses 18 talks more than 90 people online participants from Ukraine, Belgium, Brazil, The Gambia, India and other countries Big thanks to the computer engineering students of Chernihiv Polytechnic. They were active during both days and asked speakers a lot of good questions. Thanks to our partners The conference would not be possible without our partners: Chernihiv Polytechnic National University - for hosting and organizing the conference. WIN-WIN EDIH - the conference was held as part of this project, co-funded by the European Union. Percona - for the support and for six speakers. MariaDB Foundation - for the support and the keynote about MariaDB Server. Amazon Web Services (AWS) - for the support and the keynote about migration from proprietary virtualization. Readyset - for the support and the keynote about myflames. OurSQL Foundation - for the support of the conference. PortaOne - for the support of the conference. Chernihiv IT Cluster (Chernihiv.IT) - for helping to connect the university with the local IT community. Thanks to you, the conference stays free, and Ukrainian students and researchers can listen to and talk with people from the international open source community. Recordings and materials Keynote recordings on YouTube: https://www.youtube.com/@foss-che Program and slides: https://feit.stu.cn.ua/foss/ Conference proceedings will be published at the end of October. Thanks to all speakers, authors and participants. See you at FOSS 2027! On October 1-2, Chernihiv Polytechnic National University held the II International scientific and practical conference “Free/Open Source Software in Education, Science & Business”. The conference was online and free for everyone. The idea is to bring together people from universities, from industry and from open source communities and talk about how free software is used in education, science and business.

This year Percona was well represented. There were six speakers from Percona, and two more keynote speakers also worked at Percona in the past.

Day 1: keynotes

The first day started with welcome words from Volodymyr Bazylevych, Chair of the Organizing Committee, and Anatolii Prystupa, Vice-Rector for Research. After that there were five keynote talks.

Yevhen Patlan (Percona) spoke about embedding DuckDB into MySQL. MySQL works very well for transactions, but analytical queries over hundreds of millions of rows can take a long time on InnoDB. Usually people solve this by copying data to a separate column store, so they need to support two systems and a pipeline between them. Yevhen showed another way: run DuckDB inside MySQL. He talked about Percona’s experimental DuckDB engine for MySQL 9.7: how query pushdown works, how results stay the same as in InnoDB, and the TPC-H results on 500 GB. He also told about a two-phase commit bug that made a replica lose data without any error.

Frédéric Descamps (MariaDB Foundation), known in the community as “lefred”, worked many years at Percona and then in the MySQL Community Team. This year he joined the MariaDB Foundation. In his talk he showed MariaDB Server features that people often don’t know about: system-versioned tables, flashback, sequences, invisible columns and others. For every feature he explained which real problem it solves.

Sergii Shchaslyvyi (AWS) talked about virtualization. After the licensing changes, many companies started to look for alternatives to their current virtualization platform. Sergii explained what migration options exist and where open source technologies make sense.

Oleksiy Lukin (Percona) is not a new person for this conference. He worked at Chernihiv Polytechnic for more than 20 years and has been one of the organizers since 2009. Oleksiy presented kmipclient and kmipcore, an open source C++ library for KMIP (Key Management Interoperability Protocol). It supports KMIP 1.4 and 2.0, it is tested with five different KMIP servers, and Percona Server for MySQL already uses it.

Vinicius Grippa (Readyset, Oracle ACE Director) finished the first day. Before Readyset he worked at Percona for eight years. He presented myflames, an open source tool that builds flame graphs, treemaps and Visual Explain style diagrams from MySQL execution plans. With these pictures it is much easier to understand where a query spends time.

Day 2: track talks

On the second day there were 13 talks from researchers, teachers and students. A team from Chernihiv Polytechnic presented a browser-based simulator for STM32 firmware and electrical circuits. Other talks were about microgrid optimization with CVXPY, local AI on x86 NPUs, digital education in crisis conditions, cyber hygiene for students and teaching steganography with free software.

Four more Percona engineers spoke on this day:

  • Adnan Šupić - zero-downtime migration with Percona ClusterSync for MongoDB
  • Puneet Kala - a practical open source stack for running MongoDB
  • Fabio Silva - how PMM hosts another application inside an iframe
  • Briana Swift - how to manage public messaging for seven database technologies without a PMO, using a public GitHub repository and treating text like code

Some numbers

  • 82 registered participants
  • 51 submitted theses
  • 18 talks
  • more than 90 people online
  • participants from Ukraine, Belgium, Brazil, The Gambia, India and other countries

Big thanks to the computer engineering students of Chernihiv Polytechnic. They were active during both days and asked speakers a lot of good questions.

Thanks to our partners

The conference would not be possible without our partners:

  • Chernihiv Polytechnic National University - for hosting and organizing the conference.
  • WIN-WIN EDIH - the conference was held as part of this project, co-funded by the European Union.
  • Percona - for the support and for six speakers.
  • MariaDB Foundation - for the support and the keynote about MariaDB Server.
  • Amazon Web Services (AWS) - for the support and the keynote about migration from proprietary virtualization.
  • Readyset - for the support and the keynote about myflames.
  • OurSQL Foundation - for the support of the conference.
  • PortaOne - for the support of the conference.
  • Chernihiv IT Cluster (Chernihiv.IT) - for helping to connect the university with the local IT community.

Thanks to you, the conference stays free, and Ukrainian students and researchers can listen to and talk with people from the international open source community.

Recordings and materials

Thanks to all speakers, authors and participants. See you at FOSS 2027!

]]> Mon, 12 Oct 2026 10:00:00 +0000 Percona Community Percona Community Events Conferences Open Source Education MySQL MongoDB PMM Meet MySQL 26.7: A New Versioning Era Begins https://www.percona.com/?p=54581 https://www.percona.com/blog/meet-mysql-26-7-a-new-versioning-era-begins/ Over the period, we have witnessed how MySQL versioning evolved from 8.0 to 9.7 LTS, and now it has entered a new era: a calendar-based model. Let’s first take a brief look at how MySQL versioning has evolved over time. Phase 1 – MySQL 8.0 (2018–2026) used a traditional single-track release model, with all new features and bug fixes rolled into the same 8.0 release line. Phase 2 – Oracle launches an Innovative/LTS dual-track model (Mid 2023). Innovation releases focus on new features and changes to short-lived performance on a quarterly basis. The LTS (Long Term Stable) release is steadier and has a long lifespan, with a cycle of 5 years of Premier Support + 3 years of Extended Support, and it typically launches every 2 years. Phase 3 – There are chains of Innovation releases following (8.1, 8.2, 8.3) between (2023- 2024), leading to the first LTS (8.4) in April 2024. Phase 4 – There was a shift in the Innovative versioning numbering, and we moved in the next 9.7 series. v9.0 to v9.6 are part of the Innovative release model (2024-26). We got our second LTS (9.7) in April 2026, with a 5-year Premier Support cycle followed by 3 years of Extended Support. After going through all those phases, we have finally reached a major shift in the overall versioning model. Instead of the next Innovation release to v9.8, Oracle switched to calendar versioning mode (Year.Month.Patch). So, the new Innovative release shipped as v26.7.0 instead of v9.8. Now, let’s discuss the new versioning model a bit. The new versioning model is denoted as Year.Month.Patch, or YY.M.P. 26 represents 2026(Year). 7 represents July (Month). 0 is the PATCH Number. The same quarterly cycle is followed for each subsequent Innovative release; the only difference is that it now uses the calendar versioning model. For example,  the next innovation release would tentatively be v26.10.0, scheduled for launch in October 2026. The LTS release can also be considered roughly every 2 years under the new versioning model. The next LTS release in the new scheme is expected in April 2028 as 28.4.0. Why choose the new Versioning Model? The real advantage we see is that with the format (Year.Month.Patch), we can better assess when releases/packages were shipped and how they relate to overall release planning (Innovative/LTS, etc.). This was a problem with the previous versioning model, say 8.4/9.7, as we don’t get much clue or information. This has no impact on the existing LTS (8.4/9.7); they will get the usual quarterly maintenance updates (bug fixes and security patches) while all Innovation or new features are offered under the new Calendar series. How does the upgrade cycle work? MySQL 9.7.x LTS is the sole entry point into the first calendar-versioned lineage 26.7.0, so we must first upgrade to 9.7 LTS if we need to go with a calendar version upgrade to 26.7.0. 8.4.x LTS or any previous version won’t simply work when considering upgrading to a new lineage. Singe hope upgrade from v9.7.x LTS to 26.7.0 or any later release in that same lineage (e.g., 26.10.0, 27.1.0, 27.4.0) is still valid until the next LTS, v28.4.0. E.g,8.4.x LTS -&gt; 9.7.x LTS -&gt; 26.7.0So if someone is still using an EOL version like 5.7, 8.0 has to follow the above path and upgrade to 8.4 LTS first. What features and enhancements does v26.7 bring to the table? 1) Change Stream Applier for Multi-Threaded Replication The new change stream applier (CSA) introduces further improvements to parallel transaction scheduling and execution. The original model, Multi-threaded Applier (MTA), basically uses a coordinator to read transactions sequentially from the relay files and then hand them off to worker threads for parallel applying and execution. This can be a limitation, especially when a worker thread is waiting/sitting Idle for a commit due to any prior or dependency transaction. Change stream applier (CSA)  basically separates the transaction applying from commit ordering. For example, if a worker finishes applying a transaction but has to wait for the commit order, it can work on another dependency-ready transaction instead.  The replication configurations below are introduced by CSR adoption. Please note that it works for each channel separately.APPLIER_VERSION - “1” denotes for MTA and “2“ denotes for CSA APPLIER_WORKER_COUNT - different worker count based on each replication channel APPLIER_EVENT_MEMORY_LIMIT - Per-channel memory limits for cached binlog events CSA has some more added advantages: CSA workers can simultaneously read transaction events from the relay log and release event memory when events are consumed. Relay log files can also be marked for concurrent purging once all transactions have been applied successfully and there is nothing left to read from the relay files. How to apply the change stream applier? mysql&gt; STOP REPLICA FOR CHANNEL 'mysql1';mysql&gt; CHANGE REPLICATION SOURCE TO APPLIER_VERSION = 2, APPLIER_WORKER_COUNT = 8, APPLIER_EVENT_MEMORY_LIMIT = 1073741824, REQUIRE_ROW_FORMAT = 1, GTID_ONLY = 1 FOR CHANNEL 'mysql1';mysql&gt; START REPLICA FOR CHANNEL 'mysql1'; Requirements to use CSA Row-Based replication (binlog_format=ROW) GTID enforcement (gtid_mode=ON) 2) Thread Pool Plugin The good news is that the Thread Pool Plugin is now available in MySQL Community releases, removing the previous dependency on the Enterprise Edition to use the Thread Pool mechanism. MySQL traditionally uses a one-thread-per-connection model, where each client connection creates a dedicated server thread. This approach can become a problem when an application establishes a very large number of concurrent connections, especially when many of those connections are Idle/Sleep(waiting for work). The Thread Pool addresses this by managing a high volume of client connections through a smaller number of worker threads. So, instead of creating a new thread for each connection, the Thread Pool uses the connections from the existing pool of threads and schedules their work across a controlled number of worker threads. This could be useful in an environment where applications trigger a very high number of concurrent database connections, and simply increasing max_connections is not an option, as it would lead to excessive thread creation and resource consumption during database setup. How to enable the Thread Pool? Load thread pool plugin [mysqld] plugin-load-add=thread_pool.so Restart the database service bash&gt; Systemctl restart mysql Verify plugin active mysql &gt; SELECT PLUGIN_NAME, PLUGIN_STATUS -&gt; FROM INFORMATION_SCHEMA.PLUGINS -&gt; WHERE PLUGIN_NAME LIKE 'thread%'; +-------------+---------------+ | PLUGIN_NAME | PLUGIN_STATUS | +-------------+---------------+ | thread_pool | ACTIVE | +-------------+---------------+ 1 row in set (0.004 sec)  Thread Pool configuration/Monitoring The thread pool size and other related parameters can be listed and configured as needed.mysql&gt; show variables like 'thread_pool%'; +----------------------------------------+-------+ | Variable_name | Value | +----------------------------------------+-------+ | thread_pool_algorithm | 1 | | thread_pool_connection_report_interval | 120 | | thread_pool_dedicated_listeners | OFF | | thread_pool_high_priority_connection | 0 | | thread_pool_longrun_trx_limit | 2000 | | thread_pool_max_active_query_threads | 0 | | thread_pool_max_transactions_limit | 192 | | thread_pool_max_unused_threads | 32 | | thread_pool_prio_kickup_timer | 1000 | | thread_pool_query_threads_per_group | 2 | | thread_pool_size | 6 | | thread_pool_stall_limit | 6 | | thread_pool_transaction_delay | 0 | +----------------------------------------+-------+ 13 rows in set (0.003 sec)Performance_schema provided a few tables to monitor thread pool activity, along with the status variables.mysql &gt; show tables like 'tp%'; +------------------------------------+ | Tables_in_performance_schema (tp%) | +------------------------------------+ | tp_connections | | tp_thread_group_state | | tp_thread_group_stats | | tp_thread_state | +------------------------------------+ 4 rows in set (0.001 sec)mysql&gt; show status like 'thread_pool_%'; +---------------------------+-------+ | Variable_name | Value | +---------------------------+-------+ | Thread_pool_query_threads | 14 | +---------------------------+-------+ 1 row in set (0.001 sec) 3) Undo Log Truncation Improvement MySQL 26.7 improves the robustness of InnoDB undo tablespace truncation. Previously, InnoDB used a separate undo_{space_number}_trunc.log file to keep track of an undo tablespace truncation operation. This external file was used to determine whether a truncation operation was in progress, particularly during recovery. Now this approach has changed: instead of maintaining a separate truncate log file, the truncation state is captured directly in the undo tablespace header. This improves both undo truncation and recovery as it does not rely on a separate dedicated file. 4)  InnoDB recovery and metadata infrastructure Improvement There were some internal changes, or rather improvements, to the structure and efficiency of the core InnoDB recovery, metadata, redo-log, tablespace, and transaction-visibility code paths. This doesn’t bring any changes to the usage point of view.  The goal is to make these components easier to maintain, test, and extend without changing user-visible SQL semantics. 5) Group Replication default communication stack The group_replication_communication_stack now defaults to “MySQL” from “XCOM”. This change is more aligned with the MySQL Server authentication and connection security.  This also means no separate communication port needs to be specified for group replication variables (group_replication_local_address and group_replication_group_seeds). The native MySQL port (3306)/ MEMBER_PORT or whatever is defined for MySQL nodes can be used instead. In short, the default MySQL communication stack provides the benefits below.  Overall, it simplifies the creation of InnoDB Clusters by using MySQL Server’s connection security in place of the Group Replication implementation. It removes the need for an extra network address or port for internal Group Replication communications. Using the MySQL protocol means that standard user authentication methods can be used to grant or revoke access to the group, in place of the allow list. Supports network namespaces for Group Replication. Note – If migrating from the old environment or switching from “XCOM” to “MySQL” simply switching the communication stack wouldn’t work.mysql&gt; set global group_replication_communication_stack = 'MySQL';Group replication must be stopped on all members, and then the relevant changes should be applied. Finally, the cluster needs to be bootstrapped.  Basically, this activity requires downtime to successfully switch the communication stack. Also, the XCom communication stack is deprecated and could be removed in future releases, so better to switch to the MySQL stack. Further complete release notes can be checked here – https://dev.mysql.com/doc/relnotes/mysql/26.7/en/news-26-7-0.html Conclusion The transition to MySQL 26.7 introduces a major shift to a calendar-based versioning scheme (Year.Month.Patch), providing clearer alignment with release timelines and future planning across both Innovation releases and long-term support cycles. Upgrading to this new series requires being on MySQL 9.7 LTS as the primary baseline before moving forward into the 26.x series. Alongside versioning improvements, MySQL 26.7 delivers significant architecture and usability enhancements. Key features include the Change Stream Applier (CSA) for optimised parallel transaction execution and replication performance, as well as the native availability of the Thread Pool Plugin in Community editions to manage high-concurrency connections effectively without relying on Enterprise Edition extras. The post Meet MySQL 26.7: A New Versioning Era Begins appeared first on Percona. Over the period, we have witnessed how MySQL versioning evolved from 8.0 to 9.7 LTS, and now it has entered a new era: a calendar-based model.

Let’s first take a brief look at how MySQL versioning has evolved over time.

  • Phase 1 – MySQL 8.0 (2018–2026) used a traditional single-track release model, with all new features and bug fixes rolled into the same 8.0 release line.
  • Phase 2 – Oracle launches an Innovative/LTS dual-track model (Mid 2023). Innovation releases focus on new features and changes to short-lived performance on a quarterly basis. The LTS (Long Term Stable) release is steadier and has a long lifespan, with a cycle of 5 years of Premier Support + 3 years of Extended Support, and it typically launches every 2 years.
  • Phase 3 – There are chains of Innovation releases following (8.1, 8.2, 8.3) between (2023- 2024), leading to the first LTS (8.4) in April 2024.
  • Phase 4 – There was a shift in the Innovative versioning numbering, and we moved in the next 9.7 series. v9.0 to v9.6 are part of the Innovative release model (2024-26). We got our second LTS (9.7) in April 2026, with a 5-year Premier Support cycle followed by 3 years of Extended Support.

After going through all those phases, we have finally reached a major shift in the overall versioning model. Instead of the next Innovation release to v9.8, Oracle switched to calendar versioning mode (Year.Month.Patch). So, the new Innovative release shipped as v26.7.0 instead of v9.8.

Now, let’s discuss the new versioning model a bit.

The new versioning model is denoted as Year.Month.Patch, or YY.M.P.

  • 26 represents 2026(Year).
  • 7 represents July (Month).
  • 0 is the PATCH Number.

The same quarterly cycle is followed for each subsequent Innovative release; the only difference is that it now uses the calendar versioning model. For example,  the next innovation release would tentatively be v26.10.0, scheduled for launch in October 2026.

The LTS release can also be considered roughly every 2 years under the new versioning model. The next LTS release in the new scheme is expected in April 2028 as 28.4.0.

Why choose the new Versioning Model?

The real advantage we see is that with the format (Year.Month.Patch), we can better assess when releases/packages were shipped and how they relate to overall release planning (Innovative/LTS, etc.). This was a problem with the previous versioning model, say 8.4/9.7, as we don’t get much clue or information.

This has no impact on the existing LTS (8.4/9.7); they will get the usual quarterly maintenance updates (bug fixes and security patches) while all Innovation or new features are offered under the new Calendar series.

How does the upgrade cycle work?

  • MySQL 9.7.x LTS is the sole entry point into the first calendar-versioned lineage 26.7.0, so we must first upgrade to 9.7 LTS if we need to go with a calendar version upgrade to 26.7.0.
  • 8.4.x LTS or any previous version won’t simply work when considering upgrading to a new lineage.
  • Singe hope upgrade from v9.7.x LTS to 26.7.0 or any later release in that same lineage (e.g., 26.10.0, 27.1.0, 27.4.0) is still valid until the next LTS, v28.4.0.

E.g,

8.4.x LTS -> 9.7.x LTS -> 26.7.0

So if someone is still using an EOL version like 5.7, 8.0 has to follow the above path and upgrade to 8.4 LTS first.

What features and enhancements does v26.7 bring to the table?

1) Change Stream Applier for Multi-Threaded Replication

The new change stream applier (CSA) introduces further improvements to parallel transaction scheduling and execution. The original model, Multi-threaded Applier (MTA), basically uses a coordinator to read transactions sequentially from the relay files and then hand them off to worker threads for parallel applying and execution. This can be a limitation, especially when a worker thread is waiting/sitting Idle for a commit due to any prior or dependency transaction.

Change stream applier (CSA)  basically separates the transaction applying from commit ordering. For example, if a worker finishes applying a transaction but has to wait for the commit order, it can work on another dependency-ready transaction instead. 

The replication configurations below are introduced by CSR adoption. Please note that it works for each channel separately.

APPLIER_VERSION  -  “1” denotes for MTA and “2“ denotes for CSA
APPLIER_WORKER_COUNT - different worker count based on each replication channel 
APPLIER_EVENT_MEMORY_LIMIT - Per-channel memory limits for cached binlog events

CSA has some more added advantages:

  • CSA workers can simultaneously read transaction events from the relay log and release event memory when events are consumed.
  • Relay log files can also be marked for concurrent purging once all transactions have been applied successfully and there is nothing left to read from the relay files.

How to apply the change stream applier?

mysql> STOP REPLICA FOR CHANNEL 'mysql1';

mysql> CHANGE REPLICATION SOURCE TO
APPLIER_VERSION = 2,
APPLIER_WORKER_COUNT = 8,
APPLIER_EVENT_MEMORY_LIMIT = 1073741824,
REQUIRE_ROW_FORMAT = 1,
GTID_ONLY = 1
FOR CHANNEL 'mysql1';

mysql> START REPLICA FOR CHANNEL 'mysql1';

Requirements to use CSA

Row-Based replication (binlog_format=ROW)
GTID enforcement (gtid_mode=ON)

2) Thread Pool Plugin

The good news is that the Thread Pool Plugin is now available in MySQL Community releases, removing the previous dependency on the Enterprise Edition to use the Thread Pool mechanism.

MySQL traditionally uses a one-thread-per-connection model, where each client connection creates a dedicated server thread. This approach can become a problem when an application establishes a very large number of concurrent connections, especially when many of those connections are Idle/Sleep(waiting for work).

The Thread Pool addresses this by managing a high volume of client connections through a smaller number of worker threads. So, instead of creating a new thread for each connection, the Thread Pool uses the connections from the existing pool of threads and schedules their work across a controlled number of worker threads.

This could be useful in an environment where applications trigger a very high number of concurrent database connections, and simply increasing max_connections is not an option, as it would lead to excessive thread creation and resource consumption during database setup.

How to enable the Thread Pool?

  • Load thread pool plugin

[mysqld]
plugin-load-add=thread_pool.so

  • Restart the database service

bash> Systemctl restart mysql

  • Verify plugin active

mysql >  SELECT PLUGIN_NAME, PLUGIN_STATUS
    ->        FROM INFORMATION_SCHEMA.PLUGINS
    ->        WHERE PLUGIN_NAME LIKE 'thread%';
+-------------+---------------+
| PLUGIN_NAME | PLUGIN_STATUS |
+-------------+---------------+
| thread_pool | ACTIVE        |
+-------------+---------------+
1 row in set (0.004 sec)

 

Thread Pool configuration/Monitoring

The thread pool size and other related parameters can be listed and configured as needed.

mysql> show variables like 'thread_pool%';
+----------------------------------------+-------+
| Variable_name                          | Value |
+----------------------------------------+-------+
| thread_pool_algorithm                  | 1     |
| thread_pool_connection_report_interval | 120   |
| thread_pool_dedicated_listeners        | OFF   |
| thread_pool_high_priority_connection   | 0     |
| thread_pool_longrun_trx_limit          | 2000  |
| thread_pool_max_active_query_threads   | 0     |
| thread_pool_max_transactions_limit     | 192   |
| thread_pool_max_unused_threads         | 32    |
| thread_pool_prio_kickup_timer          | 1000  |
| thread_pool_query_threads_per_group    | 2     |
| thread_pool_size                       | 6     |
| thread_pool_stall_limit                | 6     |
| thread_pool_transaction_delay          | 0     |
+----------------------------------------+-------+
13 rows in set (0.003 sec)

Performance_schema provided a few tables to monitor thread pool activity, along with the status variables.

mysql > show tables like 'tp%';
+------------------------------------+
| Tables_in_performance_schema (tp%) |
+------------------------------------+
| tp_connections                     |
| tp_thread_group_state              |
| tp_thread_group_stats              |
| tp_thread_state                    |
+------------------------------------+
4 rows in set (0.001 sec)

mysql>  show status like 'thread_pool_%';
+---------------------------+-------+
| Variable_name             | Value |
+---------------------------+-------+
| Thread_pool_query_threads | 14    |
+---------------------------+-------+
1 row in set (0.001 sec)

3) Undo Log Truncation Improvement

MySQL 26.7 improves the robustness of InnoDB undo tablespace truncation. Previously, InnoDB used a separate undo_{space_number}_trunc.log file to keep track of an undo tablespace truncation operation. This external file was used to determine whether a truncation operation was in progress, particularly during recovery.

Now this approach has changed: instead of maintaining a separate truncate log file, the truncation state is captured directly in the undo tablespace header.

This improves both undo truncation and recovery as it does not rely on a separate dedicated file.

4)  InnoDB recovery and metadata infrastructure Improvement

There were some internal changes, or rather improvements, to the structure and efficiency of the core InnoDB recovery, metadata, redo-log, tablespace, and transaction-visibility code paths.

This doesn’t bring any changes to the usage point of view.  The goal is to make these components easier to maintain, test, and extend without changing user-visible SQL semantics.

5) Group Replication default communication stack

The group_replication_communication_stack now defaults to “MySQL” from “XCOM”. This change is more aligned with the MySQL Server authentication and connection security. 

This also means no separate communication port needs to be specified for group replication variables (group_replication_local_address and group_replication_group_seeds). The native MySQL port (3306)/ MEMBER_PORT or whatever is defined for MySQL nodes can be used instead.

In short, the default MySQL communication stack provides the benefits below. 

  • Overall, it simplifies the creation of InnoDB Clusters by using MySQL Server’s connection security in place of the Group Replication implementation.
  • It removes the need for an extra network address or port for internal Group Replication communications.
  • Using the MySQL protocol means that standard user authentication methods can be used to grant or revoke access to the group, in place of the allow list.
  • Supports network namespaces for Group Replication.

Note – If migrating from the old environment or switching from “XCOM” to “MySQL” simply switching the communication stack wouldn’t work.

mysql> set global group_replication_communication_stack = 'MySQL';

Group replication must be stopped on all members, and then the relevant changes should be applied. Finally, the cluster needs to be bootstrapped. 

Basically, this activity requires downtime to successfully switch the communication stack.

Also, the XCom communication stack is deprecated and could be removed in future releases, so better to switch to the MySQL stack. Further complete release notes can be checked here – https://dev.mysql.com/doc/relnotes/mysql/26.7/en/news-26-7-0.html

Conclusion

The transition to MySQL 26.7 introduces a major shift to a calendar-based versioning scheme (Year.Month.Patch), providing clearer alignment with release timelines and future planning across both Innovation releases and long-term support cycles. Upgrading to this new series requires being on MySQL 9.7 LTS as the primary baseline before moving forward into the 26.x series.

Alongside versioning improvements, MySQL 26.7 delivers significant architecture and usability enhancements. Key features include the Change Stream Applier (CSA) for optimised parallel transaction execution and replication performance, as well as the native availability of the Thread Pool Plugin in Community editions to manage high-concurrency connections effectively without relying on Enterprise Edition extras.

The post Meet MySQL 26.7: A New Versioning Era Begins appeared first on Percona.

]]> Mon, 12 Oct 2026 04:05:49 +0000 MySQL Performance Blog Cloud Featured Insight for DBAs Insight for Developers Kubernetes MySQL Open Source Operator Percona Services Percona Software Storage Engine Uncategorized MySQL Asynchronous replication MySQL GTID MySQL Replication MySQL Thread Pool MySQL V26.7 Docker DBA Lab https://percona.community/blog/2026/10/12/docker-dba-lab/ https://percona.community/blog/2026/10/12/docker-dba-lab/ Building a Database DBA Lab with Docker, PostgreSQL, MySQL, and Percona Monitoring and Management How I built a local database playground, learned a few lessons the hard way, and created a practical command reference along the way. Introduction One of the best ways to learn database administration is to get your hands dirty. Documentation is important, but there’s something different about building your own environment, breaking it, troubleshooting it, and eventually understanding why everything works. Recently, I decided to create a local database lab using Docker Desktop on my Mac. My goal was straightforward: build an environment where I could experiment with MySQL, PostgreSQL, Percona XtraDB Cluster (PXC), and Percona Monitoring and Management (PMM), without worrying about disrupting a production environment. What started as a few containers quickly became a much more interesting learning experience. Along the way, I ran into Docker networking problems, container persistence challenges, cluster bootstrap issues, and PostgreSQL monitoring configuration. This article shows the setup I recommend after working through those problems, so you can start with the right foundation instead of discovering it at the end. 1. Building the Lab The foundation of my environment is Docker Desktop. The lab includes: Percona Server for MySQL 8.4 A three-node Percona XtraDB Cluster PostgreSQL 17 PMM Server and PMM Client Persistent Docker volumes A shared Docker network The architecture looks like this: text Copy Copied! Docker Desktop │ ├── pmm-network │ ├── pmm-server │ ├── pmm-client │ ├── percona-server │ ├── postgres-server │ ├── pxc1 │ ├── pxc2 │ └── pxc3 │ └── Persistent Docker Volumes ├── pmm-data ├── pmm-client-data ├── mysql-data ├── postgres-data ├── pxc1_data ├── pxc2_data └── pxc3_data There are two ways to build this. Option A (recommended): Docker Compose Docker Compose lets you define every container, volume, and network attachment in one file. The whole lab becomes reproducible: one command brings it up, one command shuts it down, and nothing depends on remembering which docker run flags you used weeks ago. I started out with individual docker run commands and later moved everything to Compose. If I were starting again, I would begin here. First, keep passwords out of the Compose file by putting them in a .env file next to it (and out of version control): bash Copy Copied! # .env MYSQL_ROOT_PASSWORD=change-me PXC_ROOT_PASSWORD=change-me POSTGRES_PASSWORD=change-me PMM_ADMIN_PASSWORD=change-me PG_MONITOR_PASSWORD=change-me MYSQL_MONITOR_PASSWORD=change-me Then define the lab in docker-compose.yml: yaml Copy Copied! services: pmm-server: image: percona/pmm-server:3.9.1 platform: linux/amd64 container_name: pmm-server ports: - "443:8443" volumes: - pmm-data:/srv networks: [pmm-network] pmm-client: image: percona/pmm-client:3.9.1 container_name: pmm-client depends_on: [pmm-server] environment: PMM_AGENT_SERVER_ADDRESS: pmm-server:8443 PMM_AGENT_SERVER_USERNAME: admin PMM_AGENT_SERVER_PASSWORD: ${PMM_ADMIN_PASSWORD} PMM_AGENT_SERVER_INSECURE_TLS: 1 PMM_AGENT_SETUP: 1 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml volumes: - pmm-client-data:/usr/local/percona/pmm/config networks: [pmm-network] percona-server: image: percona/percona-server:8.4.11 container_name: percona-server environment: MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD} ports: - "3306:3306" volumes: - mysql-data:/var/lib/mysql networks: [pmm-network] postgres-server: image: postgres:17 container_name: postgres-server environment: POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} command: &gt; postgres -c shared_preload_libraries=pg_stat_statements -c track_io_timing=on ports: - "5432:5432" volumes: - postgres-data:/var/lib/postgresql/data networks: [pmm-network] pxc1: image: percona/percona-xtradb-cluster:8.4.11 container_name: pxc1 environment: CLUSTER_NAME: pxc-lab MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD} volumes: - pxc1_data:/var/lib/mysql - ./certs:/etc/mysql/certs:ro - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro stop_grace_period: 120s healthcheck: test: ["CMD", "mysqladmin", "ping", "-uroot", "-p${PXC_ROOT_PASSWORD}"] interval: 10s retries: 30 networks: [pmm-network] ports: - "3307:3306" pxc2: image: percona/percona-xtradb-cluster:8.4.11 container_name: pxc2 environment: CLUSTER_NAME: pxc-lab CLUSTER_JOIN: pxc1 MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD} volumes: - pxc2_data:/var/lib/mysql - ./certs:/etc/mysql/certs:ro - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro stop_grace_period: 120s depends_on: pxc1: condition: service_healthy networks: [pmm-network] ports: - "3308:3306" pxc3: image: percona/percona-xtradb-cluster:8.4.11 container_name: pxc3 environment: CLUSTER_NAME: pxc-lab CLUSTER_JOIN: pxc1 MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD} volumes: - pxc3_data:/var/lib/mysql - ./certs:/etc/mysql/certs:ro - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro stop_grace_period: 120s depends_on: - pxc2 networks: [pmm-network] ports: - "3309:3306" networks: pmm-network: name: pmm-network volumes: pmm-data: pmm-client-data: mysql-data: postgres-data: pxc1_data: pxc2_data: pxc3_data: A few design choices are worth calling out, because each one comes from a problem I hit: Volumes and the network are declared up front. Every database has persistent storage and is attached to pmm-network from the moment it is created, so there is nothing to wire up afterward. All PXC nodes share the same SSL certificates. Mismatched certificates were behind several of my early cluster join failures. stop_grace_period: 120s gives each PXC node time to shut down cleanly instead of being killed (exit code 137). depends_on defines the PXC order. Compose starts pxc1 first and waits for it to be healthy before the others join. On shutdown, Compose stops dependents first, so the nodes leave in the order pxc3, pxc2, pxc1. Bring the lab up and down with: bash Copy Copied! docker compose up -d docker compose stop # stops containers, keeps everything docker compose down # removes containers, keeps named volumes Option B (alternative): Individual docker run commands If you prefer to build containers one at a time, or want to understand what Compose is doing for you, you can create the network yourself and attach containers to it: bash Copy Copied! docker network create pmm-network docker run -d --name postgres-server \ --network pmm-network \ -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \ -v postgres-data:/var/lib/postgresql/data \ -p 5432:5432 \ postgres:17 An existing container can be attached later: bash Copy Copied! docker network connect pmm-network postgres-server This works, but you are responsible for recreating each container with exactly the same flags, volumes, and network settings every time. That is the main reason I recommend Option A. Whichever option you choose, avoid mixing the two for the same lab, because it becomes hard to tell which containers are managed by Compose and which are not. A networking lesson that applies either way At one point, PMM couldn’t resolve my PostgreSQL container: text Copy Copied! lookup postgres-server on 127.0.0.11:53: no such host The problem wasn’t PostgreSQL or PMM. The PostgreSQL container was stopped, so it had no active network endpoint, and Docker’s internal DNS had nothing to return. That was an important reminder: always verify the container and network state before troubleshooting the application. 2. Making Monitoring Persistent with PMM Once the databases were running, I wanted visibility into their performance. PMM provides dashboards and metrics for MySQL, PXC, and PostgreSQL. It consists of a server that stores and visualizes monitoring data and a client agent that collects database metrics. The key thing to understand is that persistent storage and service registration are related, but they are not the same thing. PMM Server maintains the monitoring inventory, while PMM Client has its own configuration and agent identity. If the client is recreated with a new identity, the association between the client and its registered services is lost. In my early setup, after recreating containers, the client listing showed only the basic agents: text Copy Copied! pmm_agent Connected node_exporter Running vmagent Running Two things prevent this, and both should be in place from the start: A persistent volume for the client configuration. In the Compose file above, pmm-client-data keeps the agent identity across restarts and recreations. An idempotent registration script. It adds each database only if it isn’t already registered, so it is safe to run after every docker compose up. Before registering, create a monitoring user in each database. For PostgreSQL, for example: sql Copy Copied! CREATE USER pmm WITH PASSWORD 'change-me'; GRANT pg_monitor TO pmm; CREATE EXTENSION IF NOT EXISTS pg_stat_statements; Then the registration script, which reads credentials from the .env file rather than hardcoding them: bash Copy Copied! #!/usr/bin/env bash # register-pmm.sh set -euo pipefail source .env registered() { docker exec pmm-client pmm-admin list | grep -qw "$1" } if ! registered postgres-server; then docker exec pmm-client pmm-admin add postgresql \ --username=pmm \ --password="$PG_MONITOR_PASSWORD" \ --host=postgres-server \ --port=5432 \ postgres-server fi for node in percona-server pxc1 pxc2 pxc3; do if ! registered "$node"; then docker exec pmm-client pmm-admin add mysql \ --username=pmm \ --password="$MYSQL_MONITOR_PASSWORD" \ --host="$node" \ --port=3306 \ --query-source=perfschema \ "$node" fi done docker exec pmm-client pmm-admin list The takeaway: a monitoring system should be designed to survive a restart, not just work immediately after installation. 3. Learning PostgreSQL Through Real Workloads With PostgreSQL running and connected to PMM, I wanted to do more than look at an idle dashboard. I used sysbench to generate a workload against PostgreSQL 17. My test environment included ten tables with approximately 750,000 rows each, giving me about 7.5 million rows to experiment with. A sample workload command looked like this: bash Copy Copied! sysbench \ --db-driver=pgsql \ --pgsql-host=127.0.0.1 \ --pgsql-port=5432 \ --pgsql-user=postgres \ --pgsql-db=sbtest \ --tables=10 \ --table-size=750000 \ --threads=16 \ --time=300 \ --report-interval=5 \ oltp_read_write run With the workload running, PMM became much more interesting. I could observe database activity, disk latency, query execution statistics, and changes in resource utilization. This also helped me understand an important distinction: PostgreSQL tuple activity is not the same thing as SQL queries per second. One query can read or return many tuples. Looking only at tuple counts without understanding the underlying queries can lead to incorrect conclusions about workload intensity. 4. Understanding PostgreSQL Query Performance One of the most useful PostgreSQL features for troubleshooting is EXPLAIN. For example: sql Copy Copied! EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM sbtest1 WHERE id BETWEEN 100 AND 200; This shows the execution plan, actual execution measurements, and buffer activity. I started paying attention to several key pieces of information: Whether PostgreSQL used a sequential scan or an index Estimated versus actual rows Execution time Buffer hits and reads Number of execution loops Because the Compose file loads pg_stat_statements at startup, query statistics are aggregated from the beginning. A useful query is: sql Copy Copied! SELECT calls, total_exec_time, mean_exec_time, rows, LEFT(query, 100) AS query FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10; This provides a different perspective from looking at individual slow queries. A query that takes only a fraction of a millisecond might still consume substantial cumulative execution time if it runs hundreds of thousands of times. For a DBA, both individual latency and aggregate workload matter. 5. MVCC, Dead Tuples, and VACUUM My next learning objective was understanding PostgreSQL’s Multi-Version Concurrency Control (MVCC). Unlike an in-place overwrite model, PostgreSQL commonly creates a new row version when a row is updated. Older versions may remain until PostgreSQL determines they are no longer needed. That leads directly to one of PostgreSQL’s most important maintenance mechanisms: VACUUM. I experimented with updating large numbers of rows: sql Copy Copied! UPDATE sbtest1 SET k = k + 1 WHERE id 300000; Then checked the table statistics: sql Copy Copied! SELECT relname, n_live_tup, n_dead_tup, last_autovacuum, autovacuum_count FROM pg_stat_user_tables WHERE relname = 'sbtest1'; This gave me a practical way to observe dead tuple estimates and autovacuum activity. I also explored what happens when a long-running transaction holds an old snapshot. By starting a REPEATABLE READ transaction and leaving it open while another session updated rows, I could demonstrate how PostgreSQL may need to retain older row versions. That helped connect several important DBA concepts: Long-running transactions → old snapshots → delayed tuple cleanup → potential table bloat. I also learned that normal VACUUM generally makes space reusable within PostgreSQL rather than immediately shrinking the table file. VACUUM FULL can reclaim disk space by rewriting a table, but it requires an exclusive lock and is not something to run casually in production. 6. Operating the PXC Cluster Safely My three-node Percona XtraDB Cluster taught me that database containers need database-aware operational procedures. That’s why startup and shutdown are built into the Compose file rather than left to chance. Early on, I ran into certificate problems, donor timeouts, cluster membership issues, and shutdowns that ended with exit code 137 (the container being killed before MySQL finished). The shared certificate directory, the healthcheck on pxc1, and stop_grace_period address those directly. The shutdown procedure is simply: bash Copy Copied! docker compose stop pxc3 pxc2 pxc1 If you use Option B, do the same thing explicitly, one node at a time, with a long timeout: bash Copy Copied! docker stop -t 120 pxc3 docker stop -t 120 pxc2 docker stop -t 120 pxc1 To start the cluster again, check grastate.dat on each node and understand the meaning of safe_to_bootstrap: bash Copy Copied! docker run --rm -v pxc1_data:/data alpine cat /data/grastate.dat After a clean shutdown, the last node to leave the cluster (here, pxc1) will have safe_to_bootstrap: 1 and is the right node to start first. After an unclean shutdown, selecting the correct bootstrap node requires recovery checks. Compare the recovered sequence numbers across nodes and bootstrap from the most advanced one. Simply changing safe_to_bootstrap to 1 without verifying that can lose data. 7. My Most Useful Commands I kept a command cheat sheet from the start of the project, and it has become one of the most useful outcomes. It captures the commands I actually needed while learning and troubleshooting. I recommend starting one on day one. For Docker: bash Copy Copied! # Running containers docker ps # All containers docker ps -a # Status of the Compose-managed lab docker compose ps # Follow logs docker logs -f postgres-server # Inspect network configuration docker inspect postgres-server \ --format '{{json .NetworkSettings.Networks}}' # Resource usage docker stats # List Docker volumes docker volume ls # Check PMM services docker exec pmm-client pmm-admin list These commands are my first line of investigation whenever something isn’t behaving as expected. For PostgreSQL, my daily reference includes: sql Copy Copied! -- Current sessions SELECT pid, usename, state, query FROM pg_stat_activity; -- Long-running transactions SELECT pid, state, now() - xact_start AS transaction_age, query FROM pg_stat_activity WHERE xact_start IS NOT NULL; -- Dead tuples SELECT relname, n_live_tup, n_dead_tup FROM pg_stat_user_tables ORDER BY n_dead_tup DESC; -- Blocking sessions SELECT pid, pg_blocking_pids(pid) AS blocked_by FROM pg_stat_activity WHERE cardinality(pg_blocking_pids(pid)) &gt; 0; These are simple queries, but they form a useful starting point for troubleshooting. 8. Design Principles Behind the Lab To summarize, these are the principles the setup above is built on: Define the whole lab in Docker Compose, rather than mixing docker run commands with Compose-managed containers. Declare persistent volumes and network attachments before creating any databases. Treat startup and shutdown as part of the design, especially for PXC. Make PMM registration reproducible with a persistent client volume and an idempotent script, and keep credentials out of scripts and command history. Keep a command cheat sheet from the beginning. Final Thoughts What began as an effort to build a small local database environment turned into a practical DBA learning experience. I now have a place to explore PostgreSQL performance, MVCC, autovacuum, MySQL replication concepts, PXC clustering, and database monitoring without risking a production system. Perhaps the biggest lesson is that the most valuable learning moments often happen when something breaks. A Docker DNS failure teaches networking. A missing PMM registration teaches persistence and identity. A PXC bootstrap problem teaches cluster recovery. A growing dead tuple count teaches PostgreSQL MVCC. And once you understand why something broke, you’re much better prepared to troubleshoot it the next time. My next goal is to continue expanding the lab with PostgreSQL replication, backup and recovery, and more deliberate performance troubleshooting exercises. This article describes a local learning environment. The commands and configurations are intended for experimentation and should be reviewed and adapted before use in production. Building a Database DBA Lab with Docker, PostgreSQL, MySQL, and Percona Monitoring and Management

How I built a local database playground, learned a few lessons the hard way, and created a practical command reference along the way.

Introduction

One of the best ways to learn database administration is to get your hands dirty.

Documentation is important, but there’s something different about building your own environment, breaking it, troubleshooting it, and eventually understanding why everything works.

Recently, I decided to create a local database lab using Docker Desktop on my Mac. My goal was straightforward: build an environment where I could experiment with MySQL, PostgreSQL, Percona XtraDB Cluster (PXC), and Percona Monitoring and Management (PMM), without worrying about disrupting a production environment.

What started as a few containers quickly became a much more interesting learning experience.

Along the way, I ran into Docker networking problems, container persistence challenges, cluster bootstrap issues, and PostgreSQL monitoring configuration. This article shows the setup I recommend after working through those problems, so you can start with the right foundation instead of discovering it at the end.

1. Building the Lab

The foundation of my environment is Docker Desktop.

The lab includes:

  • Percona Server for MySQL 8.4
  • A three-node Percona XtraDB Cluster
  • PostgreSQL 17
  • PMM Server and PMM Client
  • Persistent Docker volumes
  • A shared Docker network

The architecture looks like this:

text 
Docker Desktop
│
├── pmm-network
│ ├── pmm-server
│ ├── pmm-client
│ ├── percona-server
│ ├── postgres-server
│ ├── pxc1
│ ├── pxc2
│ └── pxc3
│
└── Persistent Docker Volumes
 ├── pmm-data
 ├── pmm-client-data
 ├── mysql-data
 ├── postgres-data
 ├── pxc1_data
 ├── pxc2_data
 └── pxc3_data

There are two ways to build this.

Option A (recommended): Docker Compose

Docker Compose lets you define every container, volume, and network attachment in one file. The whole lab becomes reproducible: one command brings it up, one command shuts it down, and nothing depends on remembering which docker run flags you used weeks ago.

I started out with individual docker run commands and later moved everything to Compose. If I were starting again, I would begin here.

First, keep passwords out of the Compose file by putting them in a .env file next to it (and out of version control):

bash 
# .env
MYSQL_ROOT_PASSWORD=change-me
PXC_ROOT_PASSWORD=change-me
POSTGRES_PASSWORD=change-me
PMM_ADMIN_PASSWORD=change-me
PG_MONITOR_PASSWORD=change-me
MYSQL_MONITOR_PASSWORD=change-me

Then define the lab in docker-compose.yml:

yaml 
services:
 pmm-server:
 image: percona/pmm-server:3.9.1
 platform: linux/amd64
 container_name: pmm-server
 ports:
 - "443:8443"
 volumes:
 - pmm-data:/srv
 networks: [pmm-network]

 pmm-client:
 image: percona/pmm-client:3.9.1
 container_name: pmm-client
 depends_on: [pmm-server]
 environment:
 PMM_AGENT_SERVER_ADDRESS: pmm-server:8443
 PMM_AGENT_SERVER_USERNAME: admin
 PMM_AGENT_SERVER_PASSWORD: ${PMM_ADMIN_PASSWORD}
 PMM_AGENT_SERVER_INSECURE_TLS: 1
 PMM_AGENT_SETUP: 1
 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml
 volumes:
 - pmm-client-data:/usr/local/percona/pmm/config
 networks: [pmm-network]

 percona-server:
 image: percona/percona-server:8.4.11
 container_name: percona-server
 environment:
 MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
 ports:
 - "3306:3306"
 volumes:
 - mysql-data:/var/lib/mysql
 networks: [pmm-network]

 postgres-server:
 image: postgres:17
 container_name: postgres-server
 environment:
 POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
 command: >
 postgres
 -c shared_preload_libraries=pg_stat_statements
 -c track_io_timing=on
 ports:
 - "5432:5432"
 volumes:
 - postgres-data:/var/lib/postgresql/data
 networks: [pmm-network]

 pxc1:
 image: percona/percona-xtradb-cluster:8.4.11
 container_name: pxc1
 environment:
 CLUSTER_NAME: pxc-lab
 MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD}
 volumes:
 - pxc1_data:/var/lib/mysql
 - ./certs:/etc/mysql/certs:ro
 - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
 stop_grace_period: 120s
 healthcheck:
 test: ["CMD", "mysqladmin", "ping", "-uroot", "-p${PXC_ROOT_PASSWORD}"]
 interval: 10s
 retries: 30
 networks: [pmm-network]
 ports:
 - "3307:3306"

 pxc2:
 image: percona/percona-xtradb-cluster:8.4.11
 container_name: pxc2
 environment:
 CLUSTER_NAME: pxc-lab
 CLUSTER_JOIN: pxc1
 MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD}
 volumes:
 - pxc2_data:/var/lib/mysql
 - ./certs:/etc/mysql/certs:ro
 - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
 stop_grace_period: 120s
 depends_on:
 pxc1:
 condition: service_healthy
 networks: [pmm-network]
 ports:
 - "3308:3306"

 pxc3:
 image: percona/percona-xtradb-cluster:8.4.11
 container_name: pxc3
 environment:
 CLUSTER_NAME: pxc-lab
 CLUSTER_JOIN: pxc1
 MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD}
 volumes:
 - pxc3_data:/var/lib/mysql
 - ./certs:/etc/mysql/certs:ro
 - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
 stop_grace_period: 120s
 depends_on:
 - pxc2
 networks: [pmm-network]
 ports:
 - "3309:3306"

networks:
 pmm-network:
 name: pmm-network

volumes:
 pmm-data:
 pmm-client-data:
 mysql-data:
 postgres-data:
 pxc1_data:
 pxc2_data:
 pxc3_data:

A few design choices are worth calling out, because each one comes from a problem I hit:

  • Volumes and the network are declared up front. Every database has persistent storage and is attached to pmm-network from the moment it is created, so there is nothing to wire up afterward.
  • All PXC nodes share the same SSL certificates. Mismatched certificates were behind several of my early cluster join failures.
  • stop_grace_period: 120s gives each PXC node time to shut down cleanly instead of being killed (exit code 137).
  • depends_on defines the PXC order. Compose starts pxc1 first and waits for it to be healthy before the others join. On shutdown, Compose stops dependents first, so the nodes leave in the order pxc3, pxc2, pxc1.

Bring the lab up and down with:

bash 
docker compose up -d
docker compose stop # stops containers, keeps everything
docker compose down # removes containers, keeps named volumes

Option B (alternative): Individual docker run commands

If you prefer to build containers one at a time, or want to understand what Compose is doing for you, you can create the network yourself and attach containers to it:

bash 
docker network create pmm-network

docker run -d --name postgres-server \
 --network pmm-network \
 -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \
 -v postgres-data:/var/lib/postgresql/data \
 -p 5432:5432 \
 postgres:17

An existing container can be attached later:

bash 
docker network connect pmm-network postgres-server

This works, but you are responsible for recreating each container with exactly the same flags, volumes, and network settings every time. That is the main reason I recommend Option A. Whichever option you choose, avoid mixing the two for the same lab, because it becomes hard to tell which containers are managed by Compose and which are not.

A networking lesson that applies either way

At one point, PMM couldn’t resolve my PostgreSQL container:

text 
lookup postgres-server on 127.0.0.11:53: no such host

The problem wasn’t PostgreSQL or PMM. The PostgreSQL container was stopped, so it had no active network endpoint, and Docker’s internal DNS had nothing to return.

That was an important reminder: always verify the container and network state before troubleshooting the application.

2. Making Monitoring Persistent with PMM

Once the databases were running, I wanted visibility into their performance.

PMM provides dashboards and metrics for MySQL, PXC, and PostgreSQL. It consists of a server that stores and visualizes monitoring data and a client agent that collects database metrics.

The key thing to understand is that persistent storage and service registration are related, but they are not the same thing.

PMM Server maintains the monitoring inventory, while PMM Client has its own configuration and agent identity. If the client is recreated with a new identity, the association between the client and its registered services is lost. In my early setup, after recreating containers, the client listing showed only the basic agents:

text 
pmm_agent Connected
node_exporter Running
vmagent Running

Two things prevent this, and both should be in place from the start:

  1. A persistent volume for the client configuration. In the Compose file above, pmm-client-data keeps the agent identity across restarts and recreations.
  2. An idempotent registration script. It adds each database only if it isn’t already registered, so it is safe to run after every docker compose up.

Before registering, create a monitoring user in each database. For PostgreSQL, for example:

sql 
CREATE USER pmm WITH PASSWORD 'change-me';
GRANT pg_monitor TO pmm;
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

Then the registration script, which reads credentials from the .env file rather than hardcoding them:

bash 
#!/usr/bin/env bash
# register-pmm.sh
set -euo pipefail
source .env

registered() {
 docker exec pmm-client pmm-admin list | grep -qw "$1"
}

if ! registered postgres-server; then
 docker exec pmm-client pmm-admin add postgresql \
 --username=pmm \
 --password="$PG_MONITOR_PASSWORD" \
 --host=postgres-server \
 --port=5432 \
 postgres-server
fi

for node in percona-server pxc1 pxc2 pxc3; do
 if ! registered "$node"; then
 docker exec pmm-client pmm-admin add mysql \
 --username=pmm \
 --password="$MYSQL_MONITOR_PASSWORD" \
 --host="$node" \
 --port=3306 \
 --query-source=perfschema \
 "$node"
 fi
done

docker exec pmm-client pmm-admin list

The takeaway: a monitoring system should be designed to survive a restart, not just work immediately after installation.

3. Learning PostgreSQL Through Real Workloads

With PostgreSQL running and connected to PMM, I wanted to do more than look at an idle dashboard.

I used sysbench to generate a workload against PostgreSQL 17.

My test environment included ten tables with approximately 750,000 rows each, giving me about 7.5 million rows to experiment with.

A sample workload command looked like this:

bash 
sysbench \
 --db-driver=pgsql \
 --pgsql-host=127.0.0.1 \
 --pgsql-port=5432 \
 --pgsql-user=postgres \
 --pgsql-db=sbtest \
 --tables=10 \
 --table-size=750000 \
 --threads=16 \
 --time=300 \
 --report-interval=5 \
 oltp_read_write run

With the workload running, PMM became much more interesting.

I could observe database activity, disk latency, query execution statistics, and changes in resource utilization.

This also helped me understand an important distinction: PostgreSQL tuple activity is not the same thing as SQL queries per second.

One query can read or return many tuples. Looking only at tuple counts without understanding the underlying queries can lead to incorrect conclusions about workload intensity.

4. Understanding PostgreSQL Query Performance

One of the most useful PostgreSQL features for troubleshooting is EXPLAIN.

For example:

sql 
EXPLAIN (ANALYZE, BUFFERS)
SELECT *
FROM sbtest1
WHERE id BETWEEN 100 AND 200;

This shows the execution plan, actual execution measurements, and buffer activity.

I started paying attention to several key pieces of information:

  • Whether PostgreSQL used a sequential scan or an index
  • Estimated versus actual rows
  • Execution time
  • Buffer hits and reads
  • Number of execution loops

Because the Compose file loads pg_stat_statements at startup, query statistics are aggregated from the beginning.

A useful query is:

sql 
SELECT
 calls,
 total_exec_time,
 mean_exec_time,
 rows,
 LEFT(query, 100) AS query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

This provides a different perspective from looking at individual slow queries.

A query that takes only a fraction of a millisecond might still consume substantial cumulative execution time if it runs hundreds of thousands of times.

For a DBA, both individual latency and aggregate workload matter.

5. MVCC, Dead Tuples, and VACUUM

My next learning objective was understanding PostgreSQL’s Multi-Version Concurrency Control (MVCC).

Unlike an in-place overwrite model, PostgreSQL commonly creates a new row version when a row is updated.

Older versions may remain until PostgreSQL determines they are no longer needed.

That leads directly to one of PostgreSQL’s most important maintenance mechanisms: VACUUM.

I experimented with updating large numbers of rows:

sql 
UPDATE sbtest1
SET k = k + 1
WHERE id  300000;

Then checked the table statistics:

sql 
SELECT
 relname,
 n_live_tup,
 n_dead_tup,
 last_autovacuum,
 autovacuum_count
FROM pg_stat_user_tables
WHERE relname = 'sbtest1';

This gave me a practical way to observe dead tuple estimates and autovacuum activity.

I also explored what happens when a long-running transaction holds an old snapshot.

By starting a REPEATABLE READ transaction and leaving it open while another session updated rows, I could demonstrate how PostgreSQL may need to retain older row versions.

That helped connect several important DBA concepts:

Long-running transactions → old snapshots → delayed tuple cleanup → potential table bloat.

I also learned that normal VACUUM generally makes space reusable within PostgreSQL rather than immediately shrinking the table file.

VACUUM FULL can reclaim disk space by rewriting a table, but it requires an exclusive lock and is not something to run casually in production.

6. Operating the PXC Cluster Safely

My three-node Percona XtraDB Cluster taught me that database containers need database-aware operational procedures. That’s why startup and shutdown are built into the Compose file rather than left to chance.

Early on, I ran into certificate problems, donor timeouts, cluster membership issues, and shutdowns that ended with exit code 137 (the container being killed before MySQL finished). The shared certificate directory, the healthcheck on pxc1, and stop_grace_period address those directly.

The shutdown procedure is simply:

bash 
docker compose stop pxc3 pxc2 pxc1

If you use Option B, do the same thing explicitly, one node at a time, with a long timeout:

bash 
docker stop -t 120 pxc3
docker stop -t 120 pxc2
docker stop -t 120 pxc1

To start the cluster again, check grastate.dat on each node and understand the meaning of safe_to_bootstrap:

bash 
docker run --rm -v pxc1_data:/data alpine cat /data/grastate.dat

After a clean shutdown, the last node to leave the cluster (here, pxc1) will have safe_to_bootstrap: 1 and is the right node to start first.

After an unclean shutdown, selecting the correct bootstrap node requires recovery checks. Compare the recovered sequence numbers across nodes and bootstrap from the most advanced one. Simply changing safe_to_bootstrap to 1 without verifying that can lose data.

7. My Most Useful Commands

I kept a command cheat sheet from the start of the project, and it has become one of the most useful outcomes. It captures the commands I actually needed while learning and troubleshooting. I recommend starting one on day one.

For Docker:

bash 
# Running containers
docker ps

# All containers
docker ps -a

# Status of the Compose-managed lab
docker compose ps

# Follow logs
docker logs -f postgres-server

# Inspect network configuration
docker inspect postgres-server \
 --format '{{json .NetworkSettings.Networks}}'

# Resource usage
docker stats

# List Docker volumes
docker volume ls

# Check PMM services
docker exec pmm-client pmm-admin list

These commands are my first line of investigation whenever something isn’t behaving as expected.

For PostgreSQL, my daily reference includes:

sql 
-- Current sessions
SELECT pid, usename, state, query
FROM pg_stat_activity;

-- Long-running transactions
SELECT
 pid,
 state,
 now() - xact_start AS transaction_age,
 query
FROM pg_stat_activity
WHERE xact_start IS NOT NULL;

-- Dead tuples
SELECT relname, n_live_tup, n_dead_tup
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC;

-- Blocking sessions
SELECT
 pid,
 pg_blocking_pids(pid) AS blocked_by
FROM pg_stat_activity
WHERE cardinality(pg_blocking_pids(pid)) > 0;

These are simple queries, but they form a useful starting point for troubleshooting.

8. Design Principles Behind the Lab

To summarize, these are the principles the setup above is built on:

  1. Define the whole lab in Docker Compose, rather than mixing docker run commands with Compose-managed containers.
  2. Declare persistent volumes and network attachments before creating any databases.
  3. Treat startup and shutdown as part of the design, especially for PXC.
  4. Make PMM registration reproducible with a persistent client volume and an idempotent script, and keep credentials out of scripts and command history.
  5. Keep a command cheat sheet from the beginning.

Final Thoughts

What began as an effort to build a small local database environment turned into a practical DBA learning experience.

I now have a place to explore PostgreSQL performance, MVCC, autovacuum, MySQL replication concepts, PXC clustering, and database monitoring without risking a production system.

Perhaps the biggest lesson is that the most valuable learning moments often happen when something breaks.

A Docker DNS failure teaches networking. A missing PMM registration teaches persistence and identity. A PXC bootstrap problem teaches cluster recovery. A growing dead tuple count teaches PostgreSQL MVCC.

And once you understand why something broke, you’re much better prepared to troubleshoot it the next time.

My next goal is to continue expanding the lab with PostgreSQL replication, backup and recovery, and more deliberate performance troubleshooting exercises.

This article describes a local learning environment. The commands and configurations are intended for experimentation and should be reviewed and adapted before use in production.

]]> Mon, 12 Oct 2026 00:00:00 +0000 Percona Community OpenSource Percona MySQL Docker PostgreSQL PMM PXC SecureBridge 12.1, EntityDAC 4.1, and dbExpress drivers: Support for RAD Studio 13 Florence Release 2, SQL Server 2025, and MySQL 9.7 https://www.devart.com/blog/?p=184288 https://www.devart.com/blog/securebridge-12-1-entitydac-4-1-and-dbexpress-drivers-support-for-rad-studio-13-florence-release-2-sql-server-2025-and-mysql-9-7.html New versions of SecureBridge, EntityDAC, and dbExpress drivers are out. All three now work with RAD Studio 13 Florence Release 2 and offer improved support for the Arm64EC target platform. The dbExpress drivers also add support for Microsoft SQL Server 2025 and MySQL 9.7, while SecureBridge gains SFTP upload resume, multipart/form-data support, and a new SSL/TLS demo. The post SecureBridge 12.1, EntityDAC 4.1, and dbExpress drivers: Support for RAD Studio 13 Florence Release 2, SQL Server 2025, and MySQL 9.7 appeared first on Devart Blog. New versions of SecureBridge, EntityDAC, and dbExpress drivers are out. All three now work with RAD Studio 13 Florence Release 2 and offer improved support for the Arm64EC target platform. The dbExpress drivers also add support for Microsoft SQL Server 2025 and MySQL 9.7, while SecureBridge gains SFTP upload resume, multipart/form-data support, and a new SSL/TLS demo.

The post SecureBridge 12.1, EntityDAC 4.1, and dbExpress drivers: Support for RAD Studio 13 Florence Release 2, SQL Server 2025, and MySQL 9.7 appeared first on Devart Blog.

]]> Fri, 09 Oct 2026 17:00:52 +0000 Alena Subotina ORM Solutions Productivity Tools What’s New dbexpress dbExpress driver entitydac RAD Studio 13 Florence securebridge PGO or not PGO this is the dilemma. Step 3 https://www.percona.com/?p=54438 https://www.percona.com/blog/pgo-or-not-pgo-this-is-the-dilemma-step-3/ Step 3: Why my sysbench-trained build loses and how to do it right. Given the topic complexity and the length of this article I have split it in 3 three different blog-post: What is PGO How PGO it works Why my sysbench-trained build loses and how to do it right. Three compounding reasons: Uncovered code gets pessimized. Sysbench-tpcc touches a narrow slice of mysqld. Every function with zero counts is treated as cold: GCC optimizes it for size, skips inlining, and shoves it into cold sections. But at runtime I still execute plenty of code my training never touched, such as purge, flushing, stats recalculation, error paths, different optimizer plans, connection churn. All of that is now running de-optimized code, and I pay icache penalties every time hot code calls into “cold” regions. This is why GCC added -fprofile-partial-training, without it, a narrow profile actively hurts everything outside it. High-concurrency instrumented runs produce corrupted or skewed profiles. GCC’s profile counters are non-atomic by default. With 128–1024 threads hammering the same counters I get lost updates and internally inconsistent counts (I actually had “profile count data file corrupted/inconsistent” warnings at the -fprofile-use compile). I need to use -fprofile-update=atomic, which almost nobody sets and was at the beginning not aware of. In short my profile was garbage and a garbage profile is worse than no profile, consistent with my PGO build being slower than plain -O3 at 128 threads. Instrumentation distorts what “hot” means under contention. The instrumented binary is 2–10x slower, which shifts where threads pile up. Spin loops in mutexes and rw-locks record enormous counts, so the compiler lavishes optimization on waiting code instead of useful work. While at high thread counts my real bottleneck is lock contention and memory latency things branch layout can’t fix. Do I have a way to merge the different profiles like MTR + sysbench? The answer is yes. For GCC it’s simple to do so, because the runtime automatically merges profile data across multiple training runs against the same instrumented binary. I don’t need a separate merge step like Clang does. What it does is that each time an instrumented binary exits, it writes its counters into files. If those files already exist (from a previous run), GCC’s runtime adds the new counts to the existing ones rather than overwriting them. So if I run MTR first, then run sysbench against the same instrumented build with the same FPROFILE_DIR, the second run’s counts accumulate on top of the first. The final profile reflects both workloads combined.  Combining MTR with a moderate-thread sysbench run for PGO training is helpful because the two workloads cover different, complementary dimensions of mysqld’s behavior:  MTR sweeps broad functional breadth parser, optimizer, DDL, replication, error paths. However it runs almost entirely single-connection, so it never exercises the branches that only exist under real concurrency, like the contended slow-path of a latch, MVCC visibility checks against in-flight writers, lock-wait queuing, or redo-log group-commit batching; A moderate-concurrency sysbench run (something like 4–64 threads, enough to create actual simultaneous access without descending into the timing-distortion and counter-corruption problems), fills exactly that gap by giving those concurrency-only branches nonzero execution counts, which keeps the compiler from treating them as cold, since cold code paths get optimized for size instead of speed. So the merged profile ends up with both the wide code coverage MTR provides and the concurrent-path coverage MTR structurally can’t, at the cost of only a modest, second-order improvement over MTR alone since PGO’s overall gains are already small and this specific slice of the binary is a narrow fraction of total execution.   However even where it does help, I am stacking a small effect on top of a small effect. I have already found PGO vs non-PGO gives me under 5%. The incremental gain from better-covering a narrow slice of concurrency-only code within that is a second-order refinement. It is plausibly in the sub-1% range, quite  smaller than the run-to-run noise I had seen just from benchmark variance. Or at least that is what I now think, let me validate it. Commands to build the code:cmake ../mysql-9.7.2 \ -DCMAKE_INSTALL_PREFIX=/opt/mysql_templates/mysql-9.7.2-PGO-instrument \ -DCMAKE_BUILD_TYPE=Release \ -DENABLED_LOCAL_INFILE=1 \ -DWITH_FEDERATED_STORAGE_ENGINE=1 \ -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ -DWITH_PACKAGE_FLAGS=OFF \ -DCOMPILATION_COMMENT_SERVER="Marco compile 9.7.2-PGO instrument" \ -DCOMPILATION_COMMENT="Marco compile 9.7.2 PGO instrument" \ -DCMAKE_C_COMPILER=clang-20 \ -DCMAKE_CXX_COMPILER=clang++-20 \ -DCMAKE_C_FLAGS="-fuse-ld=lld" \ -DCMAKE_CXX_FLAGS="-fuse-ld=lld" \ -DFPROFILE_GENERATE=ON \ -DWITH_LTO=OFF \ -DFPROFILE_DIR=/opt/mysql_source/profileRun the mtr: perl mysql-test-run.pl --force --max-test-fail=0 --parallel=8 --suite=main,innodb,innodb_undo,binlog,rpl,perfschema,sys_vars  Then run the sysbench-tpcc test with 64 threadssysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307  --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1  --histogram --report_csv=yes  --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=600 --threads=64 --mysql-ssl=PREFERRED --mysql-ignore-errors=none  --reconnect=0  runAnd got the profile as[root@sm-blade03 profile]# llvm-profdata-20 show -detailed-summary /opt/mysql_source/profile/mysql.profdata /opt/mysql_source/profile/mysql.profdata Instrumentation level: IR entry_first = 0 instrument_loop_entries = 0 Total functions: 52403 Maximum function count: 289910820864 Maximum internal block count: 16881288068 Total number of blocks: 619050 Total count: 2473216025729 Detailed summary: 2 blocks (0.00%) with count &gt;= 289910820864 account for 1% of the total counts. 2 blocks (0.00%) with count &gt;= 289910820864 account for 10% of the total counts. 2 blocks (0.00%) with count &gt;= 289910820864 account for 20% of the total counts. 11 blocks (0.00%) with count &gt;= 16777125216 account for 30% of the total counts. 32 blocks (0.01%) with count &gt;= 7806039615 account for 40% of the total counts. 77 blocks (0.01%) with count &gt;= 3839409465 account for 50% of the total counts. 160 blocks (0.03%) with count &gt;= 2273821592 account for 60% of the total counts. 312 blocks (0.05%) with count &gt;= 1111890303 account for 70% of the total counts. 719 blocks (0.12%) with count &gt;= 365989756 account for 80% of the total counts. 2165 blocks (0.35%) with count &gt;= 85858392 account for 90% of the total counts. 4893 blocks (0.79%) with count &gt;= 24359463 account for 95% of the total counts. 16760 blocks (2.71%) with count &gt;= 2841200 account for 99% of the total counts. 45006 blocks (7.27%) with count &gt;= 148881 account for 99.9% of the total counts. 96619 blocks (15.61%) with count &gt;= 8640 account for 99.99% of the total counts. 157023 blocks (25.37%) with count &gt;= 960 account for 99.999% of the total counts. 220001 blocks (35.54%) with count &gt;= 91 account for 99.9999% of the total countsChecking on how counts concentrate: just 2 blocks account for 20% of all executed instructions across the entire training run (almost certainly a tight InnoDB buffer-pool/redo-log/lock_manager loop  counts in the hundreds of billions), while 90% of total execution volume is concentrated in only 2,165 blocks (0.35% of all covered blocks).  That’s the “hot core” PGO is designed to find and optimize aggressively. Meanwhile the long tail,  the other 96%+ of blocks, still has nonzero counts (this is a sparse profile, so anything appearing here was actually executed at least once). Meaning a huge amount of MTR’s functional-path breadth got captured even though it’s numerically dwarfed by sysbench’s tight hot loops. This is the ideal shape for a merged profile: a small, extremely hot core (from sustained sysbench load) sitting on top of broad, low-frequency-but-nonzero coverage (from MTR’s functional sweep).  If MTR had contributed nothing, I would have a much flatter, narrower distribution with far fewer total functions covered. If sysbench had swamped everything with no MTR contribution, I would have a similar shape but with a much smaller “Total functions” number, since sysbench-tpcc only touches a fraction of mysqld’s total surface. Command to build the final optimized binaries:llvm-profdata-20 merge -sparse /opt/mysql_source/profile/*.profraw -o /opt/mysql_source/profile/mysql.profdata cmake ../mysql-9.7.2 \ -DCMAKE_INSTALL_PREFIX=/opt/mysql_templates/mysql-9.7.2-PGO-optimized-MTR-sysbench \ -DCMAKE_BUILD_TYPE=Release \ -DENABLED_LOCAL_INFILE=1 \ -DWITH_FEDERATED_STORAGE_ENGINE=1 \ -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ -DWITH_PACKAGE_FLAGS=OFF \ -DCOMPILATION_COMMENT_SERVER="Marco compile 9.7.2-PGO optimized MTR+sysbench" \ -DCOMPILATION_COMMENT="Marco compile 9.7.2 PGO optimized MTR+sysbench" \ -DCMAKE_C_COMPILER=clang-20 \ -DCMAKE_CXX_COMPILER=clang++-20 \ -DFPROFILE_USE=ON \ -DFPROFILE_DIR=/opt/mysql_source/profile/mysql.profdata \ -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_LZ4=system -DWITH_ICU=system \ -DWITH_NUMA=ON -DWITH_LTO=ON -DWITH_LD=lld -DWITH_SYSTEMD=1 \ -DWITH_UNIT_TESTS=OFF -DWITH_ROUTER=OFF -DMYSQL_MAINTAINER_MODE=OFFRe-running the test I got this:                 As expected the benefit I got was minimal, something was there but that disappeared while the concurrency increased.   Conclusions Non-PGO vs PGO comparison: we had a 12% increase with low concurrency. But when simulating a more realistic load with higher contention the win becomes smaller and smaller, under 2%. Training-workload choice mattered a lot: a sysbench-tpcc-trained PGO build ended up slower than a non-PGO. After switching to a merged MTR + moderate-concurrency-sysbench training profile and re-running, the gain was still minimal, and it shrank further as concurrency increased.   Why the sysbench-only build lost Sysbench-tpcc only exercises a narrow slice of mysqld, so everything outside that slice (purge, flushing, stats, error paths, alternate optimizer plans) gets pessimized as “cold” code. At 128–1024 threads, GCC’s non-atomic profile counters get corrupted under contention unless I explicitly set -fprofile-update=atomic,a garbage profile is worse than no profile at all. Instrumentation overhead (2–10x slowdown) distorts what looks “hot” under load mutex/rwlock spin loops dominate the counts, so the compiler optimizes waiting code instead of real work, while the actual bottleneck (lock contention, memory latency) is something branch layout can’t fix anyway. Bottom line: PGO for MySQL/Percona Server binaries does work in the sense that the mechanism is sound whole-binary function reordering for something the size of mysqld is a legitimate, often the single biggest, win in PGO generally.  But empirically here the payoff was consistently small (sub-5%, trending toward sub-1% for the concurrency-specific refinement), fragile to training-workload choice, fragile to build-flag correctness (atomic counters, -fprofile-partial-training), and it erodes further as thread count rises,  which is exactly what happens in production and what we care about most. So my practical answer: it’s not a clear “yes, always compile with PGO.” It’s more a “maybe, and only if you get every detail right“. Correct broad-coverage training data (MTR, ideally merged with moderate-concurrency sysbench), atomic profile counters, and realistic expectations that the win is marginal and shrinks under heavy concurrency.  Get any of those wrong and you can end up worse than a plain build. Given the size of the benefit versus the number of ways to mess up the training methodology, PGO reads more like a niche optimization for a well-controlled build pipeline than a default you’d flip on broadly. I would be more than happy to prove wrong and I am eager to get other people’s feedback, so please test, test, test and let me know.    Happy MySQL to everyone Go to: What is PGO How PGO it works The post PGO or not PGO this is the dilemma. Step 3 appeared first on Percona. Step 3: Why my sysbench-trained build loses and how to do it right.

Given the topic complexity and the length of this article I have split it in 3 three different blog-post:

  1. What is PGO
  2. How PGO it works
  3. Why my sysbench-trained build loses and how to do it right.

Three compounding reasons:

  1. Uncovered code gets pessimized.
    Sysbench-tpcc touches a narrow slice of mysqld.
    Every function with zero counts is treated as cold: GCC optimizes it for size, skips inlining, and shoves it into cold sections.
    But at runtime I still execute plenty of code my training never touched, such as purge, flushing, stats recalculation, error paths, different optimizer plans, connection churn.
    All of that is now running de-optimized code, and I pay icache penalties every time hot code calls into “cold” regions.
    This is why GCC added -fprofile-partial-training, without it, a narrow profile actively hurts everything outside it.
  2. High-concurrency instrumented runs produce corrupted or skewed profiles.
    GCC’s profile counters are non-atomic by default.
    With 128–1024 threads hammering the same counters I get lost updates and internally inconsistent counts (I actually had “profile count data file corrupted/inconsistent” warnings at the -fprofile-use compile).
    I need to use -fprofile-update=atomic, which almost nobody sets and was at the beginning not aware of.
    In short my profile was garbage and a garbage profile is worse than no profile, consistent with my PGO build being slower than plain -O3 at 128 threads.
  3. Instrumentation distorts what “hot” means under contention.
    The instrumented binary is 2–10x slower, which shifts where threads pile up. Spin loops in mutexes and rw-locks record enormous counts, so the compiler lavishes optimization on waiting code instead of useful work. While at high thread counts my real bottleneck is lock contention and memory latency things branch layout can’t fix.

Do I have a way to merge the different profiles like MTR + sysbench?
The answer is yes. For GCC it’s simple to do so, because the runtime automatically merges profile data across multiple training runs against the same instrumented binary. I don’t need a separate merge step like Clang does.

What it does is that each time an instrumented binary exits, it writes its counters into files. If those files already exist (from a previous run), GCC’s runtime adds the new counts to the existing ones rather than overwriting them.
So if I run MTR first, then run sysbench against the same instrumented build with the same FPROFILE_DIR, the second run’s counts accumulate on top of the first. The final profile reflects both workloads combined. 

Combining MTR with a moderate-thread sysbench run for PGO training is helpful because the two workloads cover different, complementary dimensions of mysqld’s behavior: 

  • MTR sweeps broad functional breadth parser, optimizer, DDL, replication, error paths. However it runs almost entirely single-connection, so it never exercises the branches that only exist under real concurrency, like the contended slow-path of a latch, MVCC visibility checks against in-flight writers, lock-wait queuing, or redo-log group-commit batching;
  • A moderate-concurrency sysbench run (something like 4–64 threads, enough to create actual simultaneous access without descending into the timing-distortion and counter-corruption problems), fills exactly that gap by giving those concurrency-only branches nonzero execution counts, which keeps the compiler from treating them as cold, since cold code paths get optimized for size instead of speed.

So the merged profile ends up with both the wide code coverage MTR provides and the concurrent-path coverage MTR structurally can’t, at the cost of only a modest, second-order improvement over MTR alone since PGO’s overall gains are already small and this specific slice of the binary is a narrow fraction of total execution.

 

However even where it does help, I am stacking a small effect on top of a small effect. I have already found PGO vs non-PGO gives me under 5%.
The incremental gain from better-covering a narrow slice of concurrency-only code within that is a second-order refinement.
It is plausibly in the sub-1% range, quite  smaller than the run-to-run noise I had seen just from benchmark variance.

Or at least that is what I now think, let me validate it.

Commands to build the code:

cmake ../mysql-9.7.2 \
-DCMAKE_INSTALL_PREFIX=/opt/mysql_templates/mysql-9.7.2-PGO-instrument \
-DCMAKE_BUILD_TYPE=Release \
-DENABLED_LOCAL_INFILE=1 \
-DWITH_FEDERATED_STORAGE_ENGINE=1 \
-DWITH_ARCHIVE_STORAGE_ENGINE=1 \
-DWITH_PACKAGE_FLAGS=OFF \
-DCOMPILATION_COMMENT_SERVER="Marco compile 9.7.2-PGO instrument" \
-DCOMPILATION_COMMENT="Marco compile 9.7.2 PGO instrument" \
-DCMAKE_C_COMPILER=clang-20 \
-DCMAKE_CXX_COMPILER=clang++-20 \
-DCMAKE_C_FLAGS="-fuse-ld=lld" \
-DCMAKE_CXX_FLAGS="-fuse-ld=lld" \
-DFPROFILE_GENERATE=ON \
-DWITH_LTO=OFF \
-DFPROFILE_DIR=/opt/mysql_source/profile

Run the mtr:

perl mysql-test-run.pl --force --max-test-fail=0 --parallel=8 --suite=main,innodb,innodb_undo,binlog,rpl,perfschema,sys_vars

 

Then run the sysbench-tpcc test with 64 threads

sysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307  --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1  --histogram --report_csv=yes  --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=600 --threads=64 --mysql-ssl=PREFERRED --mysql-ignore-errors=none  --reconnect=0  run

And got the profile as

[root@sm-blade03 profile]# llvm-profdata-20 show -detailed-summary /opt/mysql_source/profile/mysql.profdata
/opt/mysql_source/profile/mysql.profdata
Instrumentation level: IR entry_first = 0 instrument_loop_entries = 0
Total functions: 52403
Maximum function count: 289910820864
Maximum internal block count: 16881288068
Total number of blocks: 619050
Total count: 2473216025729
Detailed summary:
2 blocks (0.00%) with count >= 289910820864 account for 1% of the total counts.
2 blocks (0.00%) with count >= 289910820864 account for 10% of the total counts.
2 blocks (0.00%) with count >= 289910820864 account for 20% of the total counts.
11 blocks (0.00%) with count >= 16777125216 account for 30% of the total counts.
32 blocks (0.01%) with count >= 7806039615 account for 40% of the total counts.
77 blocks (0.01%) with count >= 3839409465 account for 50% of the total counts.
160 blocks (0.03%) with count >= 2273821592 account for 60% of the total counts.
312 blocks (0.05%) with count >= 1111890303 account for 70% of the total counts.
719 blocks (0.12%) with count >= 365989756 account for 80% of the total counts.
2165 blocks (0.35%) with count >= 85858392 account for 90% of the total counts.
4893 blocks (0.79%) with count >= 24359463 account for 95% of the total counts.
16760 blocks (2.71%) with count >= 2841200 account for 99% of the total counts.
45006 blocks (7.27%) with count >= 148881 account for 99.9% of the total counts.
96619 blocks (15.61%) with count >= 8640 account for 99.99% of the total counts.
157023 blocks (25.37%) with count >= 960 account for 99.999% of the total counts.
220001 blocks (35.54%) with count >= 91 account for 99.9999% of the total counts

Checking on how counts concentrate: just 2 blocks account for 20% of all executed instructions across the entire training run (almost certainly a tight InnoDB buffer-pool/redo-log/lock_manager loop  counts in the hundreds of billions), while 90% of total execution volume is concentrated in only 2,165 blocks (0.35% of all covered blocks). 

That’s the “hot core” PGO is designed to find and optimize aggressively. Meanwhile the long tail,  the other 96%+ of blocks, still has nonzero counts (this is a sparse profile, so anything appearing here was actually executed at least once).

Meaning a huge amount of MTR’s functional-path breadth got captured even though it’s numerically dwarfed by sysbench’s tight hot loops.

This is the ideal shape for a merged profile: a small, extremely hot core (from sustained sysbench load) sitting on top of broad, low-frequency-but-nonzero coverage (from MTR’s functional sweep). 

If MTR had contributed nothing, I would have a much flatter, narrower distribution with far fewer total functions covered.
 If sysbench had swamped everything with no MTR contribution, I would have a similar shape but with a much smaller “Total functions” number, since sysbench-tpcc only touches a fraction of mysqld’s total surface.

Command to build the final optimized binaries:

llvm-profdata-20 merge -sparse /opt/mysql_source/profile/*.profraw -o /opt/mysql_source/profile/mysql.profdata

cmake ../mysql-9.7.2 \
 -DCMAKE_INSTALL_PREFIX=/opt/mysql_templates/mysql-9.7.2-PGO-optimized-MTR-sysbench \
  -DCMAKE_BUILD_TYPE=Release \
  -DENABLED_LOCAL_INFILE=1 \
  -DWITH_FEDERATED_STORAGE_ENGINE=1 \
  -DWITH_ARCHIVE_STORAGE_ENGINE=1 \
  -DWITH_PACKAGE_FLAGS=OFF \
  -DCOMPILATION_COMMENT_SERVER="Marco compile 9.7.2-PGO optimized MTR+sysbench" \
  -DCOMPILATION_COMMENT="Marco compile 9.7.2 PGO optimized MTR+sysbench" \
  -DCMAKE_C_COMPILER=clang-20 \
  -DCMAKE_CXX_COMPILER=clang++-20 \
  -DFPROFILE_USE=ON \
  -DFPROFILE_DIR=/opt/mysql_source/profile/mysql.profdata \
  -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_LZ4=system -DWITH_ICU=system \
  -DWITH_NUMA=ON -DWITH_LTO=ON -DWITH_LD=lld -DWITH_SYSTEMD=1 \
  -DWITH_UNIT_TESTS=OFF -DWITH_ROUTER=OFF -DMYSQL_MAINTAINER_MODE=OFF

Re-running the test I got this:

 

 

 

 

 

 

 

 

As expected the benefit I got was minimal, something was there but that disappeared while the concurrency increased.

 

Conclusions

  • Non-PGO vs PGO comparison: we had a 12% increase with low concurrency. But when simulating a more realistic load with higher contention the win becomes smaller and smaller, under 2%.
  • Training-workload choice mattered a lot: a sysbench-tpcc-trained PGO build ended up slower than a non-PGO.
  • After switching to a merged MTR + moderate-concurrency-sysbench training profile and re-running, the gain was still minimal, and it shrank further as concurrency increased.

 

Why the sysbench-only build lost

Sysbench-tpcc only exercises a narrow slice of mysqld, so everything outside that slice (purge, flushing, stats, error paths, alternate optimizer plans) gets pessimized as “cold” code.

At 128–1024 threads, GCC’s non-atomic profile counters get corrupted under contention unless I explicitly set -fprofile-update=atomic,a garbage profile is worse than no profile at all.

Instrumentation overhead (2–10x slowdown) distorts what looks “hot” under load mutex/rwlock spin loops dominate the counts, so the compiler optimizes waiting code instead of real work, while the actual bottleneck (lock contention, memory latency) is something branch layout can’t fix anyway.

Bottom line: PGO for MySQL/Percona Server binaries does work in the sense that the mechanism is sound whole-binary function reordering for something the size of mysqld is a legitimate, often the single biggest, win in PGO generally. 

But empirically here the payoff was consistently small (sub-5%, trending toward sub-1% for the concurrency-specific refinement), fragile to training-workload choice, fragile to build-flag correctness (atomic counters, -fprofile-partial-training), and it erodes further as thread count rises,  which is exactly what happens in production and what we care about most.

So my practical answer: it’s not a clear “yes, always compile with PGO.” It’s more a “maybe, and only if you get every detail right“. Correct broad-coverage training data (MTR, ideally merged with moderate-concurrency sysbench), atomic profile counters, and realistic expectations that the win is marginal and shrinks under heavy concurrency. 

Get any of those wrong and you can end up worse than a plain build.
Given the size of the benefit versus the number of ways to mess up the training methodology, PGO reads more like a niche optimization for a well-controlled build pipeline than a default you’d flip on broadly.

I would be more than happy to prove wrong and I am eager to get other people’s feedback, so please test, test, test and let me know. 

 

Happy MySQL to everyone

Go to:

What is PGO

How PGO it works

The post PGO or not PGO this is the dilemma. Step 3 appeared first on Percona.

]]> Fri, 09 Oct 2026 11:47:55 +0000 Marco Tusa Insight for DBAs Insight for Developers MySQL Benchmarking Compiler Optimization MySQL Performance Percona Server for MySQL PGO PGO or not PGO this is the dilemma. Step 2 https://www.percona.com/?p=54435 https://www.percona.com/blog/pgo-or-not-pgo-this-is-the-dilemma-step-2/ Step 2: How it works Given the topic complexity and the length of this article I have split it in 3 three different blog-post: What is PGO How PGO it works Why my sysbench-trained build loses and how to do it right. How PGO works: PGO is a two-pass build. First pass compiles with instrumentation (-fprofile-generate): every basic block and branch gets a counter. You run a training workload, counters are dumped to profraw files. Second pass recompiles using those counts to drive inlining decisions, branch layout (hot path falls through, cold path jumps away), hot/cold function splitting, code ordering for icache/iTLB locality, loop unrolling, and indirect-call promotion. Crucially, PGO is not “make the trained workload fast” it’s “tell the compiler which code is hot and which is cold, and let it reshape the whole binary accordingly.”   But how does it work? Phase 1: what actually gets recorded. The instrumented binary has counters injected at compile time one per edge in the control-flow graph, not just per function. So for every branch, every loop back-edge, every call site, there’s a counter that increments each time execution takes that path. This is finer-grained than “function X was called N times” it’s “when we reached this branch, we went left 950,000 times and right 50 times.” That per-edge granularity is what lets the second phase make surgical decisions rather than just “function X is hot, function Y is cold.” Phase 2: what the compiler does with those counts. Several distinct transformations, all driven by the same counter data: Inlining. Normally the compiler inlines based on static heuristics: function size call-site count estimated cost/benefit.  With profile data it can override those heuristics: a call site executed millions of times gets inlined even if it looks “too expensive” by static cost rules, because the runtime benefit clearly outweighs the code-size cost.  A call site that’s technically inlinable but sits in dead-cold code gets left as a real call inlining it would only bloat the binary for no benefit. Branch layout. Every “if” in our code compiles down to a branch instruction with two possible outcomes:  continue straight to the next instruction jump somewhere else.  Continuing straight is basically free; the CPU is already fetching instructions in order, so there’s no extra cost. Jumping is not free: the CPU has to guess in advance which way a branch will go so it can keep fetching ahead of time, and if it guesses wrong, it has to throw away the work it already queued up and start over from the right place. That is the “pipeline bubble,” a small stall. So the “straight through” path is cheap and the “jump elsewhere” path carries a penalty when the guess is wrong.  The compiler arranges the hot path as the fall-through and pushes the cold path out of line; literally relocated to a separate location in the binary, often into a .text.unlikely section. So an if (unlikely_error_condition) { …rare handling… } block doesn’t sit inline interrupting the hot path anymore; it is moved somewhere else entirely, and the hot path becomes a straight run of instructions with no diversion. To be clear, reordering our if/else in the source code usually doesn’t change how the compiler lays out the machine code. Optimizing compilers decide branch layout themselves based on either profile data (PGO) or static heuristics, not on which branch we happened to write first in the source. So swapping the order of our if blocks by hand generally has little to no effect on the compiled result. Hot/cold function splitting. This is the same idea applied within a single function. A function might have a hot core loop and a rarely-hit error-handling tail. The compiler physically splits the function into two pieces:  the hot part stays in .text.hot the cold part moves to .text.unlikely.  The function still works identically (a jump connects them when needed), but now the hot part is smaller and denser, so more of it fits in an instruction-cache line, and cold code that’s almost never touched isn’t wasting icache space sitting next to it. Whole-binary function reordering. This is where “reshape the whole binary” becomes literal. At link time (especially with LTO, which MySQL’s PGO build enables), functions get physically reordered in the final executable so that functions which call each other frequently, or execute in sequence during a hot workload, are placed near each other in memory. This maximizes instruction-cache and iTLB locality; the CPU’s fetch unit is pulling in a tight cluster of hot functions instead of jumping all over a 100+MB binary. For something the size of mysqld, this is often the single biggest win, because normal builds place functions in whatever order the source files happen to be compiled, which has no relationship to runtime call patterns. Indirect-call promotion. If profile data shows a virtual call or function-pointer call resolves to the same target the overwhelming majority of the time (common in C++ with vtables, e.g. a storage-engine interface with basically only InnoDB registered), the compiler inserts a guarded direct call: “if target == this specific address, call it directly and skip the indirect jump; otherwise fall back to the indirect call.” Direct calls are cheaper and more predictable for the branch predictor than an indirect jump through a table. Register allocation and code density trade-offs. Hot code gets compiled favoring speed, more aggressive unrolling, more registers dedicated to hot-path values. Cold code, especially with -fprofile-partial-training and cold-path treatment, gets compiled favoring size, fewer registers, less unrolling because it barely executes. So runtime cost there is irrelevant but its footprint in the binary is not free (it still occupies disk/page-cache space and can evict hot lines from cache if placed carelessly, which is exactly why it gets segregated into .text.unlikely rather than just left unoptimized in place). Switch/jump-table lowering. A switch statement with many cases can be compiled as a jump table (fast, O(1), but requires a full table load and indirect jump) or as a cascade of compares (slower per-case but better branch prediction if one case dominates). Profile data tells the compiler which case actually dominates in practice and picks accordingly. Given the above,  “reshape the whole binary” is not metaphorical. The compiler is redrawing the physical layout of machine code in the executable: which instructions are adjacent to which, which code sections are hot and packed tightly versus cold and shoved to the side, which calls are direct versus indirect, and where the CPU’s fetch/prediction effort gets spent. None of this requires the values processed during training to resemble production traffic. It only requires the shape of control flow. Which branches, functions, and paths are frequently exercised to resemble production traffic. That’s the core reason MTR’s broad-but-different-data coverage transfers well: it walks nearly every code path in mysqld even though the actual queries and data are nothing like a TPCC workload, and control-flow shape is exactly what PGO optimizes for.   Why my sysbench-trained build loses and how to do it right. The post PGO or not PGO this is the dilemma. Step 2 appeared first on Percona. Step 2: How it works

Given the topic complexity and the length of this article I have split it in 3 three different blog-post:

  1. What is PGO
  2. How PGO it works
  3. Why my sysbench-trained build loses and how to do it right.

How PGO works: PGO is a two-pass build. First pass compiles with instrumentation (-fprofile-generate): every basic block and branch gets a counter.
You run a training workload, counters are dumped to profraw files.
Second pass recompiles using those counts to drive inlining decisions, branch layout (hot path falls through, cold path jumps away), hot/cold function splitting, code ordering for icache/iTLB locality, loop unrolling, and indirect-call promotion.
Crucially, PGO is not “make the trained workload fast” it’s “tell the compiler which code is hot and which is cold, and let it reshape the whole binary accordingly.”

 

But how does it work?

Phase 1: what actually gets recorded. The instrumented binary has counters injected at compile time one per edge in the control-flow graph, not just per function.
So for every branch, every loop back-edge, every call site, there’s a counter that increments each time execution takes that path.
This is finer-grained than “function X was called N times” it’s “when we reached this branch, we went left 950,000 times and right 50 times.”
That per-edge granularity is what lets the second phase make surgical decisions rather than just “function X is hot, function Y is cold.”

Phase 2: what the compiler does with those counts. Several distinct transformations, all driven by the same counter data:

Inlining.
Normally the compiler inlines based on static heuristics:

  • function size call-site count
  • estimated cost/benefit. 

With profile data it can override those heuristics: a call site executed millions of times gets inlined even if it looks “too expensive” by static cost rules, because the runtime benefit clearly outweighs the code-size cost. 

A call site that’s technically inlinable but sits in dead-cold code gets left as a real call inlining it would only bloat the binary for no benefit.

Branch layout.
Every “if” in our code compiles down to a branch instruction with two possible outcomes: 

  • continue straight to the next instruction
  • jump somewhere else. 

Continuing straight is basically free; the CPU is already fetching instructions in order, so there’s no extra cost.
Jumping is not free: the CPU has to guess in advance which way a branch will go so it can keep fetching ahead of time, and if it guesses wrong, it has to throw away the work it already queued up and start over from the right place.

That is the “pipeline bubble,” a small stall. So the “straight through” path is cheap and the “jump elsewhere” path carries a penalty when the guess is wrong. 

The compiler arranges the hot path as the fall-through and pushes the cold path out of line; literally relocated to a separate location in the binary, often into a .text.unlikely section.
So an if (unlikely_error_condition) { …rare handling… } block doesn’t sit inline interrupting the hot path anymore; it is moved somewhere else entirely, and the hot path becomes a straight run of instructions with no diversion.

To be clear, reordering our if/else in the source code usually doesn’t change how the compiler lays out the machine code. Optimizing compilers decide branch layout themselves based on either profile data (PGO) or static heuristics, not on which branch we happened to write first in the source. So swapping the order of our if blocks by hand generally has little to no effect on the compiled result.

Hot/cold function splitting.
This is the same idea applied within a single function.
A function might have a hot core loop and a rarely-hit error-handling tail.
The compiler physically splits the function into two pieces: 

  • the hot part stays in .text.hot
  • the cold part moves to .text.unlikely. 

The function still works identically (a jump connects them when needed), but now the hot part is smaller and denser, so more of it fits in an instruction-cache line, and cold code that’s almost never touched isn’t wasting icache space sitting next to it.

Whole-binary function reordering.
This is where “reshape the whole binary” becomes literal.
At link time (especially with LTO, which MySQL’s PGO build enables), functions get physically reordered in the final executable so that functions which call each other frequently, or execute in sequence during a hot workload, are placed near each other in memory.
This maximizes instruction-cache and iTLB locality; the CPU’s fetch unit is pulling in a tight cluster of hot functions instead of jumping all over a 100+MB binary.
For something the size of mysqld, this is often the single biggest win, because normal builds place functions in whatever order the source files happen to be compiled, which has no relationship to runtime call patterns.

Indirect-call promotion.
If profile data shows a virtual call or function-pointer call resolves to the same target the overwhelming majority of the time (common in C++ with vtables, e.g. a storage-engine interface with basically only InnoDB registered), the compiler inserts a guarded direct call: “if target == this specific address, call it directly and skip the indirect jump; otherwise fall back to the indirect call.”
Direct calls are cheaper and more predictable for the branch predictor than an indirect jump through a table.

Register allocation and code density trade-offs.
Hot code gets compiled favoring speed, more aggressive unrolling, more registers dedicated to hot-path values.
Cold code, especially with -fprofile-partial-training and cold-path treatment, gets compiled favoring size, fewer registers, less unrolling because it barely executes. So runtime cost there is irrelevant but its footprint in the binary is not free (it still occupies disk/page-cache space and can evict hot lines from cache if placed carelessly, which is exactly why it gets segregated into .text.unlikely rather than just left unoptimized in place).

Switch/jump-table lowering.
A switch statement with many cases can be compiled as a jump table (fast, O(1), but requires a full table load and indirect jump) or as a cascade of compares (slower per-case but better branch prediction if one case dominates).
Profile data tells the compiler which case actually dominates in practice and picks accordingly.

Given the above,  “reshape the whole binary” is not metaphorical. The compiler is redrawing the physical layout of machine code in the executable: which instructions are adjacent to which, which code sections are hot and packed tightly versus cold and shoved to the side, which calls are direct versus indirect, and where the CPU’s fetch/prediction effort gets spent.
None of this requires the values processed during training to resemble production traffic. It only requires the shape of control flow.
Which branches, functions, and paths are frequently exercised to resemble production traffic.
That’s the core reason MTR’s broad-but-different-data coverage transfers well: it walks nearly every code path in mysqld even though the actual queries and data are nothing like a TPCC workload, and control-flow shape is exactly what PGO optimizes for.

 

Why my sysbench-trained build loses and how to do it right.

The post PGO or not PGO this is the dilemma. Step 2 appeared first on Percona.

]]> Fri, 09 Oct 2026 11:47:32 +0000 Marco Tusa Insight for DBAs Insight for Developers MySQL Benchmarking Compiler Optimization MySQL Performance Percona Server for MySQL PGO PGO or not PGO this is the dilemma. Step 1 https://www.percona.com/?p=54428 https://www.percona.com/blog/pgo-or-not-pgo-this-is-the-dilemma-step-1/ Given the topic complexity and the length of this article I have split it in 3 three different blog-post: What is PGO How PGO it works Why my sysbench-trained build loses and how to do it right. Step 1: What is PGO PGO (Profile-Guided Optimization) is a two-pass compilation technique.  First, we build the program with instrumentation that adds counters to every branch, loop, and call site. Next, we run it against a representative training workload so these counters can record which code paths are actually hot versus cold. We then recompile the program using that data to drive the compiler’s decisions. With this profiling data, the compiler inlines hot call sites more aggressively and lays out hot paths as straight-line, fall-through code while pushing cold paths out of the way. Finally, it physically reorders functions in the binary so frequently interacting hot code sits close together for better instruction-cache locality, and converts frequently resolved indirect calls into direct ones. The result isn’t “make the training run fast” but rather “reshape the entire binary’s layout around real execution patterns,” which is why the quality and representativeness of the training workload matters so much to whether PGO actually helps.  It is nice to be wrong Not so far ago I was wondering if having Percona Server compiled with PGO default is a good idea or not. Then I started to do some tests and I end up with this: If I compare non PGO with PGO release I can see an optimization, minimal below 5% but is there. If I compile MySQL using PGO and use as sample the recording of a specific test say sysbench-tpcc my compile will always be slower than my compile without PGO, no mater how many threads I use during recording, I tried from 128 to 1024.                   Given my understanding of PGO was that it should be the other way around I was a bit disoriented. So I decided to read a bit and get a better understanding of what PGO really means/does and if it makes sense or not.   How PGO it works The post PGO or not PGO this is the dilemma. Step 1 appeared first on Percona. Given the topic complexity and the length of this article I have split it in 3 three different blog-post:

  1. What is PGO
  2. How PGO it works
  3. Why my sysbench-trained build loses and how to do it right.

Step 1: What is PGO

PGO (Profile-Guided Optimization) is a two-pass compilation technique. 

First, we build the program with instrumentation that adds counters to every branch, loop, and call site.
Next, we run it against a representative training workload so these counters can record which code paths are actually hot versus cold.
We then recompile the program using that data to drive the compiler’s decisions.

With this profiling data, the compiler inlines hot call sites more aggressively and lays out hot paths as straight-line, fall-through code while pushing cold paths out of the way.
Finally, it physically reorders functions in the binary so frequently interacting hot code sits close together for better instruction-cache locality, and converts frequently resolved indirect calls into direct ones.

The result isn’t “make the training run fast” but rather “reshape the entire binary’s layout around real execution patterns,” which is why the quality and representativeness of the training workload matters so much to whether PGO actually helps. 

It is nice to be wrong

Not so far ago I was wondering if having Percona Server compiled with PGO default is a good idea or not. Then I started to do some tests and I end up with this:

If I compare non PGO with PGO release I can see an optimization, minimal below 5% but is there.

If I compile MySQL using PGO and use as sample the recording of a specific test say sysbench-tpcc my compile will always be slower than my compile without PGO, no mater how many threads I use during recording, I tried from 128 to 1024.

 

 

 

 

 

 

 

 

 

Given my understanding of PGO was that it should be the other way around I was a bit disoriented. So I decided to read a bit and get a better understanding of what PGO really means/does and if it makes sense or not.

 

How PGO it works

The post PGO or not PGO this is the dilemma. Step 1 appeared first on Percona.

]]> Fri, 09 Oct 2026 11:47:08 +0000 Marco Tusa Insight for DBAs Insight for Developers MySQL Benchmarking Compiler Optimization MySQL Performance Percona Server for MySQL PGO Step by Step: ProxySQL HA with BGP ECMP Anycast https://percona.community/blog/2026/10/08/step-by-step-proxysql-ha-with-bgp-ecmp-anycast/ https://percona.community/blog/2026/10/08/step-by-step-proxysql-ha-with-bgp-ecmp-anycast/ In our previous blog post ProxySQL HA with BGP ECMP Anycast, we established why we would want to use BGP ECMP as a strategy for making our ProxySQL cluster highly available. Now let’s look at how to implement it technically. For this scenario we assume that we have: an OPNsense instance as our BGP Router with the IP address 10.5.8.251 Two ProxySQL nodes (proxysql-01 with IP 10.5.20.4 and proxysql-02 with IP 10.5.20.5) 10.5.200.1 as the anycast IP we want both ProxySQL nodes to accept connections on Setup on the router Enable BGP on OPNsense The out-of-the-box installation of OPNsense does not come with BGP support. To add it, install the FRRouting package first (os-frr), which will enable available protocols for dynamic routing. Install the os-frr plugin in System -&gt; Firmware -&gt; Plugins. We need to activate the routing service on our OPNsense host. In Routing -&gt; General check “enable” Next, assign an Autonomous System (AS) number to the OPNsense BGP peer. An AS number identifies a network, or a group of routers, under a single administrative domain. Choose a private AS number from the range 64512-65534. In Routing -&gt; BGP (advanced mode): check “enable”. set the chosen AS number (in this example we use 64512). set Maximum Paths to 2 to match the number of ProxySQL nodes that we have. Maximum Paths configures OPNsense to perform Equal-Cost Multi-Path (ECMP) load balancing across the two paths. Any additional paths are kept as backups and only used if an active route is withdrawn. Create a firewall rule for BGP traffic BGP exchanges routing information through a TCP connection on port 179. Our ProxySQL nodes will establish this connection towards OPNsense, so we need a firewall rule to allow this traffic. First, we will create an alias for the ProxySQL nodes. This keeps the firewall rule readable, and new nodes only need to be added in one place. In Firewall -&gt; Aliases, add: Name: proxysql_nodes Type: Host(s) Content: 10.5.20.4, 10.5.20.5 (The IP addresses of our ProxySQL nodes) With the Alias in place, we can now continue to define the actual Firewall rule: Navigate to Firewall -&gt; Rules and create a new rule: Description: Allow ProxySQL network to BGP peer with OPNsense Interface: Select the interface the ProxySQL nodes are in. Action: Pass Direction: in Version: IPv4 Protocol: TCP Source: proxysql_nodes (the alias we created in the previous step) Source port: any Destination: the OPNsense address the ProxySQL nodes peer with (10.5.8.251) Destination port: BGP (179) Note, here we do not explicitly enable logging. It may be helpful to enable logging that if you need to debug the BGP peering. Set up route filtering with a prefix list We need to set up an inbound filter using a prefix list. The prefix list defines the networks that OPNsense will accept routes for, and should be as narrow as possible to prevent a misbehaving peer from advertising routes that could re-route sensitive traffic through it. Prefix lists are configured in Routing -&gt; BGP -&gt; Prefix Lists. Create the prefix list: Description: ProxySQL Anycast IP Name: ANYCAST-PROXYSQL-IN IP Version: IPV4 Sequence Number: for this example we use 10. Sequence numbers are used in order to set the order with which to apply the rules, in case you would have multiple rules for the same prefix list. We only need to create one entry for the ProxySQL Anycast IP. Action: Permit, as we want to allow the ProxySQL Anycast IP. Network: 10.5.200.1/32, the Anycast IP of our ProxySQL nodes. Tip: Click the ⓘ icon next to a field in the OPNsense UI for more information. Configure BGP neighbors In this step we will configure OPNsense to know about our ProxySQL servers and their intent to peer. Neighbor configuration defines which IPs are allowed to speak the BGP protocol with OPNsense. In our case, we need to define a neighbor for both of our ProxySQL hosts: Choose a unique AS number for the cluster. All ProxySQL nodes belonging to this cluster will share this AS number, whilst separate clusters must use different, dedicated AS numbers. Ensure that the cluster AS number is distinct from the one configured for OPNsense. For this example we choose the AS number 64513. Navigate to Routing -&gt; BGP -&gt; Neighbors and add a new one. Description: ProxySQL-01 Peer-IP: 10.5.20.4 Remote AS: 64513 Local AS: 64512 Prefix-List In: ANYCAST-PROXYSQL-IN:10 Make sure you repeat this for the second ProxySQL. All configurable BGP options can be found in the OPNsense documentation. Setup on the ProxySQL nodes Next, assign the anycast virtual IP (10.5.200.1) to the loopback interface of each ProxySQL node, so that the node accepts traffic intended for that IP address. bash Copy Copied! # Add the Anycast VIP to the loopback interface sudo ip addr add 10.5.200.1/32 dev lo Make the address persistent across reboots by adding it to netplan, systemd-networkd or /etc/network/interfaces. Install ExaBGP ExaBGP is a tool that can speak the BGP protocol with OPNsense and is able to announce / withdraw routes. Install ExaBGP on the ProxySQL hosts with: bash Copy Copied! sudo apt-get update sudo apt-get install exabgp For information on installing ExaBGP on other distributions, you can check the ExaBGP wiki here. Note that BIRD (BIRD Internet Routing Daemon) or FRR can be used as alternative tools to ExaBGP, but in this example we will go with ExaBGP. Create the health check As we only want to route traffic to a ProxySQL host when the ProxySQL process is running, we need a way to tell ExaBGP when to announce the route and when to withdraw it. We will write a health check script that ExaBGP can execute to determine the state of our ProxySQL process. If this health check fails, ExaBGP withdraws the node’s route from OPNsense. OPNsense removes that node as an anycast next hop, and routes the traffic to the remaining healthy nodes. In our example, we will use a simple bash script that uses mysqladmin to send a PING to ProxySQL: bash Copy Copied! #!/usr/bin/env bash ANYCAST_IP="10.5.200.1/32" FAILED=1 while true; do # Execute the health check command and depending on the return code # announce or withdraw the route. mysqladmin --defaults-extra-file=/etc/exabgp/proxysql-monitor.cnf --connect-timeout=2 ping &amp;&gt; /dev/null STATUS=$? if [ $STATUS -eq 0 ]; then if [ $FAILED -ne 0 ]; then # Recovered: Announce route echo "announce route $ANYCAST_IP next-hop self" FAILED=0 fi else if [ $FAILED -eq 0 ]; then # Health check failed: Withdraw route echo "withdraw route $ANYCAST_IP next-hop self" FAILED=1 fi fi sleep 2 done Save the health check file as /etc/exabgp/healthcheck-proxysql, and make it executable with chmod +x /etc/exabgp/healthcheck-proxysql. To avoid passwords in the check script, we tell mysqladmin to load them from a separate file. Create that file (/etc/exabgp/proxysql-monitor.cnf) with the credentials that you want the check to use to connect to ProxySQL: bash Copy Copied! [client] user=user password=password host=127.0.0.1 port=6033 Ensure that you replace “user” and “password” with your actual user credentials. Restrict the file permissions with: bash Copy Copied! sudo chown exabgp:exabgp /etc/exabgp/proxysql-monitor.cnf sudo chmod 600 /etc/exabgp/proxysql-monitor.cnf In our example, the script only checks if connecting to ProxySQL succeeds, and not whether ProxySQL can reach any backend servers. The checks you come up with should ideally only test the readiness of the ProxySQL process, regardless of the health of the MySQL nodes behind it. Otherwise you might get unwanted side effects. For example: you could write a check to verify whether there are hosts with status ONLINE in the runtime_mysql_servers table. At first it might sound like a good idea, as a misconfigured ProxySQL server would be taken offline. BUT: In case your MySQL cluster is experiencing a downtime, ALL ProxySQLs will withdraw their routes, causing clients to see timeouts instead of potentially helpful error messages. Define your health checks carefully based on your specific architecture. Configuring ExaBGP ExaBGP configuration lives in the /etc/exabgp/exabgp.conf file. You can find full details on what can be configured there in the ExaBGP documentation. For the purpose of this blog post, we will define three blocks. The process block defines the path to the health check script. The template block defines the BGP settings and which health check process to run. Each neighbor block defines the router we want ExaBGP to peer with. bash Copy Copied! # ------------------------------------------------------------------- # Process Definitions # ------------------------------------------------------------------- process proxysql-healthcheck { run "/etc/exabgp/healthcheck-proxysql"; encoder text; } # ------------------------------------------------------------------- # Neighbor Templates # ------------------------------------------------------------------- template { neighbor opnsense-nodes { router-id 10.5.20.4; local-as 64513; peer-as 64512; api { processes [ proxysql-healthcheck ]; } } } # ------------------------------------------------------------------- # OPNsense nodes # ------------------------------------------------------------------- neighbor 10.5.8.251 { inherit opnsense-nodes; local-address 10.5.20.4; } Create the file on proxysql-01 and proxysql-02, but make sure to update the neighbor and template block. The router-id and local-address should be set to 10.5.20.5 on proxysql-02. Once you have created this file, restart ExaBGP. bash Copy Copied! systemctl restart exabgp You can check the status of ExaBGP by running exabgpcli show neighbor summary on the ProxySQL hosts. An overview of the workflow for a healthy ProxySQL node ExaBGP starts the health check script that we configured in exabgp.conf The script sees that ProxySQL is running, and outputs announce route 10.5.200.1/32 next-hop self. ExaBGP receives this output and sends a BGP UPDATE message to OPNsense. OPNsense adds the ProxySQL node as potential “next hop” to its routing table for 10.5.200.1. If the health checks detects that ProxySQL is not running, it will output withdraw route 10.5.200.1/32 next-hop self, which tells ExaBGP to send a BGP update for OPNsense to remove that route from its routing table. The traffic is redistributed to the remaining healthy ProxySQL node. Check BGP ECMP is configured correctly Check the firewall Ensure that the firewall is not blocking traffic by navigating to Firewall -&gt; Log Files -&gt; Live View. Make sure you have enabled logging in the firewall rule you created. Filter for “address” “is” “10.5.200.1” and check that the connections are not blocked. Check ExaBGP on the ProxySQL Run the health check on the ProxySQL to confirm that the ProxySQL reports that it is healthy, and announces the route. bash Copy Copied! sudo -u exabgp /etc/exabgp/healthcheck-proxysql You should see output like: bash Copy Copied! announce route 10.5.200.1/32 next-hop self Check that ExaBGP announces the route with: bash Copy Copied! exabgpcli show adj-rib out You should see output like: bash Copy Copied! neighbor 10.5.8.251 ipv4 unicast 10.5.200.1/32 next-hop self Check that ExaBGP has an established BGP session to OPNsense with: bash Copy Copied! exabgpcli show neighbor summary You want to see that the state is established. State active means “ready to connect”, but no connection is actually made. bash Copy Copied! Peer AS up/down state | #sent #recvd 10.5.8.251 64512 0:00:57 established 2 6 Verify the BGP connections and routes on OPNsense. Navigate to Routing -&gt; Diagnostics -&gt; BGP to check the routing status. The Anycast IP should appear twice, one entry for each ProxySQL. Both entries should be marked valid. The path should show the AS number 64513. BGP always selects a single best path. With Maximum Paths set, the other equal-cost paths are also installed in the routing table and are flagged as multipath. You can also run vtysh on the OPNsense CLI to check this: bash Copy Copied! vtysh -c "show ip route 10.5.200.1/32" You should see an entry like: bash Copy Copied! Routing entry for 10.5.200.1/32 Known via "bgp", distance 20, metric 0, best Last update 00:05:12 ago Flags: Selected Status: Installed * 10.5.20.4, via vtnet1, weight 1 * 10.5.20.5, via vtnet1, weight 1 The * next to each result shows that this is an active next-hop path. Because the two results share the same weight (1), ECMP is active, and OPNsense will load balance the tcp connections equally across the two routes. Summary In this blog post we stepped through an example setup of BGP ECMP Anycast for ProxySQL. We configured OPNsense to accept anycast routes from our ProxySQL nodes and load-balance across them. On each node, ExaBGP announces the route whilst ProxySQL is healthy and withdraws the route when ProxySQL fails. To scale the cluster, add the new node to the proxysql_nodes alias, create a BGP neighbor for it, and raise Maximum Paths to match the new node count. This post is part of the Percona Community Writers Program. In our previous blog post ProxySQL HA with BGP ECMP Anycast, we established why we would want to use BGP ECMP as a strategy for making our ProxySQL cluster highly available. Now let’s look at how to implement it technically.

For this scenario we assume that we have:

  • an OPNsense instance as our BGP Router with the IP address 10.5.8.251
  • Two ProxySQL nodes (proxysql-01 with IP 10.5.20.4 and proxysql-02 with IP 10.5.20.5)
  • 10.5.200.1 as the anycast IP we want both ProxySQL nodes to accept connections on

Setup on the router

Enable BGP on OPNsense

The out-of-the-box installation of OPNsense does not come with BGP support. To add it, install the FRRouting package first (os-frr), which will enable available protocols for dynamic routing.

Install the os-frr plugin in System -> Firmware -> Plugins.

OPNsense Plugin

We need to activate the routing service on our OPNsense host.

  • In Routing -> General check “enable”

OPNsense Enable Routing

Next, assign an Autonomous System (AS) number to the OPNsense BGP peer. An AS number identifies a network, or a group of routers, under a single administrative domain. Choose a private AS number from the range 64512-65534.

In Routing -> BGP (advanced mode):

  • check “enable”.
  • set the chosen AS number (in this example we use 64512).
  • set Maximum Paths to 2 to match the number of ProxySQL nodes that we have.

Maximum Paths configures OPNsense to perform Equal-Cost Multi-Path (ECMP) load balancing across the two paths. Any additional paths are kept as backups and only used if an active route is withdrawn.

OPNsense BGP Routing

Create a firewall rule for BGP traffic

BGP exchanges routing information through a TCP connection on port 179. Our ProxySQL nodes will establish this connection towards OPNsense, so we need a firewall rule to allow this traffic.

First, we will create an alias for the ProxySQL nodes. This keeps the firewall rule readable, and new nodes only need to be added in one place.

In Firewall -> Aliases, add:

  • Name: proxysql_nodes
  • Type: Host(s)
  • Content: 10.5.20.4, 10.5.20.5 (The IP addresses of our ProxySQL nodes)

OPNsense Alias

With the Alias in place, we can now continue to define the actual Firewall rule:

Navigate to Firewall -> Rules and create a new rule:

  • Description: Allow ProxySQL network to BGP peer with OPNsense
  • Interface: Select the interface the ProxySQL nodes are in.
  • Action: Pass
  • Direction: in
  • Version: IPv4
  • Protocol: TCP
  • Source: proxysql_nodes (the alias we created in the previous step)
  • Source port: any
  • Destination: the OPNsense address the ProxySQL nodes peer with (10.5.8.251)
  • Destination port: BGP (179)

Note, here we do not explicitly enable logging. It may be helpful to enable logging that if you need to debug the BGP peering.

OPNsense Firewall Rule BGP Alias

Set up route filtering with a prefix list

We need to set up an inbound filter using a prefix list. The prefix list defines the networks that OPNsense will accept routes for, and should be as narrow as possible to prevent a misbehaving peer from advertising routes that could re-route sensitive traffic through it.

Prefix lists are configured in Routing -> BGP -> Prefix Lists. Create the prefix list:

  • Description: ProxySQL Anycast IP
  • Name: ANYCAST-PROXYSQL-IN
  • IP Version: IPV4
  • Sequence Number: for this example we use 10. Sequence numbers are used in order to set the order with which to apply the rules, in case you would have multiple rules for the same prefix list. We only need to create one entry for the ProxySQL Anycast IP.
  • Action: Permit, as we want to allow the ProxySQL Anycast IP.
  • Network: 10.5.200.1/32, the Anycast IP of our ProxySQL nodes.

Tip: Click the ⓘ icon next to a field in the OPNsense UI for more information.

OPNsense Create Prefix List

Configure BGP neighbors

In this step we will configure OPNsense to know about our ProxySQL servers and their intent to peer. Neighbor configuration defines which IPs are allowed to speak the BGP protocol with OPNsense. In our case, we need to define a neighbor for both of our ProxySQL hosts:

Choose a unique AS number for the cluster. All ProxySQL nodes belonging to this cluster will share this AS number, whilst separate clusters must use different, dedicated AS numbers. Ensure that the cluster AS number is distinct from the one configured for OPNsense. For this example we choose the AS number 64513.

Navigate to Routing -> BGP -> Neighbors and add a new one.

  • Description: ProxySQL-01
  • Peer-IP: 10.5.20.4
  • Remote AS: 64513
  • Local AS: 64512
  • Prefix-List In: ANYCAST-PROXYSQL-IN:10

Make sure you repeat this for the second ProxySQL.

OPNsense BGP Neighbor

All configurable BGP options can be found in the OPNsense documentation.

Setup on the ProxySQL nodes

Next, assign the anycast virtual IP (10.5.200.1) to the loopback interface of each ProxySQL node, so that the node accepts traffic intended for that IP address.

bash 
# Add the Anycast VIP to the loopback interface
sudo ip addr add 10.5.200.1/32 dev lo

Make the address persistent across reboots by adding it to netplan, systemd-networkd or /etc/network/interfaces.

Install ExaBGP

ExaBGP is a tool that can speak the BGP protocol with OPNsense and is able to announce / withdraw routes.

Install ExaBGP on the ProxySQL hosts with:

bash 
sudo apt-get update
sudo apt-get install exabgp

For information on installing ExaBGP on other distributions, you can check the ExaBGP wiki here.

Note that BIRD (BIRD Internet Routing Daemon) or FRR can be used as alternative tools to ExaBGP, but in this example we will go with ExaBGP.

Create the health check

As we only want to route traffic to a ProxySQL host when the ProxySQL process is running, we need a way to tell ExaBGP when to announce the route and when to withdraw it. We will write a health check script that ExaBGP can execute to determine the state of our ProxySQL process. If this health check fails, ExaBGP withdraws the node’s route from OPNsense. OPNsense removes that node as an anycast next hop, and routes the traffic to the remaining healthy nodes.

In our example, we will use a simple bash script that uses mysqladmin to send a PING to ProxySQL:

bash 
#!/usr/bin/env bash

ANYCAST_IP="10.5.200.1/32"
FAILED=1

while true; do
 # Execute the health check command and depending on the return code
 # announce or withdraw the route.
 mysqladmin --defaults-extra-file=/etc/exabgp/proxysql-monitor.cnf --connect-timeout=2 ping &> /dev/null
 STATUS=$?

 if [ $STATUS -eq 0 ]; then
 if [ $FAILED -ne 0 ]; then
 # Recovered: Announce route
 echo "announce route $ANYCAST_IP next-hop self"
 FAILED=0
 fi
 else
 if [ $FAILED -eq 0 ]; then
 # Health check failed: Withdraw route
 echo "withdraw route $ANYCAST_IP next-hop self"
 FAILED=1
 fi
 fi
 sleep 2
done

Save the health check file as /etc/exabgp/healthcheck-proxysql, and make it executable with chmod +x /etc/exabgp/healthcheck-proxysql.

To avoid passwords in the check script, we tell mysqladmin to load them from a separate file. Create that file (/etc/exabgp/proxysql-monitor.cnf) with the credentials that you want the check to use to connect to ProxySQL:

bash 
[client]
user=user
password=password
host=127.0.0.1
port=6033

Ensure that you replace “user” and “password” with your actual user credentials. Restrict the file permissions with:

bash 
sudo chown exabgp:exabgp /etc/exabgp/proxysql-monitor.cnf
sudo chmod 600 /etc/exabgp/proxysql-monitor.cnf

In our example, the script only checks if connecting to ProxySQL succeeds, and not whether ProxySQL can reach any backend servers. The checks you come up with should ideally only test the readiness of the ProxySQL process, regardless of the health of the MySQL nodes behind it. Otherwise you might get unwanted side effects. For example: you could write a check to verify whether there are hosts with status ONLINE in the runtime_mysql_servers table. At first it might sound like a good idea, as a misconfigured ProxySQL server would be taken offline. BUT: In case your MySQL cluster is experiencing a downtime, ALL ProxySQLs will withdraw their routes, causing clients to see timeouts instead of potentially helpful error messages. Define your health checks carefully based on your specific architecture.

Configuring ExaBGP

ExaBGP configuration lives in the /etc/exabgp/exabgp.conf file. You can find full details on what can be configured there in the ExaBGP documentation. For the purpose of this blog post, we will define three blocks.

  • The process block defines the path to the health check script.
  • The template block defines the BGP settings and which health check process to run.
  • Each neighbor block defines the router we want ExaBGP to peer with.
bash 
# -------------------------------------------------------------------
# Process Definitions
# -------------------------------------------------------------------
process proxysql-healthcheck {
 run "/etc/exabgp/healthcheck-proxysql";
 encoder text;
}

# -------------------------------------------------------------------
# Neighbor Templates
# -------------------------------------------------------------------
template {
 neighbor opnsense-nodes {
 router-id 10.5.20.4;
 local-as 64513;
 peer-as 64512;

 api {
 processes [ proxysql-healthcheck ];
 }
 }
}

# -------------------------------------------------------------------
# OPNsense nodes
# -------------------------------------------------------------------
neighbor 10.5.8.251 {
 inherit opnsense-nodes;
 local-address 10.5.20.4;
}

Create the file on proxysql-01 and proxysql-02, but make sure to update the neighbor and template block. The router-id and local-address should be set to 10.5.20.5 on proxysql-02.

Once you have created this file, restart ExaBGP.

bash 
systemctl restart exabgp

You can check the status of ExaBGP by running exabgpcli show neighbor summary on the ProxySQL hosts.

An overview of the workflow for a healthy ProxySQL node

  • ExaBGP starts the health check script that we configured in exabgp.conf
  • The script sees that ProxySQL is running, and outputs announce route 10.5.200.1/32 next-hop self.
  • ExaBGP receives this output and sends a BGP UPDATE message to OPNsense.
  • OPNsense adds the ProxySQL node as potential “next hop” to its routing table for 10.5.200.1.

If the health checks detects that ProxySQL is not running, it will output withdraw route 10.5.200.1/32 next-hop self, which tells ExaBGP to send a BGP update for OPNsense to remove that route from its routing table. The traffic is redistributed to the remaining healthy ProxySQL node.

Check BGP ECMP is configured correctly

Check the firewall

Ensure that the firewall is not blocking traffic by navigating to Firewall -> Log Files -> Live View. Make sure you have enabled logging in the firewall rule you created. Filter for “address” “is” “10.5.200.1” and check that the connections are not blocked.

Check ExaBGP on the ProxySQL

Run the health check on the ProxySQL to confirm that the ProxySQL reports that it is healthy, and announces the route.

bash 
sudo -u exabgp /etc/exabgp/healthcheck-proxysql

You should see output like:

bash 
announce route 10.5.200.1/32 next-hop self

Check that ExaBGP announces the route with:

bash 
exabgpcli show adj-rib out

You should see output like:

bash 
neighbor 10.5.8.251 ipv4 unicast 10.5.200.1/32 next-hop self

Check that ExaBGP has an established BGP session to OPNsense with:

bash 
exabgpcli show neighbor summary

You want to see that the state is established. State active means “ready to connect”, but no connection is actually made.

bash 
Peer AS up/down state | #sent #recvd
10.5.8.251 64512 0:00:57 established 2 6

Verify the BGP connections and routes on OPNsense.

Navigate to Routing -> Diagnostics -> BGP to check the routing status. The Anycast IP should appear twice, one entry for each ProxySQL. Both entries should be marked valid. The path should show the AS number 64513.

OPNsense BGP Diagnostics Routing Table

BGP always selects a single best path. With Maximum Paths set, the other equal-cost paths are also installed in the routing table and are flagged as multipath. You can also run vtysh on the OPNsense CLI to check this:

bash 
vtysh -c "show ip route 10.5.200.1/32"

You should see an entry like:

bash 
Routing entry for 10.5.200.1/32
 Known via "bgp", distance 20, metric 0, best
 Last update 00:05:12 ago
 Flags: Selected
 Status: Installed
 * 10.5.20.4, via vtnet1, weight 1
 * 10.5.20.5, via vtnet1, weight 1

The * next to each result shows that this is an active next-hop path. Because the two results share the same weight (1), ECMP is active, and OPNsense will load balance the tcp connections equally across the two routes.

Summary

In this blog post we stepped through an example setup of BGP ECMP Anycast for ProxySQL. We configured OPNsense to accept anycast routes from our ProxySQL nodes and load-balance across them. On each node, ExaBGP announces the route whilst ProxySQL is healthy and withdraws the route when ProxySQL fails. To scale the cluster, add the new node to the proxysql_nodes alias, create a BGP neighbor for it, and raise Maximum Paths to match the new node count.

This post is part of the Percona Community Writers Program.

]]> Thu, 08 Oct 2026 18:50:00 +0000 Percona Community MySQL ProxySQL Opensource BGP Percona Server DevOps Your Galera cluster’s hardest problem was never the replication library https://www.percona.com/?p=54528 https://www.percona.com/blog/your-galera-clusters-hardest-problem-was-never-the-replication-library/ Ask anyone who runs a multi-node MySQL Galera Cluster in production about their worst night. Almost nobody says write-set replication was wrong. They say something closer to this: “All queries piling up… no error messages in logs.” Twenty minutes of downtime, ended only when a human picked a node to kill. (codership-team) Or the flow-control stall where writes time out and wsrep_flow_control_sent still reads 0, so there’s nothing to alert on and nothing to write in the post-mortem. (codership-team) You have your own version of these. You didn’t need mine. And here’s the thing about all of them: the root cause is usually not Galera. It’s flashcache, or a shadowed wsrep_provider_options line, or DNS resolving to the wrong interface, or a one-way firewall path. One team diagnosed a Kubernetes crash loop by reading the operator’s source code. So your MTTR isn’t dominated by repair. It’s dominated by diagnosis. The cluster is not hard to fix once you know what’s wrong, but it’s hard to know what’s wrong at 3am with writes stalled and clean logs. That’s what changed on 30 September 2026. What the EOL actually takes away MariaDB plc acquired Codership in May 2025 and set the end of support for all current MySQL Galera Cluster versions at 30 September 2026. No development, no maintenance, no binary releases after that. Your cluster will behave on 1 October exactly as it did the day before. Nothing new breaks. What disappears is the two things that used to bound those incidents: someone to escalate to, and a release that eventually contains the fix. MariaDB Enterprise’s recommended way out is migrating to MariaDB Galera Cluster. That’s a different database: distinct system table structure, a data dictionary that differs fundamentally from MySQL’s, and user privileges that MariaDB’s own migration notes say you have to systematically recreate. You built this HA layer on open source MySQL because you wanted control. Changing database vendor to keep a support contract is the opposite of control. Two paths that don’t ask you to: Keep the cluster, buy back the escalation path If diagnosis is the expensive part, what shortens it is a senior engineer who has seen your failure mode before. Not a new binary. Percona MySQL Support covers your existing MySQL Galera Cluster as it is. No migration, no re-platforming, no privilege rebuild. Senior MySQL engineers 24x7x365 on follow-the-sun, consultative support as well as incident triage (wsrep tuning, DDL strategy, write distribution, quorum design), and contractual SLAs down to 15-minute initial response for Severity 1 incidents on Premium tier. Details in the support datasheet. Or move to PXC, where the cluster layer still gets fixed Percona XtraDB Cluster is Percona Server for MySQL plus the Galera library – our own fork, in a public repo, compatible with MySQL Community Edition. The difference from the MariaDB path is mechanical: your schema, system tables, and user accounts carry over as they are. Same wsrep behavior, same ecosystem, same tools, same connectors. The ten-step walkthrough is public and free to follow on your own. The reason to pick PXC isn’t that it’s another Galera build. It’s that the failure modes at the top of this post are what we ship fixes for: IST and SST behavior, node eviction, flow control, DDL under concurrent writes. Public release notes, Jira IDs attached, every quarter: That cadence goes back to 2012, across 5.5, 5.6, 5.7, 8.0 and now 8.4, with the 8.0 line still shipping alongside. In the last 90 days, nearly 4,000 PXC host instances reported telemetry outside Kubernetes, and the PXC Operator runs across thousands more Kubernetes deployments. The engineers you’d escalate to are in this cluster layer every week.- See the recent work on gcache inspection and cross-site replication in the Operator. If you’d rather not migrate alone, our migration service includes a test environment on the target cluster with a wsrep behavior comparison against your current one: you watch it handle the same failures before you commit, plus a documented rollback plan, scheduled cutover with off-hours support, and a health audit a few weeks after go-live. What’s not an option Running unmaintained. Not because of the date since your cluster won’t notice the date. Because the next time all the queries pile up and the logs say nothing, there’s no ticket to open and no release to wait for. Support buys time. Migration ends the exposure. Doing both, in that order, is a perfectly good plan. Either way your clustering layer stays open source, stays on MySQL, and stays supported. Talk to us about which one fits. The post Your Galera cluster’s hardest problem was never the replication library appeared first on Percona. Ask anyone who runs a multi-node MySQL Galera Cluster in production about their worst night. Almost nobody says write-set replication was wrong. They say something closer to this:

“All queries piling up… no error messages in logs.”

Twenty minutes of downtime, ended only when a human picked a node to kill. (codership-team)

Or the flow-control stall where writes time out and wsrep_flow_control_sent still reads 0, so there’s nothing to alert on and nothing to write in the post-mortem. (codership-team)

You have your own version of these. You didn’t need mine.

And here’s the thing about all of them: the root cause is usually not Galera. It’s flashcache, or a shadowed wsrep_provider_options line, or DNS resolving to the wrong interface, or a one-way firewall path. One team diagnosed a Kubernetes crash loop by reading the operator’s source code.

So your MTTR isn’t dominated by repair. It’s dominated by diagnosis. The cluster is not hard to fix once you know what’s wrong, but it’s hard to know what’s wrong at 3am with writes stalled and clean logs.

That’s what changed on 30 September 2026.

What the EOL actually takes away

MariaDB plc acquired Codership in May 2025 and set the end of support for all current MySQL Galera Cluster versions at 30 September 2026. No development, no maintenance, no binary releases after that.

Your cluster will behave on 1 October exactly as it did the day before. Nothing new breaks. What disappears is the two things that used to bound those incidents: someone to escalate to, and a release that eventually contains the fix.

MariaDB Enterprise’s recommended way out is migrating to MariaDB Galera Cluster. That’s a different database: distinct system table structure, a data dictionary that differs fundamentally from MySQL’s, and user privileges that MariaDB’s own migration notes say you have to systematically recreate. You built this HA layer on open source MySQL because you wanted control. Changing database vendor to keep a support contract is the opposite of control.

Two paths that don’t ask you to:

Keep the cluster, buy back the escalation path

If diagnosis is the expensive part, what shortens it is a senior engineer who has seen your failure mode before. Not a new binary.

Percona MySQL Support covers your existing MySQL Galera Cluster as it is. No migration, no re-platforming, no privilege rebuild. Senior MySQL engineers 24x7x365 on follow-the-sun, consultative support as well as incident triage (wsrep tuning, DDL strategy, write distribution, quorum design), and contractual SLAs down to 15-minute initial response for Severity 1 incidents on Premium tier. Details in the support datasheet.

Or move to PXC, where the cluster layer still gets fixed

Percona XtraDB Cluster is Percona Server for MySQL plus the Galera library – our own fork, in a public repo, compatible with MySQL Community Edition.

The difference from the MariaDB path is mechanical: your schema, system tables, and user accounts carry over as they are. Same wsrep behavior, same ecosystem, same tools, same connectors. The ten-step walkthrough is public and free to follow on your own.

The reason to pick PXC isn’t that it’s another Galera build. It’s that the failure modes at the top of this post are what we ship fixes for: IST and SST behavior, node eviction, flow control, DDL under concurrent writes. Public release notes, Jira IDs attached, every quarter:

That cadence goes back to 2012, across 5.5, 5.6, 5.7, 8.0 and now 8.4, with the 8.0 line still shipping alongside. In the last 90 days, nearly 4,000 PXC host instances reported telemetry outside Kubernetes, and the PXC Operator runs across thousands more Kubernetes deployments. The engineers you’d escalate to are in this cluster layer every week.- See the recent work on gcache inspection and cross-site replication in the Operator.

If you’d rather not migrate alone, our migration service includes a test environment on the target cluster with a wsrep behavior comparison against your current one: you watch it handle the same failures before you commit, plus a documented rollback plan, scheduled cutover with off-hours support, and a health audit a few weeks after go-live.

What’s not an option

Running unmaintained. Not because of the date since your cluster won’t notice the date. Because the next time all the queries pile up and the logs say nothing, there’s no ticket to open and no release to wait for.

Support buys time. Migration ends the exposure. Doing both, in that order, is a perfectly good plan. Either way your clustering layer stays open source, stays on MySQL, and stays supported.

Talk to us about which one fits.

The post Your Galera cluster’s hardest problem was never the replication library appeared first on Percona.

]]> Thu, 08 Oct 2026 16:07:52 +0000 MySQL Performance Blog MySQL Percona Announcements XtraDB Cluster (PXC) Making Sense of MySQL Telemetry Metrics 9d926ac03ca810fa61af4cd77beb8613 https://blogs.oracle.com/mysql/making-sense-of-mysql-telemetry-metrics In the first post in this series, we configured MySQL Telemetry to export OpenTelemetry metrics to Prometheus. Once the metrics arrive, the more important question is how a DBA should read them. This post is not a catalogue of every metric or meter. Its purpose is to build a practical way of thinking: start with […] Wed, 07 Oct 2026 03:04:33 +0000 Oracle MySQL Group Monitoring MySQL MySQL Enterprise Why MySQL Bug Fixes Can Differ Across LTS Releases d6c08698c8939b8bce61bb94e585e703 https://blogs.oracle.com/mysql/why-mysql-bug-fixes-can-differ-across-lts-releases At a recent discussion with MySQL ACEs and Rockstars, we were asked a fair question: If a bug is fixed in one supported MySQL LTS series, why isn’t it always fixed in another? A quick refresher: LTS means Long-Term Support. Within an LTS series, MySQL aims to keep the feature set and data format stable […] Tue, 06 Oct 2026 19:54:58 +0000 Oracle MySQL Group MySQL Community mysql mysqlcommunity DBTrail: Analytical Reports on Your MySQL Data, Without Running Them on MySQL – Part 1 https://www.percona.com/?p=54477 https://www.percona.com/blog/dbtrail-analytical-reports-on-your-mysql-data-without-running-them-on-mysql-part-1/ MySQL is very good at serving application workloads, but things get more complicated when somebody decides to run a large report on the same server. A scan can push hot pages out of the buffer pool. A long-running read can hold back purge. A GROUP BY over millions of rows can start writing temporary tables to disk. The traditional answer is a reporting replica. But that means running another MySQL server, and at the end of the day it is still a row-oriented database. Another option is moving the data into an analytical database. That can work very well, but now you have a data pipeline to build, operate, monitor, and eventually debug. We have been looking at what MySQL users can do with DuckDB, and DBTrail is an interesting approach. DBTrail is open source under Apache 2.0. It is being built by Daniel Guzman-Burgos, who previously worked at Percona as a MySQL Technical Lead. The basic idea is simple. DBTrail connects to MySQL similarly to a replica. It makes an initial copy of the tables using mydumper and stores the data as Parquet files, either locally or in S3. After that, it reads the MySQL binary log and periodically applies changes to the copy. You query the resulting data with DuckDB. Nothing is installed inside MySQL, there is no MySQL plugin, agent, or trigger. The setup would look like this: Let’s do some performance testing. The setup We used three AWS machines in the same subnet in us-west-2a. Role Machine Software Source r7i.4xlarge Percona Server for MySQL 8.4.11-11 DBTrail and DuckDB r7i.4xlarge DBTrail 0.99.0, DuckDB 1.5.6 Load generator c7i.2xlarge sysbench 1.0.20, sysbench-tpcc Machines in more detail Source and DBTrail machines Load machine CPU Intel Xeon Platinum 8488C, 8 cores, 16 threads Same CPU, 4 cores, 8 threads Memory 128 GB, 123.8 GB usable 16 GB Disk EBS gp3, 400 GB, 16,000 IOPS, 1,000 MB/s provisioned Not used by the test Filesystem ext4, relatime,discard,commit=30 I/O scheduler none, 128 KB read-ahead OS Ubuntu 24.04.5, kernel 7.0.0-1013-aws Same Kernel settings swappiness 60, dirty ratio 20/10, THP madvise Same Docker 29.1.3 The MySQL source Percona Server runs in Docker using host networking. We used full durability and a buffer pool large enough to keep the whole data set in memory:&lt;code&gt;innodb_buffer_pool_size = 96G innodb_redo_log_capacity = 32G innodb_flush_log_at_trx_commit = 1 sync_binlog = 1 innodb_flush_method = O_DIRECT innodb_io_capacity = 8000 innodb_io_capacity_max = 16000 binlog_format = ROW binlog_row_image = FULL gtid_mode = ON&lt;/code&gt;The workload is sysbench-tpcc with 200 warehouses, which produces 102 million rows and about 19 GB of data when the run started. For the initial workload we held the workload at 300 transactions per second. One TPC-C transaction in this test is roughly 28 SQL statements, so that means about: 8,500 QPS 5,200 row changes per second It is important to put that load into context—on the same server flat out with 64 threads and nothing else attached, it reached 4,049 transactions per second, or about 115,000 QPS. So the 300 TPS test uses only about 7% of the available capacity, and we chose a relatively light load intentionally. We wanted to see what analytical queries cost even when the source has plenty of headroom. DBTrail DBTrail installs with one command that starts its Docker Compose stack:&lt;code&gt;curl -fsSL https://raw.githubusercontent.com/dbtrail/dbtrail/v0.99.0/install.sh \ | DBTRAIL_REF=v0.99.0 sh&lt;/code&gt;The stack contains two containers: DBTrail itself and a MySQL 8.4.9 instance DBTrail uses as an index of row changes. This internal MySQL needs attention, as the default installation leaves its buffer pool at MySQL’s 128 MB default. For this workload, that is nowhere near enough. For the main tests we configured it like this:&lt;code&gt;innodb_buffer_pool_size = 48G innodb_redo_log_capacity = 16G innodb_log_buffer_size = 256M innodb_flush_log_at_trx_commit = 2 innodb_flush_method = O_DIRECT innodb_io_capacity = 8000 innodb_io_capacity_max = 16000 innodb_page_cleaners = 16 innodb_flush_neighbors = 0 skip-log-bin&lt;/code&gt;DuckDB ran on the same machine with its defaults: 16 threads and a 99 GB memory limit. Does DBTrail slow MySQL down? Short answer: at this workload, almost not at all. The load ran continuously. Each row below represents one phase of the same test. Phase TPS QPS Median 95th percentile Worst second Nothing attached, 10 min 299.9 8,554 41.9 ms 46.6 ms DBTrail reading binlog, 10 min 300.5 8,535 41.9 ms 47.5 ms Initial copy of all tables, 4m 40s 298.1 8,460 41.1 ms 118.9 ms Copy updated every 5 min, 75 min 300.2 8,539 41.1 ms 46.6 ms There is one spike here that stands out. When the initial copy started, at approximately second 1,500 of the workload, the 95th percentile latency jumped to 119 ms and throughput dropped to 271 transactions for that second and it happened once. The initial copy takes a global read lock briefly when it starts, and the timing matches that operation, after that, DBTrail read 102 million rows in less than five minutes without any measurable slowdown in the application workload. CPU usage confirms the same story: the source normally sat around 18% CPU, but during the initial table copy it increased to about 35%. When we later ran the five analytical reports directly against MySQL, CPU was around 24%. There is also a noticeable increase in source writes around minute 12. That happened three minutes before DBTrail connected, so it was unrelated to DBTrail. The DBTrail machine itself writes roughly as much as the source because it keeps an index of every captured row change. Query times on the TPC-C data DBTrail creates a views.sql file containing a DuckDB view for each source table. From there you can query the data normally:&lt;code&gt;$ duckdb -init views.sql D SELECT i.i_id, i.i_name, sum(ol.ol_quantity) AS units, sum(ol.ol_amount) AS revenue FROM tpcc.order_line1 ol JOIN tpcc.item1 i ON i.i_id = ol.ol_i_id GROUP BY i.i_id, i.i_name ORDER BY revenue DESC LIMIT 10;&lt;/code&gt;We ran the same five reports against MySQL and against DuckDB while the transactional workload continued running, during the workload the data had grown from 102 million to 113 million rows. MySQL had essentially everything cached in memory and the DBTrail copy was also in its normal operating state: a base Parquet file plus the accumulated change files that DuckDB needs to combine when executing the query. The results: Report MySQL DuckDB on DBTrail copy Revenue per warehouse 9.7 s 0.75 s Orders and revenue per district/month 46.5 s 2.8 s Ten best-selling items 5m 32s 1.7 s Customers by state and credit 3.6 s 0.18 s Warehouses low on stock 3.0 s 0.23 s Total 6m 35s 5.7 s This is where the difference becomes interesting: 6m 35s on MySQL and less than 6s on DuckDB. Running TPC-H: all 22 queries TPC-H is a more standard analytical workload; for this test, we loaded TPC-H at scale factor 10. The dataset contained: 86.6 million rows 18.2 GB of InnoDB data Normal indexes on join keys, order dates, and ship dates The initial DBTrail copy took 6 minutes 26 seconds and produced 2.8 GB of Parquet files. We ran the same SQL against both systems and compared results: MySQL DuckDB warm All 22 queries 5m 24s 10.3 s MySQL wins Q19 because an index can go directly to the relevant rows, Q17 is effectively a tie and for the rest, DuckDB is substantially faster. If you are interested in these performance numbers, stay tuned for Part 2, where we will look into how often DBTrail refreshes data to stay up to date. The post DBTrail: Analytical Reports on Your MySQL Data, Without Running Them on MySQL – Part 1 appeared first on Percona. MySQL is very good at serving application workloads, but things get more complicated when somebody decides to run a large report on the same server.

A scan can push hot pages out of the buffer pool. A long-running read can hold back purge. A GROUP BY over millions of rows can start writing temporary tables to disk.

The traditional answer is a reporting replica. But that means running another MySQL server, and at the end of the day it is still a row-oriented database.

Another option is moving the data into an analytical database. That can work very well, but now you have a data pipeline to build, operate, monitor, and eventually debug.

We have been looking at what MySQL users can do with DuckDB, and DBTrail is an interesting approach.

DBTrail is open source under Apache 2.0. It is being built by Daniel Guzman-Burgos, who previously worked at Percona as a MySQL Technical Lead.

The basic idea is simple.

DBTrail connects to MySQL similarly to a replica. It makes an initial copy of the tables using mydumper and stores the data as Parquet files, either locally or in S3.

After that, it reads the MySQL binary log and periodically applies changes to the copy.

You query the resulting data with DuckDB.

Nothing is installed inside MySQL, there is no MySQL plugin, agent, or trigger.

The setup would look like this:

Let’s do some performance testing.

The setup

We used three AWS machines in the same subnet in us-west-2a.

Role Machine Software
Source r7i.4xlarge Percona Server for MySQL 8.4.11-11
DBTrail and DuckDB r7i.4xlarge DBTrail 0.99.0, DuckDB 1.5.6
Load generator c7i.2xlarge sysbench 1.0.20, sysbench-tpcc

Machines in more detail

Source and DBTrail machines Load machine
CPU Intel Xeon Platinum 8488C, 8 cores, 16 threads Same CPU, 4 cores, 8 threads
Memory 128 GB, 123.8 GB usable 16 GB
Disk EBS gp3, 400 GB, 16,000 IOPS, 1,000 MB/s provisioned Not used by the test
Filesystem ext4, relatime,discard,commit=30
I/O scheduler none, 128 KB read-ahead
OS Ubuntu 24.04.5, kernel 7.0.0-1013-aws Same
Kernel settings swappiness 60, dirty ratio 20/10, THP madvise Same
Docker 29.1.3

The MySQL source

Percona Server runs in Docker using host networking.

We used full durability and a buffer pool large enough to keep the whole data set in memory:

<code>innodb_buffer_pool_size        = 96G
innodb_redo_log_capacity       = 32G
innodb_flush_log_at_trx_commit = 1
sync_binlog                    = 1
innodb_flush_method            = O_DIRECT
innodb_io_capacity             = 8000
innodb_io_capacity_max         = 16000
binlog_format                  = ROW
binlog_row_image               = FULL
gtid_mode                      = ON</code>

The workload is sysbench-tpcc with 200 warehouses, which produces 102 million rows and about 19 GB of data when the run started.

For the initial workload we held the workload at 300 transactions per second.

One TPC-C transaction in this test is roughly 28 SQL statements, so that means about:

  • 8,500 QPS
  • 5,200 row changes per second

It is important to put that load into context—on the same server flat out with 64 threads and nothing else attached, it reached 4,049 transactions per second, or about 115,000 QPS.

So the 300 TPS test uses only about 7% of the available capacity, and we chose a relatively light load intentionally. We wanted to see what analytical queries cost even when the source has plenty of headroom.

DBTrail

DBTrail installs with one command that starts its Docker Compose stack:

<code>curl -fsSL https://raw.githubusercontent.com/dbtrail/dbtrail/v0.99.0/install.sh \
  | DBTRAIL_REF=v0.99.0 sh</code>

The stack contains two containers: DBTrail itself and a MySQL 8.4.9 instance DBTrail uses as an index of row changes.

This internal MySQL needs attention, as the default installation leaves its buffer pool at MySQL’s 128 MB default. For this workload, that is nowhere near enough.

For the main tests we configured it like this:

<code>innodb_buffer_pool_size        = 48G
innodb_redo_log_capacity       = 16G
innodb_log_buffer_size         = 256M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method            = O_DIRECT
innodb_io_capacity             = 8000
innodb_io_capacity_max         = 16000
innodb_page_cleaners           = 16
innodb_flush_neighbors         = 0
skip-log-bin</code>

DuckDB ran on the same machine with its defaults: 16 threads and a 99 GB memory limit.

Does DBTrail slow MySQL down?

Short answer: at this workload, almost not at all.

The load ran continuously. Each row below represents one phase of the same test.

Phase TPS QPS Median 95th percentile Worst second
Nothing attached, 10 min 299.9 8,554 41.9 ms 46.6 ms
DBTrail reading binlog, 10 min 300.5 8,535 41.9 ms 47.5 ms
Initial copy of all tables, 4m 40s 298.1 8,460 41.1 ms 118.9 ms
Copy updated every 5 min, 75 min 300.2 8,539 41.1 ms 46.6 ms

There is one spike here that stands out.

When the initial copy started, at approximately second 1,500 of the workload, the 95th percentile latency jumped to 119 ms and throughput dropped to 271 transactions for that second and it happened once.

The initial copy takes a global read lock briefly when it starts, and the timing matches that operation, after that, DBTrail read 102 million rows in less than five minutes without any measurable slowdown in the application workload.

CPU usage confirms the same story: the source normally sat around 18% CPU, but during the initial table copy it increased to about 35%.

When we later ran the five analytical reports directly against MySQL, CPU was around 24%.

There is also a noticeable increase in source writes around minute 12. That happened three minutes before DBTrail connected, so it was unrelated to DBTrail.

The DBTrail machine itself writes roughly as much as the source because it keeps an index of every captured row change.

Query times on the TPC-C data

DBTrail creates a views.sql file containing a DuckDB view for each source table.

From there you can query the data normally:

<code>$ duckdb -init views.sql

D SELECT
      i.i_id,
      i.i_name,
      sum(ol.ol_quantity) AS units,
      sum(ol.ol_amount) AS revenue
  FROM tpcc.order_line1 ol
  JOIN tpcc.item1 i ON i.i_id = ol.ol_i_id
  GROUP BY i.i_id, i.i_name
  ORDER BY revenue DESC
  LIMIT 10;</code>

We ran the same five reports against MySQL and against DuckDB while the transactional workload continued running, during the workload the data had grown from 102 million to 113 million rows.

MySQL had essentially everything cached in memory and the DBTrail copy was also in its normal operating state: a base Parquet file plus the accumulated change files that DuckDB needs to combine when executing the query.

The results:

Report MySQL DuckDB on DBTrail copy
Revenue per warehouse 9.7 s 0.75 s
Orders and revenue per district/month 46.5 s 2.8 s
Ten best-selling items 5m 32s 1.7 s
Customers by state and credit 3.6 s 0.18 s
Warehouses low on stock 3.0 s 0.23 s
Total 6m 35s 5.7 s

This is where the difference becomes interesting: 6m 35s on MySQL and less than 6s on DuckDB.

Running TPC-H: all 22 queries

TPC-H is a more standard analytical workload; for this test, we loaded TPC-H at scale factor 10.

The dataset contained:

  • 86.6 million rows
  • 18.2 GB of InnoDB data
  • Normal indexes on join keys, order dates, and ship dates

The initial DBTrail copy took 6 minutes 26 seconds and produced 2.8 GB of Parquet files.

We ran the same SQL against both systems and compared results:

MySQL DuckDB warm
All 22 queries 5m 24s 10.3 s

MySQL wins Q19 because an index can go directly to the relevant rows, Q17 is effectively a tie and for the rest, DuckDB is substantially faster.

If you are interested in these performance numbers, stay tuned for Part 2, where we will look into how often DBTrail refreshes data to stay up to date.

The post DBTrail: Analytical Reports on Your MySQL Data, Without Running Them on MySQL – Part 1 appeared first on Percona.

]]> Tue, 06 Oct 2026 16:28:16 +0000 MySQL Performance Blog Benchmarks MySQL Village News: MySQL News + Events (6 October 2026) 6ac432dc38515d0001bcda84 https://villagesql.com/blog/village-news-mysql-news-events-6-october-2026/ Welcome back to Village News, our curated roundup of MySQL and database news. If you want to get these updates, just subscribe to the blog. Enjoy! MySQL News Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated) Thread Pool in Percona Server and MySQL (Part 1) Percona Blog TL;DR - MySQL 26.7.0 and 9.7.2 bring a thread pool to Community Server for the first time. TidesDB now available for MySQL v9, v26 Alex Gaetano Padula, TidesDB Blog TL;DR - TideSQL-MySQL 2.0.0 loads the TidesDB engine as a plugin into stock MySQL 9.7.0 and 26.7.0, so TidesDB tables can sit alongside InnoDB tables in the same server. For now it installs with a script from the repository. How easy is it to use the VillageSQL MCP extension? Ronald Bradford TL;DR - Ronald built and installed vsql-mcp on a throwaway server in about four minutes and answered a business question through a SELECT-only MCP user. His attempts to get past its guardrails were blocked. Database News Supabase is acquiring Turso Paul Copplestone, Supabase Blog TL;DR - Supabase is buying Turso, the team that rewrote SQLite in Rust, to build database infrastructure for AI agents. Turso founders Glauber Costa and Pekka Enberg join Supabase's leadership. Amazon Aurora's analytics is DuckDB: a reproducible side-by-side with pg_duckdb Franck Pachot TL;DR - Franck puts Aurora PostgreSQL's aurora_analytics and pg_duckdb side by side. The EXPLAIN plans match, down to DuckDB's internal compression operators and constants, which shows that Aurora's S3 Parquet/Iceberg analytics runs on embedded DuckDB. MariaDB Foundation Adds PostgreSQL to Its Engine‑Agnostic Testing Framework (TAF). Jonathan Miller, MariaDB Foundation TL;DR - TAF gains a PostgreSQL plugin and beta HammerDB TPROC-C/TPROC-H and Sysbench profiles, so MariaDB, MySQL and Postgres can be benchmarked under identical, reproducible conditions. Scale without limits: Multigres, OrioleDB, and dbarena Supabase Blog TL;DR - At Supabase Select, the company launched three things. Multigres (pooling and multi-node failover for Postgres) is in private alpha. The OrioleDB storage engine is in public beta. dbarena, an open benchmark site comparing managed Postgres providers, is live. When AI Finds the Bugs We Missed: A Very Busy Year for MariaDB Security Frédéric Descamps, MariaDB Foundation TL;DR - AI-assisted research drove 173 security reports over two quarters, which led to 27 published advisories and delayed some MariaDB releases. Update to the latest maintenance releases. Benchmark Analysis with HammerDB TPROC-C(TPC-C) on TideSQL v5.1.0, InnoDB in MariaDB v11.4.13 Alex Gaetano Padula, TidesDB Blog TL;DR - On a 3,000-warehouse TPROC-C run, TideSQL and InnoDB peaked at similar NOPM, but InnoDB had half the p99 latency. TideSQL held throughput at 128 VUs, where InnoDB dropped 34%. The author notes that one InnoDB setting was left untuned. Influence Is Not Ownership: Anna Widenius and Kaj Arnö on Governance, Open Source, and the Future of MariaDB Roberto V. Zicari, ODBMS Industry Watch TL;DR - The MariaDB Foundation's leaders explain its newly formalised governance. Authority comes from contribution rather than employer, and there is a defined path from contributor to committer, reviewer and maintainer. Designing Neki for performance Dirkjan Bussink, PlanetScale TL;DR - PlanetScale's sharded-Postgres router decodes lazily. Point lookups pass through as raw wire-protocol bytes, and only sort keys or row boundaries get decoded when a query needs them. Working with foreign key constraints in Aurora DSQL Rekha Reddy Anupati and Arnab Chowdhury, AWS Database Blog TL;DR - Aurora DSQL foreign keys are checked against the transaction snapshot, and conflicting concurrent key changes fail at commit with SQLSTATE 40001. Cascades count toward the 3,000-row transaction limit, and existing tables can use NOT VALID plus async validation. Upcoming Database Events High Performance Transaction Systems (HPTS) October 4-7, 2026 Asilomar Conference Grounds Pacific Grove, CA Open Source Summit Europe October 7-9, 2026 Prague, Czechia Community Over Code 2026 October 11-14, 2026 Hilton Glasgow Glasgow, UK TiDB SCaiLE 2026 October 15, 2026 Computer History Museum Mountain View, CA All Things Open 2026 October 19-20, 2026 Raleigh Convention Center Raleigh, NC PGConf.EU October 20–23, 2026 (PostgreSQL Europe) Valencia, Spain P99 CONF 2026 October 21-22, 2026 Online MySQL Public Discussion #6 October 22, 2026, 17:00–18:00 UTC Online Oracle AI World October 25–28, 2026 Las Vegas, NV PG Down Under 2026 October 30, 2026 Surry Hills Sydney, Australia MySQL Contributor Summit November 4-5, 2026 Virtual KubeCon + CloudNativeCon North America (VillageSQL is a sponsor) November 9-12, 2026 Salt Lake City, Utah PASS Data Community Summit November 9-11, 2026 Hyatt Regency Seattle Seattle, WA PGConf.Asia 2026 November 17-18, 2026 Hong Kong PGConf.PL 2026 November 24, 2026 ARCHE Dwór Uphagena Gdańsk, Poland AWS re:Invent 2026 November 30 – December 4, 2026 Las Vegas, NV Open Source Summit Japan December 7-9, 2026 Tokyo, Japan CIDR 2027 January 24-27, 2027 Mövenpick Hotel Amsterdam City Centre Amsterdam, The Netherlands FOSDEM 2027 January 30-31, 2027 ULB Solbosch Campus Brussels, Belgium CERN PGDay 2027 February 12, 2027 CERN Council Chamber Geneva, Switzerland PGConf India 2027 March 2-5, 2027 Sheraton Grand Hotel at Brigade Gateway Bengaluru, India KubeCon + CloudNativeCon Europe 2027 March 15-18, 2027 Barcelona, Spain Nordic PGDay 2027 March 16, 2027 Courtyard Kungsholmen Stockholm, Sweden SCaLE 24x April 1-4, 2027 Pasadena Convention Center Pasadena, CA PgDay Boston 2027 April 10, 2027 Tufts University Joyce Cummings Center Boston, MA PGConf.dev 2027 May 11-14, 2027 Plaza Centre-Ville Montréal, QC, Canada

Welcome back to Village News, our curated roundup of MySQL and database news.

If you want to get these updates, just subscribe to the blog.

Enjoy!

MySQL News

Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated)

Thread Pool in Percona Server and MySQL (Part 1)
Percona Blog

TL;DR - MySQL 26.7.0 and 9.7.2 bring a thread pool to Community Server for the first time.

TidesDB now available for MySQL v9, v26
Alex Gaetano Padula, TidesDB Blog

TL;DR - TideSQL-MySQL 2.0.0 loads the TidesDB engine as a plugin into stock MySQL 9.7.0 and 26.7.0, so TidesDB tables can sit alongside InnoDB tables in the same server. For now it installs with a script from the repository.

How easy is it to use the VillageSQL MCP extension?
Ronald Bradford

TL;DR - Ronald built and installed vsql-mcp on a throwaway server in about four minutes and answered a business question through a SELECT-only MCP user. His attempts to get past its guardrails were blocked.

Database News

Supabase is acquiring Turso
Paul Copplestone, Supabase Blog

TL;DR - Supabase is buying Turso, the team that rewrote SQLite in Rust, to build database infrastructure for AI agents. Turso founders Glauber Costa and Pekka Enberg join Supabase's leadership.

Amazon Aurora's analytics is DuckDB: a reproducible side-by-side with pg_duckdb
Franck Pachot

TL;DR - Franck puts Aurora PostgreSQL's aurora_analytics and pg_duckdb side by side. The EXPLAIN plans match, down to DuckDB's internal compression operators and constants, which shows that Aurora's S3 Parquet/Iceberg analytics runs on embedded DuckDB.

MariaDB Foundation Adds PostgreSQL to Its Engine‑Agnostic Testing Framework (TAF).
Jonathan Miller, MariaDB Foundation

TL;DR - TAF gains a PostgreSQL plugin and beta HammerDB TPROC-C/TPROC-H and Sysbench profiles, so MariaDB, MySQL and Postgres can be benchmarked under identical, reproducible conditions.

Scale without limits: Multigres, OrioleDB, and dbarena
Supabase Blog

TL;DR - At Supabase Select, the company launched three things. Multigres (pooling and multi-node failover for Postgres) is in private alpha. The OrioleDB storage engine is in public beta. dbarena, an open benchmark site comparing managed Postgres providers, is live.

When AI Finds the Bugs We Missed: A Very Busy Year for MariaDB Security
Frédéric Descamps, MariaDB Foundation

TL;DR - AI-assisted research drove 173 security reports over two quarters, which led to 27 published advisories and delayed some MariaDB releases. Update to the latest maintenance releases.

Benchmark Analysis with HammerDB TPROC-C(TPC-C) on TideSQL v5.1.0, InnoDB in MariaDB v11.4.13
Alex Gaetano Padula, TidesDB Blog

TL;DR - On a 3,000-warehouse TPROC-C run, TideSQL and InnoDB peaked at similar NOPM, but InnoDB had half the p99 latency. TideSQL held throughput at 128 VUs, where InnoDB dropped 34%. The author notes that one InnoDB setting was left untuned.

Influence Is Not Ownership: Anna Widenius and Kaj Arnö on Governance, Open Source, and the Future of MariaDB
Roberto V. Zicari, ODBMS Industry Watch

TL;DR - The MariaDB Foundation's leaders explain its newly formalised governance. Authority comes from contribution rather than employer, and there is a defined path from contributor to committer, reviewer and maintainer.

Designing Neki for performance
Dirkjan Bussink, PlanetScale

TL;DR - PlanetScale's sharded-Postgres router decodes lazily. Point lookups pass through as raw wire-protocol bytes, and only sort keys or row boundaries get decoded when a query needs them.

Working with foreign key constraints in Aurora DSQL
Rekha Reddy Anupati and Arnab Chowdhury, AWS Database Blog

TL;DR - Aurora DSQL foreign keys are checked against the transaction snapshot, and conflicting concurrent key changes fail at commit with SQLSTATE 40001. Cascades count toward the 3,000-row transaction limit, and existing tables can use NOT VALID plus async validation.

Upcoming Database Events

High Performance Transaction Systems (HPTS)
October 4-7, 2026
Asilomar Conference Grounds
Pacific Grove, CA

Open Source Summit Europe
October 7-9, 2026
Prague, Czechia

Community Over Code 2026
October 11-14, 2026
Hilton Glasgow
Glasgow, UK

TiDB SCaiLE 2026
October 15, 2026
Computer History Museum
Mountain View, CA

All Things Open 2026
October 19-20, 2026
Raleigh Convention Center
Raleigh, NC

PGConf.EU
October 20–23, 2026 (PostgreSQL Europe)
Valencia, Spain

P99 CONF 2026
October 21-22, 2026
Online

MySQL Public Discussion #6
October 22, 2026, 17:00–18:00 UTC
Online

Oracle AI World
October 25–28, 2026
Las Vegas, NV

PG Down Under 2026
October 30, 2026
Surry Hills
Sydney, Australia

MySQL Contributor Summit
November 4-5, 2026
Virtual

KubeCon + CloudNativeCon North America (VillageSQL is a sponsor)
November 9-12, 2026
Salt Lake City, Utah

PASS Data Community Summit
November 9-11, 2026
Hyatt Regency Seattle
Seattle, WA

PGConf.Asia 2026
November 17-18, 2026
Hong Kong

PGConf.PL 2026
November 24, 2026
ARCHE Dwór Uphagena
Gdańsk, Poland

AWS re:Invent 2026
November 30 – December 4, 2026
Las Vegas, NV

Open Source Summit Japan
December 7-9, 2026
Tokyo, Japan

CIDR 2027
January 24-27, 2027
Mövenpick Hotel Amsterdam City Centre
Amsterdam, The Netherlands

FOSDEM 2027
January 30-31, 2027
ULB Solbosch Campus
Brussels, Belgium

CERN PGDay 2027
February 12, 2027
CERN Council Chamber
Geneva, Switzerland

PGConf India 2027
March 2-5, 2027
Sheraton Grand Hotel at Brigade Gateway
Bengaluru, India

KubeCon + CloudNativeCon Europe 2027
March 15-18, 2027
Barcelona, Spain

Nordic PGDay 2027
March 16, 2027
Courtyard Kungsholmen
Stockholm, Sweden

SCaLE 24x
April 1-4, 2027
Pasadena Convention Center
Pasadena, CA

PgDay Boston 2027
April 10, 2027
Tufts University Joyce Cummings Center
Boston, MA

PGConf.dev 2027
May 11-14, 2027
Plaza Centre-Ville
Montréal, QC, Canada

]]> Tue, 06 Oct 2026 13:00:04 +0000 VillageSQL Movie Finder: semantic search over 1M films, inside MySQL https://askdba.net/?p=2991 https://askdba.net/movie-finder-semantic-search-over-1m-films-inside-mysql/ Type “a heist that goes wrong in a snowy town” and get back real films, with posters, in a few milliseconds. The vector search behind it runs inside MySQL, with no separate vector database. Movie Finder is the new demo app that ships with MyVector. It searches up to 1,035,695 TMDB movies by meaning, not keywords. One docker compose up gives you the whole thing: MySQL 9.7 with the MyVector component, a loader, and a small web app. Try it live: https://demo.myvector.online/ We built it to answer the questions people ask us most. Does vector search in MySQL hold up at a million rows? Can I mix it with ordinary WHERE filters? What happens when I insert a new row? The demo shows the answers on the page, with the SQL that ran under every result. Live: What you can do on the page Every feature on the page maps to a MyVector capability, and every result panel shows its SQL. Describe a movie in your own words and get the nearest films. Ask for “a boy wizard at a school of magic” and Harry Potter comes back first. Filter by genre, year, rating and language. These are plain SQL predicates combined with the vector search, and the page tells you which filtered-search path ran and why. Rank by similarity alone, or with a small boost for films many people have rated, so the famous match beats an obscure one at almost the same distance. Both are ordinary ORDER BY expressions. More like this: a film’s stored vector becomes the next query, in pure SQL. HNSW vs exact: run both side by side and compare speed and recall. Add a movie: a plain INSERT, searchable within about a second, with no index rebuild. (Not on live demo) Under the hood: live index details from myvector_index_status, a bar showing where each query’s time went, and a speed-vs-recall chart that sweeps ef_search from 10 to 640 against an exact scan. How it works The app embeds only your query text; every movie’s vector already sits in MySQL, so a search is one SQL round trip. The movie vectors come with the dataset, made by nomic-embed-text-v1.5 from each film’s title, tagline, and overview. The app uses the same model locally on the CPU, so there is no API key, and nothing leaves your machine. New rows need no rebuild. Adding a movie is a plain INSERT; MyVector’s binlog listener picks it up and adds it to the HNSW index, usually within a second. The SQL The whole index is declared in a column comment. This is the movies table’s vector column: embedding VARBINARY(3080) COMMENT 'MYVECTOR COLUMN type=HNSW,dim=768,size=...,M=16,ef=100,dist=Cosine,online=Y,idcol=id,threads=N' The loader inserts the rows, then builds the HNSW index with one call: CALL mysql.myvector_index_build('movies.movies.embedding', 'id'); A search turns the query vector into a nearest-first list of ids with myvector_ann_set(), then joins back to the table. JSON_TABLE keeps the order: SELECT m.title, myvector_distance(m.embedding, @q, 'Cosine') AS distance FROM (SELECT myvector_ann_set('movies.movies.embedding', 'id', @q, 'nn=10,ef_search=100') AS js) src, JSON_TABLE(src.js, '$[*]' COLUMNS (rank_no FOR ORDINALITY, id INT PATH '$')) nn JOIN movies.movies m ON m.id = nn.id ORDER BY nn.rank_no; “More like this” needs no embedding at all. It reads a film’s stored vector into the query variable: SELECT embedding INTO @q FROM movies.movies WHERE id = @movie_id; Filters pick one of two paths. The app counts each filter on its own index and takes the smallest count as an upper bound: 50,000 matches or fewer: it passes the matching keys as the fifth argument of myvector_ann_set, so HNSW searches only among them. More than that: it calls MYVECTOR_ANN_FILTERED, which takes the nearest candidates and keeps those that pass the filter. Run it yourself To run your own copy, from a clone of the repository: cd examples/movie-finder MOVIES=100k docker compose up # then open http://localhost:8080 The first start downloads the TMDB data, about 7 GB, once. MOVIES picks how many films to load, most-voted first: 10k, 100k (the default) or full. Measured on a 16-core Arm (Neoverse-N1) Linux host with MySQL 9.7.2: Step10k100kfull (1,035,695)Insert7 s41 s407 sBuild the HNSW index (16 threads)29 s33 s545 sHNSW search3–5 ms4–7 ms5–20 msExact search (scans every row)30 ms280 ms3 s warmRecall@10, HNSW vs exact100%100%100% At a million movies, HNSW answers in 5–20 ms where a full scan takes 3 seconds, with the same top 10. Recall is for the unfiltered test query; a broad genre filter (Drama) at full size gave 80% at ef_search 100, and the page lets you raise ef_search to trade time for recall. For the full profile, give Docker about 12 GB of memory and set MYSQL_BUFFER_POOL=6G. Running it on a remote server? The page and MySQL listen on localhost only. Forward the port instead of opening it: ssh -N -L 8080:127.0.0.1:8080 &lt;server&gt;. Version note: the demo uses features newer than the v1.26.9 images: filtered search, MYVECTOR_ANN_FILTERED, per-query ef_search, and online updates that survive a restart. Until the next release ships, the README shows how to build a local image from main. Data: the movie metadata and posters come from TMDB via a Hugging Face mirror. Your machine downloads it; it is not in the repository or any image. This is a non-commercial demo, not endorsed by TMDB. Try it and tell us Try the live demo first. To run your own, start with MOVIES=10k for a quick first look, then go to full to watch a million rows answer in milliseconds. The demo guide has screenshots and the other demos; the Movie Finder README has the full details. If a query surprises you, or you want a feature the page doesn’t show yet, open an issue on GitHub. A star helps other MySQL users find the project.

Type “a heist that goes wrong in a snowy town” and get back real films, with posters, in a few milliseconds. The vector search behind it runs inside MySQL, with no separate vector database.

Movie Finder is the new demo app that ships with MyVector. It searches up to 1,035,695 TMDB movies by meaning, not keywords. One docker compose up gives you the whole thing: MySQL 9.7 with the MyVector component, a loader, and a small web app.

Try it live: https://demo.myvector.online/

We built it to answer the questions people ask us most. Does vector search in MySQL hold up at a million rows? Can I mix it with ordinary WHERE filters? What happens when I insert a new row? The demo shows the answers on the page, with the SQL that ran under every result.

Live: What you can do on the page

Every feature on the page maps to a MyVector capability, and every result panel shows its SQL.

  • Describe a movie in your own words and get the nearest films. Ask for “a boy wizard at a school of magic” and Harry Potter comes back first.
  • Filter by genre, year, rating and language. These are plain SQL predicates combined with the vector search, and the page tells you which filtered-search path ran and why.
  • Rank by similarity alone, or with a small boost for films many people have rated, so the famous match beats an obscure one at almost the same distance. Both are ordinary ORDER BY expressions.
  • More like this: a film’s stored vector becomes the next query, in pure SQL.
  • HNSW vs exact: run both side by side and compare speed and recall.
  • Add a movie: a plain INSERT, searchable within about a second, with no index rebuild. (Not on live demo)
  • Under the hood: live index details from myvector_index_status, a bar showing where each query’s time went, and a speed-vs-recall chart that sweeps ef_search from 10 to 640 against an exact scan.

How it works

The app embeds only your query text; every movie’s vector already sits in MySQL, so a search is one SQL round trip. The movie vectors come with the dataset, made by nomic-embed-text-v1.5 from each film’s title, tagline, and overview. The app uses the same model locally on the CPU, so there is no API key, and nothing leaves your machine.

New rows need no rebuild. Adding a movie is a plain INSERT; MyVector’s binlog listener picks it up and adds it to the HNSW index, usually within a second.

The SQL

The whole index is declared in a column comment. This is the movies table’s vector column:

embedding VARBINARY(3080) COMMENT
  'MYVECTOR COLUMN type=HNSW,dim=768,size=...,M=16,ef=100,dist=Cosine,online=Y,idcol=id,threads=N'

The loader inserts the rows, then builds the HNSW index with one call:

CALL mysql.myvector_index_build('movies.movies.embedding', 'id');

A search turns the query vector into a nearest-first list of ids with myvector_ann_set(), then joins back to the table. JSON_TABLE keeps the order:

SELECT m.title, myvector_distance(m.embedding, @q, 'Cosine') AS distance
FROM (SELECT myvector_ann_set('movies.movies.embedding', 'id', @q,
                              'nn=10,ef_search=100') AS js) src,
     JSON_TABLE(src.js, '$[*]' COLUMNS (rank_no FOR ORDINALITY, id INT PATH '$')) nn
JOIN movies.movies m ON m.id = nn.id
ORDER BY nn.rank_no;

“More like this” needs no embedding at all. It reads a film’s stored vector into the query variable:

SELECT embedding INTO @q FROM movies.movies WHERE id = @movie_id;

Filters pick one of two paths. The app counts each filter on its own index and takes the smallest count as an upper bound:

  • 50,000 matches or fewer: it passes the matching keys as the fifth argument of myvector_ann_set, so HNSW searches only among them.
  • More than that: it calls MYVECTOR_ANN_FILTERED, which takes the nearest candidates and keeps those that pass the filter.

Run it yourself

To run your own copy, from a clone of the repository:

cd examples/movie-finder
MOVIES=100k docker compose up      # then open http://localhost:8080

The first start downloads the TMDB data, about 7 GB, once. MOVIES picks how many films to load, most-voted first: 10k, 100k (the default) or full. Measured on a 16-core Arm (Neoverse-N1) Linux host with MySQL 9.7.2:

Step10k100kfull (1,035,695)
Insert7 s41 s407 s
Build the HNSW index (16 threads)29 s33 s545 s
HNSW search3–5 ms4–7 ms5–20 ms
Exact search (scans every row)30 ms280 ms3 s warm
Recall@10, HNSW vs exact100%100%100%

At a million movies, HNSW answers in 5–20 ms where a full scan takes 3 seconds, with the same top 10. Recall is for the unfiltered test query; a broad genre filter (Drama) at full size gave 80% at ef_search 100, and the page lets you raise ef_search to trade time for recall. For the full profile, give Docker about 12 GB of memory and set MYSQL_BUFFER_POOL=6G.

Running it on a remote server? The page and MySQL listen on localhost only. Forward the port instead of opening it: ssh -N -L 8080:127.0.0.1:8080 <server>.

Version note: the demo uses features newer than the v1.26.9 images: filtered search, MYVECTOR_ANN_FILTERED, per-query ef_search, and online updates that survive a restart. Until the next release ships, the README shows how to build a local image from main.

Data: the movie metadata and posters come from TMDB via a Hugging Face mirror. Your machine downloads it; it is not in the repository or any image. This is a non-commercial demo, not endorsed by TMDB.

Try it and tell us

Try the live demo first. To run your own, start with MOVIES=10k for a quick first look, then go to full to watch a million rows answer in milliseconds. The demo guide has screenshots and the other demos; the Movie Finder README has the full details.

If a query surprises you, or you want a feature the page doesn’t show yet, open an issue on GitHub. A star helps other MySQL users find the project.

]]> Mon, 05 Oct 2026 20:30:03 +0000 Alkin Tezuysal mysql Technical ai llm myvector PGO or not PGO this is the dilemma. Step 3 http://www.tusacentral.com/joomla/index.php/mysql-blogs/271-pgo-or-not-pgo-this-is-the-dilemma-step-3 http://www.tusacentral.com/joomla/index.php/mysql-blogs/271-pgo-or-not-pgo-this-is-the-dilemma-step-3 Step 3: Why my sysbench-trained build loses and how to do it right. Given the topic complexity and the length of this article I have split it in 3 three different blog-post: What is PGO How PGO it works Why my sysbench-trained build loses and how to do it right. Three compounding reasons: Uncovered code gets pessimized. Sysbench-tpcc touches a narrow slice of mysqld. Every function with zero counts is treated as cold: GCC optimizes it for size, skips inlining, and shoves it into cold sections. But at runtime I still execute plenty of code my training never touched, such as purge, flushing, stats recalculation, error paths, different optimizer plans, connection churn. All of that is now running de-optimized code, and I pay icache penalties every time hot code calls into "cold" regions. This is why GCC added -fprofile-partial-training, without it, a narrow profile actively hurts everything outside it. High-concurrency instrumented runs produce corrupted or skewed profiles. GCC's profile counters are non-atomic by default. With 128–1024 threads hammering the same counters I get lost updates and internally inconsistent counts (I actually had "profile count data file corrupted/inconsistent" warnings at the -fprofile-use compile). I need to use -fprofile-update=atomic, which almost nobody sets and was at the beginning not aware of. In short my profile was garbage and a garbage profile is worse than no profile, consistent with my PGO build being slower than plain -O3 at 128 threads. Instrumentation distorts what "hot" means under contention. The instrumented binary is 2–10x slower, which shifts where threads pile up. Spin loops in mutexes and rw-locks record enormous counts, so the compiler lavishes optimization on waiting code instead of useful work. While at high thread counts my real bottleneck is lock contention and memory latency things branch layout can't fix. Do I have a way to merge the different profiles like MTR + sysbench?The answer is yes. For GCC it's simple to do so, because the runtime automatically merges profile data across multiple training runs against the same instrumented binary. I don't need a separate merge step like Clang does. What it does is that each time an instrumented binary exits, it writes its counters into files. If those files already exist (from a previous run), GCC's runtime adds the new counts to the existing ones rather than overwriting them. So if I run MTR first, then run sysbench against the same instrumented build with the same FPROFILE_DIR, the second run's counts accumulate on top of the first. The final profile reflects both workloads combined.  Combining MTR with a moderate-thread sysbench run for PGO training is helpful because the two workloads cover different, complementary dimensions of mysqld's behavior:  MTR sweeps broad functional breadth parser, optimizer, DDL, replication, error paths. However it runs almost entirely single-connection, so it never exercises the branches that only exist under real concurrency, like the contended slow-path of a latch, MVCC visibility checks against in-flight writers, lock-wait queuing, or redo-log group-commit batching; A moderate-concurrency sysbench run (something like 4–64 threads, enough to create actual simultaneous access without descending into the timing-distortion and counter-corruption problems), fills exactly that gap by giving those concurrency-only branches nonzero execution counts, which keeps the compiler from treating them as cold, since cold code paths get optimized for size instead of speed. So the merged profile ends up with both the wide code coverage MTR provides and the concurrent-path coverage MTR structurally can't, at the cost of only a modest, second-order improvement over MTR alone since PGO's overall gains are already small and this specific slice of the binary is a narrow fraction of total execution.   However even where it does help, I am stacking a small effect on top of a small effect. I have already found PGO vs non-PGO gives me under 5%. The incremental gain from better-covering a narrow slice of concurrency-only code within that is a second-order refinement. It is plausibly in the sub-1% range, quite  smaller than the run-to-run noise I had seen just from benchmark variance. Or at least that is what I now think, let me validate it.Commands to build the code: cmake ../mysql-9.7.2 \ -DCMAKE_INSTALL_PREFIX=/opt/mysql_templates/mysql-9.7.2-PGO-instrument \ -DCMAKE_BUILD_TYPE=Release \ -DENABLED_LOCAL_INFILE=1 \ -DWITH_FEDERATED_STORAGE_ENGINE=1 \ -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ -DWITH_PACKAGE_FLAGS=OFF \ -DCOMPILATION_COMMENT_SERVER="Marco compile 9.7.2-PGO instrument" \ -DCOMPILATION_COMMENT="Marco compile 9.7.2 PGO instrument" \ -DCMAKE_C_COMPILER=clang-20 \ -DCMAKE_CXX_COMPILER=clang++-20 \ -DCMAKE_C_FLAGS="-fuse-ld=lld" \ -DCMAKE_CXX_FLAGS="-fuse-ld=lld" \ -DFPROFILE_GENERATE=ON \ -DWITH_LTO=OFF \ -DFPROFILE_DIR=/opt/mysql_source/profile Run the mtr: perl mysql-test-run.pl --force --max-test-fail=0 --parallel=8 --suite=main,innodb,innodb_undo,binlog,rpl,perfschema,sys_vars   Then run the sysbench-tpcc test with 64 threads sysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307  --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1  --histogram --report_csv=yes  --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=600 --threads=64 --mysql-ssl=PREFERRED --mysql-ignore-errors=none  --reconnect=0  run And got the profile as [root@sm-blade03 profile]# llvm-profdata-20 show -detailed-summary /opt/mysql_source/profile/mysql.profdata /opt/mysql_source/profile/mysql.profdata Instrumentation level: IR entry_first = 0 instrument_loop_entries = 0 Total functions: 52403 Maximum function count: 289910820864 Maximum internal block count: 16881288068 Total number of blocks: 619050 Total count: 2473216025729 Detailed summary: 2 blocks (0.00%) with count &gt;= 289910820864 account for 1% of the total counts. 2 blocks (0.00%) with count &gt;= 289910820864 account for 10% of the total counts. 2 blocks (0.00%) with count &gt;= 289910820864 account for 20% of the total counts. 11 blocks (0.00%) with count &gt;= 16777125216 account for 30% of the total counts. 32 blocks (0.01%) with count &gt;= 7806039615 account for 40% of the total counts. 77 blocks (0.01%) with count &gt;= 3839409465 account for 50% of the total counts. 160 blocks (0.03%) with count &gt;= 2273821592 account for 60% of the total counts. 312 blocks (0.05%) with count &gt;= 1111890303 account for 70% of the total counts. 719 blocks (0.12%) with count &gt;= 365989756 account for 80% of the total counts. 2165 blocks (0.35%) with count &gt;= 85858392 account for 90% of the total counts. 4893 blocks (0.79%) with count &gt;= 24359463 account for 95% of the total counts. 16760 blocks (2.71%) with count &gt;= 2841200 account for 99% of the total counts. 45006 blocks (7.27%) with count &gt;= 148881 account for 99.9% of the total counts. 96619 blocks (15.61%) with count &gt;= 8640 account for 99.99% of the total counts. 157023 blocks (25.37%) with count &gt;= 960 account for 99.999% of the total counts. 220001 blocks (35.54%) with count &gt;= 91 account for 99.9999% of the total counts Checking on how counts concentrate: just 2 blocks account for 20% of all executed instructions across the entire training run (almost certainly a tight InnoDB buffer-pool/redo-log/lock_manager loop  counts in the hundreds of billions), while 90% of total execution volume is concentrated in only 2,165 blocks (0.35% of all covered blocks).  That's the "hot core" PGO is designed to find and optimize aggressively. Meanwhile the long tail,  the other 96%+ of blocks, still has nonzero counts (this is a sparse profile, so anything appearing here was actually executed at least once). Meaning a huge amount of MTR's functional-path breadth got captured even though it's numerically dwarfed by sysbench's tight hot loops. This is the ideal shape for a merged profile: a small, extremely hot core (from sustained sysbench load) sitting on top of broad, low-frequency-but-nonzero coverage (from MTR's functional sweep).  If MTR had contributed nothing, I would have a much flatter, narrower distribution with far fewer total functions covered. If sysbench had swamped everything with no MTR contribution, I would have a similar shape but with a much smaller "Total functions" number, since sysbench-tpcc only touches a fraction of mysqld's total surface. Command to build the final optimized binaries: llvm-profdata-20 merge -sparse /opt/mysql_source/profile/*.profraw -o /opt/mysql_source/profile/mysql.profdata cmake ../mysql-9.7.2 \ -DCMAKE_INSTALL_PREFIX=/opt/mysql_templates/mysql-9.7.2-PGO-optimized-MTR-sysbench \ -DCMAKE_BUILD_TYPE=Release \ -DENABLED_LOCAL_INFILE=1 \ -DWITH_FEDERATED_STORAGE_ENGINE=1 \ -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ -DWITH_PACKAGE_FLAGS=OFF \ -DCOMPILATION_COMMENT_SERVER="Marco compile 9.7.2-PGO optimized MTR+sysbench" \ -DCOMPILATION_COMMENT="Marco compile 9.7.2 PGO optimized MTR+sysbench" \ -DCMAKE_C_COMPILER=clang-20 \ -DCMAKE_CXX_COMPILER=clang++-20 \ -DFPROFILE_USE=ON \ -DFPROFILE_DIR=/opt/mysql_source/profile/mysql.profdata \ -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_LZ4=system -DWITH_ICU=system \ -DWITH_NUMA=ON -DWITH_LTO=ON -DWITH_LD=lld -DWITH_SYSTEMD=1 \ -DWITH_UNIT_TESTS=OFF -DWITH_ROUTER=OFF -DMYSQL_MAINTAINER_MODE=OFF Re-running the test I got this:                            As expected the benefit I got was minimal, something was there but that disappeared while the concurrency increased.   Conclusions Non-PGO vs PGO comparison: we had a 12% increase with low concurrency. But when simulating a more realistic load with higher contention the win becomes smaller and smaller, under 2%. Training-workload choice mattered a lot: a sysbench-tpcc-trained PGO build ended up slower than a non-PGO. After switching to a merged MTR + moderate-concurrency-sysbench training profile and re-running, the gain was still minimal, and it shrank further as concurrency increased.   Why the sysbench-only build lost Sysbench-tpcc only exercises a narrow slice of mysqld, so everything outside that slice (purge, flushing, stats, error paths, alternate optimizer plans) gets pessimized as "cold" code. At 128–1024 threads, GCC's non-atomic profile counters get corrupted under contention unless I explicitly set -fprofile-update=atomic,a garbage profile is worse than no profile at all. Instrumentation overhead (2–10x slowdown) distorts what looks "hot" under load mutex/rwlock spin loops dominate the counts, so the compiler optimizes waiting code instead of real work, while the actual bottleneck (lock contention, memory latency) is something branch layout can't fix anyway. Bottom line: PGO for MySQL/Percona Server binaries does work in the sense that the mechanism is sound whole-binary function reordering for something the size of mysqld is a legitimate, often the single biggest, win in PGO generally.  But empirically here the payoff was consistently small (sub-5%, trending toward sub-1% for the concurrency-specific refinement), fragile to training-workload choice, fragile to build-flag correctness (atomic counters, -fprofile-partial-training), and it erodes further as thread count rises,  which is exactly what happens in production and what we care about most. So my practical answer: it's not a clear "yes, always compile with PGO." It's more a “maybe, and only if you get every detail right". Correct broad-coverage training data (MTR, ideally merged with moderate-concurrency sysbench), atomic profile counters, and realistic expectations that the win is marginal and shrinks under heavy concurrency.  Get any of those wrong and you can end up worse than a plain build. Given the size of the benefit versus the number of ways to mess up the training methodology, PGO reads more like a niche optimization for a well-controlled build pipeline than a default you'd flip on broadly. I would be more than happy to prove wrong and I am eager to get other people's feedback, so please test, test, test and let me know.    Happy MySQL to everyone Go to: What is PGO How PGO it works Step 3: Why my sysbench-trained build loses and how to do it right.

Given the topic complexity and the length of this article I have split it in 3 three different blog-post:

  1. What is PGO
  2. How PGO it works
  3. Why my sysbench-trained build loses and how to do it right.

Three compounding reasons:

  1. Uncovered code gets pessimized.
    Sysbench-tpcc touches a narrow slice of mysqld.
    Every function with zero counts is treated as cold: GCC optimizes it for size, skips inlining, and shoves it into cold sections.
    But at runtime I still execute plenty of code my training never touched, such as purge, flushing, stats recalculation, error paths, different optimizer plans, connection churn.
    All of that is now running de-optimized code, and I pay icache penalties every time hot code calls into "cold" regions.
    This is why GCC added -fprofile-partial-training, without it, a narrow profile actively hurts everything outside it.
  2. High-concurrency instrumented runs produce corrupted or skewed profiles.
    GCC's profile counters are non-atomic by default.
    With 128–1024 threads hammering the same counters I get lost updates and internally inconsistent counts (I actually had "profile count data file corrupted/inconsistent" warnings at the -fprofile-use compile).
    I need to use -fprofile-update=atomic, which almost nobody sets and was at the beginning not aware of.
    In short my profile was garbage and a garbage profile is worse than no profile, consistent with my PGO build being slower than plain -O3 at 128 threads.
  3. Instrumentation distorts what "hot" means under contention.
    The instrumented binary is 2–10x slower, which shifts where threads pile up. Spin loops in mutexes and rw-locks record enormous counts, so the compiler lavishes optimization on waiting code instead of useful work. While at high thread counts my real bottleneck is lock contention and memory latency things branch layout can't fix.

Do I have a way to merge the different profiles like MTR + sysbench?
The answer is yes. For GCC it's simple to do so, because the runtime automatically merges profile data across multiple training runs against the same instrumented binary. I don't need a separate merge step like Clang does.

What it does is that each time an instrumented binary exits, it writes its counters into files. If those files already exist (from a previous run), GCC's runtime adds the new counts to the existing ones rather than overwriting them.
So if I run MTR first, then run sysbench against the same instrumented build with the same FPROFILE_DIR, the second run's counts accumulate on top of the first. The final profile reflects both workloads combined. 

Combining MTR with a moderate-thread sysbench run for PGO training is helpful because the two workloads cover different, complementary dimensions of mysqld's behavior: 

  • MTR sweeps broad functional breadth parser, optimizer, DDL, replication, error paths. However it runs almost entirely single-connection, so it never exercises the branches that only exist under real concurrency, like the contended slow-path of a latch, MVCC visibility checks against in-flight writers, lock-wait queuing, or redo-log group-commit batching;
  • A moderate-concurrency sysbench run (something like 4–64 threads, enough to create actual simultaneous access without descending into the timing-distortion and counter-corruption problems), fills exactly that gap by giving those concurrency-only branches nonzero execution counts, which keeps the compiler from treating them as cold, since cold code paths get optimized for size instead of speed.

So the merged profile ends up with both the wide code coverage MTR provides and the concurrent-path coverage MTR structurally can't, at the cost of only a modest, second-order improvement over MTR alone since PGO's overall gains are already small and this specific slice of the binary is a narrow fraction of total execution.

 

However even where it does help, I am stacking a small effect on top of a small effect. I have already found PGO vs non-PGO gives me under 5%.
The incremental gain from better-covering a narrow slice of concurrency-only code within that is a second-order refinement.
It is plausibly in the sub-1% range, quite  smaller than the run-to-run noise I had seen just from benchmark variance.

Or at least that is what I now think, let me validate it.

Commands to build the code:

cmake ../mysql-9.7.2 \
-DCMAKE_INSTALL_PREFIX=/opt/mysql_templates/mysql-9.7.2-PGO-instrument \
-DCMAKE_BUILD_TYPE=Release \
-DENABLED_LOCAL_INFILE=1 \
-DWITH_FEDERATED_STORAGE_ENGINE=1 \
-DWITH_ARCHIVE_STORAGE_ENGINE=1 \
-DWITH_PACKAGE_FLAGS=OFF \
-DCOMPILATION_COMMENT_SERVER="Marco compile 9.7.2-PGO instrument" \
-DCOMPILATION_COMMENT="Marco compile 9.7.2 PGO instrument" \
-DCMAKE_C_COMPILER=clang-20 \
-DCMAKE_CXX_COMPILER=clang++-20 \
-DCMAKE_C_FLAGS="-fuse-ld=lld" \
-DCMAKE_CXX_FLAGS="-fuse-ld=lld" \
-DFPROFILE_GENERATE=ON \
-DWITH_LTO=OFF \
-DFPROFILE_DIR=/opt/mysql_source/profile

Run the mtr:

perl mysql-test-run.pl --force --max-test-fail=0 --parallel=8 --suite=main,innodb,innodb_undo,binlog,rpl,perfschema,sys_vars

 

Then run the sysbench-tpcc test with 64 threads

sysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307  --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1  --histogram --report_csv=yes  --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=600 --threads=64 --mysql-ssl=PREFERRED --mysql-ignore-errors=none  --reconnect=0  run

And got the profile as

[root@sm-blade03 profile]# llvm-profdata-20 show -detailed-summary /opt/mysql_source/profile/mysql.profdata
/opt/mysql_source/profile/mysql.profdata
Instrumentation level: IR entry_first = 0 instrument_loop_entries = 0
Total functions: 52403
Maximum function count: 289910820864
Maximum internal block count: 16881288068
Total number of blocks: 619050
Total count: 2473216025729
Detailed summary:
2 blocks (0.00%) with count >= 289910820864 account for 1% of the total counts.
2 blocks (0.00%) with count >= 289910820864 account for 10% of the total counts.
2 blocks (0.00%) with count >= 289910820864 account for 20% of the total counts.
11 blocks (0.00%) with count >= 16777125216 account for 30% of the total counts.
32 blocks (0.01%) with count >= 7806039615 account for 40% of the total counts.
77 blocks (0.01%) with count >= 3839409465 account for 50% of the total counts.
160 blocks (0.03%) with count >= 2273821592 account for 60% of the total counts.
312 blocks (0.05%) with count >= 1111890303 account for 70% of the total counts.
719 blocks (0.12%) with count >= 365989756 account for 80% of the total counts.
2165 blocks (0.35%) with count >= 85858392 account for 90% of the total counts.
4893 blocks (0.79%) with count >= 24359463 account for 95% of the total counts.
16760 blocks (2.71%) with count >= 2841200 account for 99% of the total counts.
45006 blocks (7.27%) with count >= 148881 account for 99.9% of the total counts.
96619 blocks (15.61%) with count >= 8640 account for 99.99% of the total counts.
157023 blocks (25.37%) with count >= 960 account for 99.999% of the total counts.
220001 blocks (35.54%) with count >= 91 account for 99.9999% of the total counts

Checking on how counts concentrate: just 2 blocks account for 20% of all executed instructions across the entire training run (almost certainly a tight InnoDB buffer-pool/redo-log/lock_manager loop  counts in the hundreds of billions), while 90% of total execution volume is concentrated in only 2,165 blocks (0.35% of all covered blocks). 

That's the "hot core" PGO is designed to find and optimize aggressively. Meanwhile the long tail,  the other 96%+ of blocks, still has nonzero counts (this is a sparse profile, so anything appearing here was actually executed at least once).

Meaning a huge amount of MTR's functional-path breadth got captured even though it's numerically dwarfed by sysbench's tight hot loops.

This is the ideal shape for a merged profile: a small, extremely hot core (from sustained sysbench load) sitting on top of broad, low-frequency-but-nonzero coverage (from MTR's functional sweep). 

If MTR had contributed nothing, I would have a much flatter, narrower distribution with far fewer total functions covered.
 If sysbench had swamped everything with no MTR contribution, I would have a similar shape but with a much smaller "Total functions" number, since sysbench-tpcc only touches a fraction of mysqld's total surface.

Command to build the final optimized binaries:

llvm-profdata-20 merge -sparse /opt/mysql_source/profile/*.profraw -o /opt/mysql_source/profile/mysql.profdata

cmake ../mysql-9.7.2 \
 -DCMAKE_INSTALL_PREFIX=/opt/mysql_templates/mysql-9.7.2-PGO-optimized-MTR-sysbench \
  -DCMAKE_BUILD_TYPE=Release \
  -DENABLED_LOCAL_INFILE=1 \
  -DWITH_FEDERATED_STORAGE_ENGINE=1 \
  -DWITH_ARCHIVE_STORAGE_ENGINE=1 \
  -DWITH_PACKAGE_FLAGS=OFF \
  -DCOMPILATION_COMMENT_SERVER="Marco compile 9.7.2-PGO optimized MTR+sysbench" \
  -DCOMPILATION_COMMENT="Marco compile 9.7.2 PGO optimized MTR+sysbench" \
  -DCMAKE_C_COMPILER=clang-20 \
  -DCMAKE_CXX_COMPILER=clang++-20 \
  -DFPROFILE_USE=ON \
  -DFPROFILE_DIR=/opt/mysql_source/profile/mysql.profdata \
  -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_LZ4=system -DWITH_ICU=system \
  -DWITH_NUMA=ON -DWITH_LTO=ON -DWITH_LD=lld -DWITH_SYSTEMD=1 \
  -DWITH_UNIT_TESTS=OFF -DWITH_ROUTER=OFF -DMYSQL_MAINTAINER_MODE=OFF

Re-running the test I got this:

 

  

 

 

 

 

 

 

 

 

 

 

 

As expected the benefit I got was minimal, something was there but that disappeared while the concurrency increased.

 

Conclusions

  • Non-PGO vs PGO comparison: we had a 12% increase with low concurrency. But when simulating a more realistic load with higher contention the win becomes smaller and smaller, under 2%.
  • Training-workload choice mattered a lot: a sysbench-tpcc-trained PGO build ended up slower than a non-PGO.
  • After switching to a merged MTR + moderate-concurrency-sysbench training profile and re-running, the gain was still minimal, and it shrank further as concurrency increased.

 

Why the sysbench-only build lost

Sysbench-tpcc only exercises a narrow slice of mysqld, so everything outside that slice (purge, flushing, stats, error paths, alternate optimizer plans) gets pessimized as "cold" code.

At 128–1024 threads, GCC's non-atomic profile counters get corrupted under contention unless I explicitly set -fprofile-update=atomic,a garbage profile is worse than no profile at all.

Instrumentation overhead (2–10x slowdown) distorts what looks "hot" under load mutex/rwlock spin loops dominate the counts, so the compiler optimizes waiting code instead of real work, while the actual bottleneck (lock contention, memory latency) is something branch layout can't fix anyway.

Bottom line: PGO for MySQL/Percona Server binaries does work in the sense that the mechanism is sound whole-binary function reordering for something the size of mysqld is a legitimate, often the single biggest, win in PGO generally. 

But empirically here the payoff was consistently small (sub-5%, trending toward sub-1% for the concurrency-specific refinement), fragile to training-workload choice, fragile to build-flag correctness (atomic counters, -fprofile-partial-training), and it erodes further as thread count rises,  which is exactly what happens in production and what we care about most.

So my practical answer: it's not a clear "yes, always compile with PGO." It's more a “maybe, and only if you get every detail right". Correct broad-coverage training data (MTR, ideally merged with moderate-concurrency sysbench), atomic profile counters, and realistic expectations that the win is marginal and shrinks under heavy concurrency. 

Get any of those wrong and you can end up worse than a plain build.
Given the size of the benefit versus the number of ways to mess up the training methodology, PGO reads more like a niche optimization for a well-controlled build pipeline than a default you'd flip on broadly.

I would be more than happy to prove wrong and I am eager to get other people's feedback, so please test, test, test and let me know. 

 

Happy MySQL to everyone

Go to:

What is PGO

How PGO it works

]]> Mon, 05 Oct 2026 16:15:24 +0000 Marco Tusa MySQL PGO or not PGO this is the dilemma. Step 2 http://www.tusacentral.com/joomla/index.php/mysql-blogs/270-pgo-or-not-pgo-this-is-the-dilemma-step-2 http://www.tusacentral.com/joomla/index.php/mysql-blogs/270-pgo-or-not-pgo-this-is-the-dilemma-step-2 Step 2: How it works Given the topic complexity and the length of this article I have split it in 3 three different blog-post: What is PGO How PGO it works Why my sysbench-trained build loses and how to do it right. How PGO works: PGO is a two-pass build. First pass compiles with instrumentation (-fprofile-generate): every basic block and branch gets a counter. You run a training workload, counters are dumped to profraw files. Second pass recompiles using those counts to drive inlining decisions, branch layout (hot path falls through, cold path jumps away), hot/cold function splitting, code ordering for icache/iTLB locality, loop unrolling, and indirect-call promotion. Crucially, PGO is not "make the trained workload fast" it's "tell the compiler which code is hot and which is cold, and let it reshape the whole binary accordingly."   But how does it work? Phase 1: what actually gets recorded. The instrumented binary has counters injected at compile time one per edge in the control-flow graph, not just per function. So for every branch, every loop back-edge, every call site, there's a counter that increments each time execution takes that path. This is finer-grained than "function X was called N times" it's "when we reached this branch, we went left 950,000 times and right 50 times." That per-edge granularity is what lets the second phase make surgical decisions rather than just "function X is hot, function Y is cold." Phase 2: what the compiler does with those counts. Several distinct transformations, all driven by the same counter data: Inlining. Normally the compiler inlines based on static heuristics: function size call-site count estimated cost/benefit.  With profile data it can override those heuristics: a call site executed millions of times gets inlined even if it looks "too expensive" by static cost rules, because the runtime benefit clearly outweighs the code-size cost.  A call site that's technically inlinable but sits in dead-cold code gets left as a real call inlining it would only bloat the binary for no benefit. Branch layout. Every "if" in our code compiles down to a branch instruction with two possible outcomes:  continue straight to the next instruction jump somewhere else.  Continuing straight is basically free; the CPU is already fetching instructions in order, so there's no extra cost. Jumping is not free: the CPU has to guess in advance which way a branch will go so it can keep fetching ahead of time, and if it guesses wrong, it has to throw away the work it already queued up and start over from the right place. That is the "pipeline bubble," a small stall. So the "straight through" path is cheap and the "jump elsewhere" path carries a penalty when the guess is wrong.  The compiler arranges the hot path as the fall-through and pushes the cold path out of line; literally relocated to a separate location in the binary, often into a .text.unlikely section. So an if (unlikely_error_condition) { ...rare handling... } block doesn't sit inline interrupting the hot path anymore; it is moved somewhere else entirely, and the hot path becomes a straight run of instructions with no diversion. To be clear, reordering our if/else in the source code usually doesn't change how the compiler lays out the machine code. Optimizing compilers decide branch layout themselves based on either profile data (PGO) or static heuristics, not on which branch we happened to write first in the source. So swapping the order of our if blocks by hand generally has little to no effect on the compiled result. Hot/cold function splitting. This is the same idea applied within a single function. A function might have a hot core loop and a rarely-hit error-handling tail. The compiler physically splits the function into two pieces:  the hot part stays in .text.hot the cold part moves to .text.unlikely.  The function still works identically (a jump connects them when needed), but now the hot part is smaller and denser, so more of it fits in an instruction-cache line, and cold code that's almost never touched isn't wasting icache space sitting next to it. Whole-binary function reordering.This is where "reshape the whole binary" becomes literal. At link time (especially with LTO, which MySQL's PGO build enables), functions get physically reordered in the final executable so that functions which call each other frequently, or execute in sequence during a hot workload, are placed near each other in memory. This maximizes instruction-cache and iTLB locality; the CPU's fetch unit is pulling in a tight cluster of hot functions instead of jumping all over a 100+MB binary. For something the size of mysqld, this is often the single biggest win, because normal builds place functions in whatever order the source files happen to be compiled, which has no relationship to runtime call patterns. Indirect-call promotion. If profile data shows a virtual call or function-pointer call resolves to the same target the overwhelming majority of the time (common in C++ with vtables, e.g. a storage-engine interface with basically only InnoDB registered), the compiler inserts a guarded direct call: "if target == this specific address, call it directly and skip the indirect jump; otherwise fall back to the indirect call." Direct calls are cheaper and more predictable for the branch predictor than an indirect jump through a table. Register allocation and code density trade-offs. Hot code gets compiled favoring speed, more aggressive unrolling, more registers dedicated to hot-path values. Cold code, especially with -fprofile-partial-training and cold-path treatment, gets compiled favoring size, fewer registers, less unrolling because it barely executes. So runtime cost there is irrelevant but its footprint in the binary is not free (it still occupies disk/page-cache space and can evict hot lines from cache if placed carelessly, which is exactly why it gets segregated into .text.unlikely rather than just left unoptimized in place). Switch/jump-table lowering. A switch statement with many cases can be compiled as a jump table (fast, O(1), but requires a full table load and indirect jump) or as a cascade of compares (slower per-case but better branch prediction if one case dominates). Profile data tells the compiler which case actually dominates in practice and picks accordingly. Given the above,  "reshape the whole binary" is not metaphorical. The compiler is redrawing the physical layout of machine code in the executable: which instructions are adjacent to which, which code sections are hot and packed tightly versus cold and shoved to the side, which calls are direct versus indirect, and where the CPU's fetch/prediction effort gets spent. None of this requires the values processed during training to resemble production traffic. It only requires the shape of control flow. Which branches, functions, and paths are frequently exercised to resemble production traffic. That's the core reason MTR's broad-but-different-data coverage transfers well: it walks nearly every code path in mysqld even though the actual queries and data are nothing like a TPCC workload, and control-flow shape is exactly what PGO optimizes for.   Why my sysbench-trained build loses and how to do it right. Step 2: How it works

Given the topic complexity and the length of this article I have split it in 3 three different blog-post:

  1. What is PGO
  2. How PGO it works
  3. Why my sysbench-trained build loses and how to do it right.

How PGO works: PGO is a two-pass build. First pass compiles with instrumentation (-fprofile-generate): every basic block and branch gets a counter.
You run a training workload, counters are dumped to profraw files.
Second pass recompiles using those counts to drive inlining decisions, branch layout (hot path falls through, cold path jumps away), hot/cold function splitting, code ordering for icache/iTLB locality, loop unrolling, and indirect-call promotion.
Crucially, PGO is not "make the trained workload fast" it's "tell the compiler which code is hot and which is cold, and let it reshape the whole binary accordingly."

 

But how does it work?

Phase 1: what actually gets recorded. The instrumented binary has counters injected at compile time one per edge in the control-flow graph, not just per function.
So for every branch, every loop back-edge, every call site, there's a counter that increments each time execution takes that path.
This is finer-grained than "function X was called N times" it's "when we reached this branch, we went left 950,000 times and right 50 times."
That per-edge granularity is what lets the second phase make surgical decisions rather than just "function X is hot, function Y is cold."

Phase 2: what the compiler does with those counts. Several distinct transformations, all driven by the same counter data:

Inlining.
Normally the compiler inlines based on static heuristics:

  • function size call-site count
  • estimated cost/benefit. 

With profile data it can override those heuristics: a call site executed millions of times gets inlined even if it looks "too expensive" by static cost rules, because the runtime benefit clearly outweighs the code-size cost. 

A call site that's technically inlinable but sits in dead-cold code gets left as a real call inlining it would only bloat the binary for no benefit.

Branch layout.
Every "if" in our code compiles down to a branch instruction with two possible outcomes: 

  • continue straight to the next instruction
  • jump somewhere else. 

Continuing straight is basically free; the CPU is already fetching instructions in order, so there's no extra cost.
Jumping is not free: the CPU has to guess in advance which way a branch will go so it can keep fetching ahead of time, and if it guesses wrong, it has to throw away the work it already queued up and start over from the right place.

That is the "pipeline bubble," a small stall. So the "straight through" path is cheap and the "jump elsewhere" path carries a penalty when the guess is wrong. 

The compiler arranges the hot path as the fall-through and pushes the cold path out of line; literally relocated to a separate location in the binary, often into a .text.unlikely section.
So an if (unlikely_error_condition) { ...rare handling... } block doesn't sit inline interrupting the hot path anymore; it is moved somewhere else entirely, and the hot path becomes a straight run of instructions with no diversion.

To be clear, reordering our if/else in the source code usually doesn't change how the compiler lays out the machine code. Optimizing compilers decide branch layout themselves based on either profile data (PGO) or static heuristics, not on which branch we happened to write first in the source. So swapping the order of our if blocks by hand generally has little to no effect on the compiled result.

Hot/cold function splitting.
This is the same idea applied within a single function.
A function might have a hot core loop and a rarely-hit error-handling tail.
The compiler physically splits the function into two pieces: 

  • the hot part stays in .text.hot
  • the cold part moves to .text.unlikely. 

The function still works identically (a jump connects them when needed), but now the hot part is smaller and denser, so more of it fits in an instruction-cache line, and cold code that's almost never touched isn't wasting icache space sitting next to it.

Whole-binary function reordering.
This is where "reshape the whole binary" becomes literal.
At link time (especially with LTO, which MySQL's PGO build enables), functions get physically reordered in the final executable so that functions which call each other frequently, or execute in sequence during a hot workload, are placed near each other in memory.
This maximizes instruction-cache and iTLB locality; the CPU's fetch unit is pulling in a tight cluster of hot functions instead of jumping all over a 100+MB binary.
For something the size of mysqld, this is often the single biggest win, because normal builds place functions in whatever order the source files happen to be compiled, which has no relationship to runtime call patterns.

Indirect-call promotion.
If profile data shows a virtual call or function-pointer call resolves to the same target the overwhelming majority of the time (common in C++ with vtables, e.g. a storage-engine interface with basically only InnoDB registered), the compiler inserts a guarded direct call: "if target == this specific address, call it directly and skip the indirect jump; otherwise fall back to the indirect call."
Direct calls are cheaper and more predictable for the branch predictor than an indirect jump through a table.

Register allocation and code density trade-offs.
Hot code gets compiled favoring speed, more aggressive unrolling, more registers dedicated to hot-path values.
Cold code, especially with -fprofile-partial-training and cold-path treatment, gets compiled favoring size, fewer registers, less unrolling because it barely executes. So runtime cost there is irrelevant but its footprint in the binary is not free (it still occupies disk/page-cache space and can evict hot lines from cache if placed carelessly, which is exactly why it gets segregated into .text.unlikely rather than just left unoptimized in place).

Switch/jump-table lowering.
A switch statement with many cases can be compiled as a jump table (fast, O(1), but requires a full table load and indirect jump) or as a cascade of compares (slower per-case but better branch prediction if one case dominates).
Profile data tells the compiler which case actually dominates in practice and picks accordingly.

Given the above,  "reshape the whole binary" is not metaphorical. The compiler is redrawing the physical layout of machine code in the executable: which instructions are adjacent to which, which code sections are hot and packed tightly versus cold and shoved to the side, which calls are direct versus indirect, and where the CPU's fetch/prediction effort gets spent.
None of this requires the values processed during training to resemble production traffic. It only requires the shape of control flow.
Which branches, functions, and paths are frequently exercised to resemble production traffic.
That's the core reason MTR's broad-but-different-data coverage transfers well: it walks nearly every code path in mysqld even though the actual queries and data are nothing like a TPCC workload, and control-flow shape is exactly what PGO optimizes for.

 

Why my sysbench-trained build loses and how to do it right.

]]> Mon, 05 Oct 2026 16:13:01 +0000 Marco Tusa MySQL PGO or not PGO this is the dilemma. Step 1 http://www.tusacentral.com/joomla/index.php/mysql-blogs/269-pgo-or-not-pgo-this-is-the-dilemma-step-1 http://www.tusacentral.com/joomla/index.php/mysql-blogs/269-pgo-or-not-pgo-this-is-the-dilemma-step-1 Given the topic complexity and the length of this article I have split it in 3 three different blog-post: What is PGO How PGO it works Why my sysbench-trained build loses and how to do it right. Step 1: What is PGO PGO (Profile-Guided Optimization) is a two-pass compilation technique.  First, we build the program with instrumentation that adds counters to every branch, loop, and call site. Next, we run it against a representative training workload so these counters can record which code paths are actually hot versus cold. We then recompile the program using that data to drive the compiler's decisions. With this profiling data, the compiler inlines hot call sites more aggressively and lays out hot paths as straight-line, fall-through code while pushing cold paths out of the way. Finally, it physically reorders functions in the binary so frequently interacting hot code sits close together for better instruction-cache locality, and converts frequently resolved indirect calls into direct ones.The result isn't "make the training run fast" but rather "reshape the entire binary's layout around real execution patterns," which is why the quality and representativeness of the training workload matters so much to whether PGO actually helps.  It is nice to be wrong Not so far ago I was wondering if having Percona Server compiled with PGO default is a good idea or not. Then I started to do some tests and I end up with this: If I compare non PGO with PGO release I can see an optimization, minimal below 5% but is there. If I compile MySQL using PGO and use as sample the recording of a specific test say sysbench-tpcc my compile will always be slower than my compile without PGO, no mater how many threads I use during recording, I tried from 128 to 1024.                           Given my understanding of PGO was that it should be the other way around I was a bit disoriented. So I decided to read a bit and get a better understanding of what PGO really means/does and if it makes sense or not.   How PGO it works Given the topic complexity and the length of this article I have split it in 3 three different blog-post:

  1. What is PGO
  2. How PGO it works
  3. Why my sysbench-trained build loses and how to do it right.

Step 1: What is PGO

PGO (Profile-Guided Optimization) is a two-pass compilation technique. 

First, we build the program with instrumentation that adds counters to every branch, loop, and call site.
Next, we run it against a representative training workload so these counters can record which code paths are actually hot versus cold.
We then recompile the program using that data to drive the compiler's decisions.

With this profiling data, the compiler inlines hot call sites more aggressively and lays out hot paths as straight-line, fall-through code while pushing cold paths out of the way.
Finally, it physically reorders functions in the binary so frequently interacting hot code sits close together for better instruction-cache locality, and converts frequently resolved indirect calls into direct ones.

The result isn't "make the training run fast" but rather "reshape the entire binary's layout around real execution patterns," which is why the quality and representativeness of the training workload matters so much to whether PGO actually helps. 

It is nice to be wrong

Not so far ago I was wondering if having Percona Server compiled with PGO default is a good idea or not. Then I started to do some tests and I end up with this:

If I compare non PGO with PGO release I can see an optimization, minimal below 5% but is there.

If I compile MySQL using PGO and use as sample the recording of a specific test say sysbench-tpcc my compile will always be slower than my compile without PGO, no mater how many threads I use during recording, I tried from 128 to 1024.

Percona server no pg vs ps marco sysbench pgo

 

 

 

 

 

 

 

 

 

 

 

 

 

Given my understanding of PGO was that it should be the other way around I was a bit disoriented. So I decided to read a bit and get a better understanding of what PGO really means/does and if it makes sense or not.

 

How PGO it works

]]> Mon, 05 Oct 2026 16:07:41 +0000 Marco Tusa MySQL Thread Pool in Percona Server and MySQL (Part 1) https://www.percona.com/?p=54362 https://www.percona.com/blog/thread-pool-in-percona-server-and-mysql-part-1/ Thread Pool in Percona Server and MySQL (Part 1) A new version of Community MySQL Server July release contains two versions 26.7.0 and 9.7.2. It is an important milestone because it finally brings a Thread Pooling feature to the community. The official MySQL Server Thread Pool plugin existed for a long time, but was available exclusively for MySQL Enterprise Edition. Before MySQL 26.7.0 / 9.7.2 the most popular free thread pooling enabled alternatives were: Percona Server for MySQL, which shipped Thread Pool functionality as a plugin MariaDB with the Thread Pooling built into the server Third-party plugins such as xiezhenye/mysql-plugin-threadpool available on GitHub Now we will be able to see what benefits the thread pooling mechanism brings to MySQL Server and compare it with the implementation of the thread pool in Percona Server for MySQL. Part 1 will explain the basics of thread pooling and analyze the efficiency of its implementation in Percona Server for MySQL. The performance comparison between Percona Server and MySQL Server is done in Part 2. What is Thread Pooling and why is it needed? Thread Pooling is a technique that reuses a fixed number of pre-created threads to handle multiple client connections and execute statements inside a database. By default, MySQL creates a dedicated thread whenever a client connects, runs the queries, and then destroys the thread when finished. When connection counts grow very large, this default method causes performance to drop significantly because creating, destroying, and constantly switching between thousands of threads wastes system resources and causes contention. This blog post from Vadim Tkachenko explains the working of thread pool, which gives an easy to understand similarity with the car traffic: SimCity outages, traffic control and Thread Pool for MySQL More information about it can be found in Percona Documentation. WARNING: the post is several pages long because it looks into different aspects of thread pooling and compares the effect of various options. If you do not want to dive into the details, feel free to read the SPOILER below and proceed to the Conclusion section. SPOILER: Thread Pool is not a universal database booster, which always brings performance up. Like every advanced tool it should be used with full understanding of the goals and objectives. Otherwise the results might be far from perfect. How it works Thread Groups: The pool divides threads into distinct groups that are assigned to a specific CPU core. For this reason the number of groups often matches the number of CPU cores. Round-Robin Assignment: As client connections come in, they are distributed evenly across these groups. Listening and Queuing: Inside each group, a listener thread monitors incoming queries and places them into either a high-priority queue (such as queries within an active transaction) or a normal/low-priority queue. Execution and Reuse: Worker threads pick up queries from the queues (handling high-priority items first), execute them, and then wait to handle the next request instead of shutting down. See also: Core concepts and usage examples Priority connection scheduling Thread Pool configuration: In this research we will be sweeping across the range of various parameters of the Thread Pool and see how the performance of the server is affected. The base line will be the server configuration when the thread pool is not enabled. This is done by setting thread_handling=one-thread-per-connection. When the pooling is enabled (thread_handling=pool-of-threads) the benchmark will set different values for the following variables that control the thread pool behavior: Name Sweep range Description thread_pool_size 10 / 20 / 40 / 80 / 120 / 160 Number of thread groups in the pool. NOTE: thread_pool_size=5 was also tested, but the performance was unsatisfactory, for that reason it is not included in the report. thread_pool_max_threads / thread_pool_max_active_query_threads 12000 Maximum number of threads in the pool. thread_pool_oversubscribe / thread_pool_query_threads_per_group 2 / 3 / 4 Number of threads that can be active at the same time within the same group.   Other Configuration and Methodology The configuration was as follows: Benchmark Sysbench OLTP Read-Write CPU Intel Xeon Gold 6230 (2×20 cores, HT = 80 logical CPUs) RAM 187 GiB DDR4 Storage NVMe SSD (2.9 TB) INTEL SSDPE2KE032T8 OS Ubuntu 24.04, kernel 6.8.0-60-generic DB Engines Percona Server for MySQL 9.7.1-1 (release build) MySQL Server 26.7.0 (release build) Additional dimensions for the benchmark: Database Sizes (Row Number) 24Gb (100M rows) Number of tables in DB Schema 20 (this number is constant for all runs) Database Schema definition can be downloaded from here:  https://percona-lab-results.github.io/2026-interactive-metrics/schema_dump.sql Number of concurrent threads 40 / 80 / 120 / 160 / 320 / 640 / 1280 / 2560 / 5120 Buffer to Data Ratio 1:12 (I/O bound), 1:2 (Partially buffered), 1:1 (Fully buffered) Execution of the benchmarks was done as follows: Ramp-up 48G – 600 sec (10 min) The Ramp-up times were established experimentally depending on the Data Size until the point when increasing them further did not bring significant changes. Measurement window 900 sec (15 min) Ideally it should be as long as possible, but measurements should take reasonable time. Hence, we used the experience of previous benchmarks and established that this window is adequate for the purpose. Number of runs 1  Could be more runs, but testing took a long time. Important Database Configuration options (the actual config files with specific settings for each run can be downloaded from the interactive graphs): InnoDB – Buffer pool Tier innodb_buffer_pool_size 2G/12G/32G innodb_buffer_pool_load_at_startup OFF innodb_buffer_pool_dump_at_shutdown OFF Threading thread_stack 512K thread_cache_size 256 back_log 4096 InnoDB I/O innodb_io_capacity 10000 innodb_io_capacity_max 20000 innodb_read_io_threads 16 innodb_write_io_threads 16 innodb_use_native_aio ON InnoDB Log / Durability innodb_log_buffer_size 256M innodb_flush_log_at_trx_commit 1 # full ACID innodb_doublewrite ON InnoDB – Concurrency &amp; OLTP Tuning innodb_stats_on_metadata OFF innodb_open_files 65536 innodb_lock_wait_timeout 50 innodb_rollback_on_timeout ON Per-Session Buffers sort_buffer_size     4M join_buffer_size     4M read_buffer_size     2M read_rnd_buffer_size 4M tmp_table_size       256M max_heap_table_size 256M Binary Log disable_log_bin ON # Disabled binlog Other InnoDB settings innodb_redo_log_capacity     4G innodb_change_buffering      none innodb_flush_method          O_DIRECT innodb_buffer_pool_instances Calculated as (innodb_buffer_pool_size G / 5) But must be in range [1..8] Misc server settings collation_server utf8mb4_unicode_ci bulk_insert_buffer_size 256M myisam_sort_buffer_size  128M key_buffer_size          64M # MyISAM only, keep small for OLTP   Results The scope of results is broad and therefore the results presentation will be divided into the sections. Percona Server 9.7.1-1 with and without thread pool: The most impressive effect of the thread pool can be illustrated on the following graph where the level of oversubscription is 4 (os4 in the legend) and the number of thread groups is 10 (TP 10 in the legend).  Graph 1.0 – best effect of thread pooling [ INTERACTIVE GRAPH ][ TABLE ] The green line (thread_pool_size=10/thread_pool_oversubscribe=4) is closely followed by the blue line, which corresponds to the number of groups 20 and the oversubscription 2. As the graph shows – the effect of the thread pool on the performance in the low thread count (40/80) is relatively small. A notable effect starts to show at 120 threads when non-pooled configuration TPS plunges down. It is good to keep in mind that the thread pooling mechanism is not a universal performance booster. Though, it helps to avoid system thrashing when the number of threads is much larger than CPU cores can process. With 2560 connections the best configuration with thread pool shows excellent efficiency of 2304 TPS against 129 TPS without the thread pool, which is 17.9 times faster. Some thread-pooled configurations are more efficient than the other ones. The combination thread_pool_size=20 with thread_pool_oversubscribe=4 fails to prevent sharp TPS drop at 120 client connections, though its shape improves as the number of connections increases.The drop from the maximum of 2485 TPS at 640 connections to 2304 TPS at 2560 connections is visible (7.5%), but not critical. Interesting that the optimal number of groups is smaller than the number of physical CPU cores (40). Another important observation is that in this test increase of the number of thread groups causes the TPS performance to drop: the curve with thread_pool_size=20 is below thread_pool_size=10 and thread_pool_size=40/80/120/160 are even lower.  Graph 1.1 – thread pool size variations [ INTERACTIVE GRAPH ][ TABLE ] The next graph shows how TPS changes when we consider the optimal value for thread_pool_size=10, but change thread_pool_oversubscribe.  Graph 2.0 – oversubscription variations [ INTERACTIVE GRAPH ][ TABLE ] With the best configuration (thread_pool_oversubscribe=4) having 2304 TPS and the least performing pooling configuration (thread_pool_oversubscribe=2) with 2043 TPS the speed difference is 11%. So, a non-optimal oversubscription value can incur a significant penalty. The above two cases correspond to the I/O bound scenario when the data size (24G) is much larger than the server buffer size (2G). Let us see how the buffer pool efficiency changes as innodb_buffer_pool_size increased to 12G, which allows buffering approximately half of the data set.  Graph 3.0 – partially buffered data set (12G buffer)  [ INTERACTIVE GRAPH ][ TABLE ] This time thread_pool_size=20 is the optimal value for the data size (24G) and the allocated server buffer size (12G). The non-optimal oversubscription values have a similar effect (look at INTERACTIVE GRAPH) on the performance as in Graph 2.0. When the server configuration moves to the fully buffered data set the distribution of performance is changed:  Graph 4.0 – fully buffered data set (32G buffer)  [ INTERACTIVE GRAPH ][ TABLE ] The configuration without the thread pool wins in all number of client connections in this test. The largest deviation from the base line is observed around 80 to 640 connections, but then the performance converses to around 16K TPS for all configurations. The configuration with  thread_pool_size=10 is now the slowest one and larger thread pool sizes perform better. Varying oversubscription did not have a notable effect with fully buffered data. NOTE: It would be interesting to see if thread pool wins with 5K or 10K connections. We might do another research devoted to extremely high thread count, but this post describes the efficiency of the thread pool across the low (40) to the high (2560) number of client connection threads. Despite not having superior performance with regards to TPS with the fully buffered data set, the thread pool is still beneficial with regards to predictability of the server behavior. What I mean by this is most of the clients (95%) connected to the server have their transactions executed quickly, but a minority (5%) have to wait much longer. The response latency for the unlucky 5% might be out of the acceptable range. The long waiting clients do not care about other clients whose data was quickly processed. Therefore, the raw TPS does not hold much value for them.  Graph 5.0 – p95 latency  [ INTERACTIVE GRAPH ][ TABLE ] Unlike TPS when the higher value is better, the p95 Latency should be kept as low as possible. Up to 1280 client connections the p95 latency was grouping around roughly the same numbers. However, for 2560 connections it changed dramatically. The configuration with disabled thread pooling suddenly demonstrated much longer p95 than the pooled counterparts. The 5% of clients will have to wait longer than 787 ms if the thread pool is not used. The pooled configurations keep a much tighter range of 235-297 ms. The system performs as well as its slowest/weakest link and if a client application performs several transactions it is more likely to end up waiting significantly longer. For some outliers the latency reached 2000 ms.    Conclusion Thread pooling does not bring any gains and even causes a small slow down on a low number of server connections. The most visible effect of thread pooling can be seen in the scenario when I/O is one of the factors that limit performance. Careful tuning of thread pool options is required to get the best performance. Even a seemingly small variation from optimal parameter values can cause large performance drops. However, when properly configured, it can show spectacular results with almost 18 times TPS boost compared to the same server running without thread pool. For system stability we should not only consider the raw TPS numbers, but also the latency related to processing of some transactions. Even if no TPS gain is obtained, the thread pool allows to minimize the waiting time for slowest transactions. The post Thread Pool in Percona Server and MySQL (Part 1) appeared first on Percona. Thread Pool in Percona Server and MySQL (Part 1)

A new version of Community MySQL Server July release contains two versions 26.7.0 and 9.7.2. It is an important milestone because it finally brings a Thread Pooling feature to the community. The official MySQL Server Thread Pool plugin existed for a long time, but was available exclusively for MySQL Enterprise Edition.

Before MySQL 26.7.0 / 9.7.2 the most popular free thread pooling enabled alternatives were:

  • Percona Server for MySQL, which shipped Thread Pool functionality as a plugin
  • MariaDB with the Thread Pooling built into the server
  • Third-party plugins such as xiezhenye/mysql-plugin-threadpool available on GitHub

Now we will be able to see what benefits the thread pooling mechanism brings to MySQL Server and compare it with the implementation of the thread pool in Percona Server for MySQL.

Part 1 will explain the basics of thread pooling and analyze the efficiency of its implementation in Percona Server for MySQL. The performance comparison between Percona Server and MySQL Server is done in Part 2.

What is Thread Pooling and why is it needed?

Thread Pooling is a technique that reuses a fixed number of pre-created threads to handle multiple client connections and execute statements inside a database.

By default, MySQL creates a dedicated thread whenever a client connects, runs the queries, and then destroys the thread when finished. When connection counts grow very large, this default method causes performance to drop significantly because creating, destroying, and constantly switching between thousands of threads wastes system resources and causes contention.

This blog post from Vadim Tkachenko explains the working of thread pool, which gives an easy to understand similarity with the car traffic:

SimCity outages, traffic control and Thread Pool for MySQL

More information about it can be found in Percona Documentation.

WARNING: the post is several pages long because it looks into different aspects of thread pooling and compares the effect of various options. If you do not want to dive into the details, feel free to read the SPOILER below and proceed to the Conclusion section.

SPOILER: Thread Pool is not a universal database booster, which always brings performance up. Like every advanced tool it should be used with full understanding of the goals and objectives. Otherwise the results might be far from perfect.

How it works

  1. Thread Groups: The pool divides threads into distinct groups that are assigned to a specific CPU core. For this reason the number of groups often matches the number of CPU cores.
  2. Round-Robin Assignment: As client connections come in, they are distributed evenly across these groups.
  3. Listening and Queuing: Inside each group, a listener thread monitors incoming queries and places them into either a high-priority queue (such as queries within an active transaction) or a normal/low-priority queue.
  4. Execution and Reuse: Worker threads pick up queries from the queues (handling high-priority items first), execute them, and then wait to handle the next request instead of shutting down.

See also:

Thread Pool configuration:

In this research we will be sweeping across the range of various parameters of the Thread Pool and see how the performance of the server is affected.

The base line will be the server configuration when the thread pool is not enabled. This is done by setting thread_handling=one-thread-per-connection. When the pooling is enabled (thread_handling=pool-of-threads) the benchmark will set different values for the following variables that control the thread pool behavior:

Name Sweep range Description
thread_pool_size 10 / 20 / 40 / 80 / 120 / 160 Number of thread groups in the pool.
NOTE: thread_pool_size=5 was also tested, but the performance was unsatisfactory, for that reason it is not included in the report.
thread_pool_max_threads /
thread_pool_max_active_query_threads		 12000 Maximum number of threads in the pool.
thread_pool_oversubscribe /
thread_pool_query_threads_per_group		 2 / 3 / 4 Number of threads that can be active at the same time within the same group.

 

Other Configuration and Methodology

The configuration was as follows:

Benchmark Sysbench OLTP Read-Write
CPU Intel Xeon Gold 6230 (2×20 cores, HT = 80 logical CPUs)
RAM 187 GiB DDR4
Storage NVMe SSD (2.9 TB) INTEL SSDPE2KE032T8
OS Ubuntu 24.04, kernel 6.8.0-60-generic
DB Engines Percona Server for MySQL 9.7.1-1 (release build)

MySQL Server 26.7.0 (release build)

Additional dimensions for the benchmark:

Database Sizes (Row Number) 24Gb (100M rows)
Number of tables in DB Schema 20 (this number is constant for all runs)

Database Schema definition can be downloaded from here: 

https://percona-lab-results.github.io/2026-interactive-metrics/schema_dump.sql
Number of concurrent threads 40 / 80 / 120 / 160 / 320 / 640 / 1280 / 2560 / 5120
Buffer to Data Ratio 1:12 (I/O bound), 1:2 (Partially buffered), 1:1 (Fully buffered)

Execution of the benchmarks was done as follows:

Ramp-up 48G – 600 sec (10 min)

The Ramp-up times were established experimentally depending on the Data Size until the point when increasing them further did not bring significant changes.
Measurement window 900 sec (15 min)

Ideally it should be as long as possible, but measurements should take reasonable time. Hence, we used the experience of previous benchmarks and established that this window is adequate for the purpose.
Number of runs 1 

Could be more runs, but testing took a long time.

Important Database Configuration options (the actual config files with specific settings for each run can be downloaded from the interactive graphs):

InnoDB – Buffer pool Tier
innodb_buffer_pool_size 2G/12G/32G
innodb_buffer_pool_load_at_startup OFF
innodb_buffer_pool_dump_at_shutdown OFF
Threading
thread_stack 512K
thread_cache_size 256
back_log 4096
InnoDB I/O
innodb_io_capacity 10000
innodb_io_capacity_max 20000
innodb_read_io_threads 16
innodb_write_io_threads 16
innodb_use_native_aio ON
InnoDB Log / Durability
innodb_log_buffer_size 256M
innodb_flush_log_at_trx_commit 1 # full ACID
innodb_doublewrite ON
InnoDB – Concurrency & OLTP Tuning
innodb_stats_on_metadata OFF
innodb_open_files 65536
innodb_lock_wait_timeout 50
innodb_rollback_on_timeout ON
Per-Session Buffers
sort_buffer_size     4M
join_buffer_size     4M
read_buffer_size     2M
read_rnd_buffer_size 4M
tmp_table_size       256M
max_heap_table_size 256M
Binary Log
disable_log_bin ON # Disabled binlog
Other InnoDB settings
innodb_redo_log_capacity     4G
innodb_change_buffering      none
innodb_flush_method          O_DIRECT
innodb_buffer_pool_instances Calculated as
(innodb_buffer_pool_size G / 5)
But must be in range [1..8]
Misc server settings
collation_server utf8mb4_unicode_ci
bulk_insert_buffer_size 256M
myisam_sort_buffer_size  128M
key_buffer_size          64M # MyISAM only, keep small for OLTP

 

Results

The scope of results is broad and therefore the results presentation will be divided into the sections.

Percona Server 9.7.1-1 with and without thread pool:

The most impressive effect of the thread pool can be illustrated on the following graph where the level of oversubscription is 4 (os4 in the legend) and the number of thread groups is 10 (TP 10 in the legend).

 Graph 1.0 – best effect of thread pooling [ INTERACTIVE GRAPH ][ TABLE ]Graph 1.1 - thread pool size variations

The green line (thread_pool_size=10/thread_pool_oversubscribe=4) is closely followed by the blue line, which corresponds to the number of groups 20 and the oversubscription 2.

As the graph shows – the effect of the thread pool on the performance in the low thread count (40/80) is relatively small. A notable effect starts to show at 120 threads when non-pooled configuration TPS plunges down. It is good to keep in mind that the thread pooling mechanism is not a universal performance booster. Though, it helps to avoid system thrashing when the number of threads is much larger than CPU cores can process.

With 2560 connections the best configuration with thread pool shows excellent efficiency of 2304 TPS against 129 TPS without the thread pool, which is 17.9 times faster. Some thread-pooled configurations are more efficient than the other ones. The combination thread_pool_size=20 with thread_pool_oversubscribe=4 fails to prevent sharp TPS drop at 120 client connections, though its shape improves as the number of connections increases.The drop from the maximum of 2485 TPS at 640 connections to 2304 TPS at 2560 connections is visible (7.5%), but not critical.

Interesting that the optimal number of groups is smaller than the number of physical CPU cores (40).

Another important observation is that in this test increase of the number of thread groups causes the TPS performance to drop: the curve with thread_pool_size=20 is below thread_pool_size=10 and thread_pool_size=40/80/120/160 are even lower.

 Graph 1.1 – thread pool size variations [ INTERACTIVE GRAPH ][ TABLE ]Graph 1.1 - thread pool size variations

The next graph shows how TPS changes when we consider the optimal value for thread_pool_size=10, but change thread_pool_oversubscribe.

 Graph 2.0 – oversubscription variations [ INTERACTIVE GRAPH ][ TABLE ]Graph 2.0 - oversubscription variations

With the best configuration (thread_pool_oversubscribe=4) having 2304 TPS and the least performing pooling configuration (thread_pool_oversubscribe=2) with 2043 TPS the speed difference is 11%. So, a non-optimal oversubscription value can incur a significant penalty.

The above two cases correspond to the I/O bound scenario when the data size (24G) is much larger than the server buffer size (2G).

Let us see how the buffer pool efficiency changes as innodb_buffer_pool_size increased to 12G, which allows buffering approximately half of the data set.

 Graph 3.0 – partially buffered data set (12G buffer)  [ INTERACTIVE GRAPH ][ TABLE ]Graph 3.0 - partially buffered data set (12G buffer)

This time thread_pool_size=20 is the optimal value for the data size (24G) and the allocated server buffer size (12G). The non-optimal oversubscription values have a similar effect (look at INTERACTIVE GRAPH) on the performance as in Graph 2.0.

When the server configuration moves to the fully buffered data set the distribution of performance is changed:

 Graph 4.0 – fully buffered data set (32G buffer)  [ INTERACTIVE GRAPH ][ TABLE ]Graph 4.0 - fully buffered data set (32G buffer)

The configuration without the thread pool wins in all number of client connections in this test. The largest deviation from the base line is observed around 80 to 640 connections, but then the performance converses to around 16K TPS for all configurations. The configuration with  thread_pool_size=10 is now the slowest one and larger thread pool sizes perform better.

Varying oversubscription did not have a notable effect with fully buffered data.

NOTE: It would be interesting to see if thread pool wins with 5K or 10K connections. We might do another research devoted to extremely high thread count, but this post describes the efficiency of the thread pool across the low (40) to the high (2560) number of client connection threads.

Despite not having superior performance with regards to TPS with the fully buffered data set, the thread pool is still beneficial with regards to predictability of the server behavior. What I mean by this is most of the clients (95%) connected to the server have their transactions executed quickly, but a minority (5%) have to wait much longer. The response latency for the unlucky 5% might be out of the acceptable range. The long waiting clients do not care about other clients whose data was quickly processed. Therefore, the raw TPS does not hold much value for them.

 Graph 5.0 – p95 latency  [ INTERACTIVE GRAPH ][ TABLE ]Graph 5.0 - p95 latency

Unlike TPS when the higher value is better, the p95 Latency should be kept as low as possible.

Up to 1280 client connections the p95 latency was grouping around roughly the same numbers. However, for 2560 connections it changed dramatically. The configuration with disabled thread pooling suddenly demonstrated much longer p95 than the pooled counterparts.

The 5% of clients will have to wait longer than 787 ms if the thread pool is not used. The pooled configurations keep a much tighter range of 235-297 ms. The system performs as well as its slowest/weakest link and if a client application performs several transactions it is more likely to end up waiting significantly longer. For some outliers the latency reached 2000 ms. 

 

Conclusion

Thread pooling does not bring any gains and even causes a small slow down on a low number of server connections.

The most visible effect of thread pooling can be seen in the scenario when I/O is one of the factors that limit performance.

Careful tuning of thread pool options is required to get the best performance. Even a seemingly small variation from optimal parameter values can cause large performance drops.

However, when properly configured, it can show spectacular results with almost 18 times TPS boost compared to the same server running without thread pool.

For system stability we should not only consider the raw TPS numbers, but also the latency related to processing of some transactions. Even if no TPS gain is obtained, the thread pool allows to minimize the waiting time for slowest transactions.

The post Thread Pool in Percona Server and MySQL (Part 1) appeared first on Percona.

]]> Mon, 05 Oct 2026 10:48:57 +0000 MySQL Performance Blog Benchmarks MySQL benchmark Percona Server for MySQL How Nutanix Database Service Handles Oracle Database Patching: An Out-of-Place, Automated, and Reversible Workflow https://www.nutanix.com/blog/how-nutanix-database-service-handles-oracle-database-patching https://www.nutanix.com/blog/how-nutanix-database-service-handles-oracle-database-patching Oracle patching carries operational risk. The traditional approach, manual in-place updates, leaves limited margin for error. One misstep in an Oracle software home can mean hours of unplanned downtime and a difficult root-cause conversation afterward. Mon, 05 Oct 2026 10:18:00 +0000 Nutanix MOF - Interest Nutanix Database Service (NDB) Security Database-as-a-Service Brand MySQL Community Meetups Fall 2026 7ae026344ba0c21c34ecb629d6d3a430 https://blogs.oracle.com/mysql/mysql-community-meetups-fall-2026 In our recent Where can you find MySQL from August through October 2026?, we shared a broad view of MySQL conferences, community activities, and user group meetups. Since then, community gatherings in Denver, Taipei, and Nashville have brought MySQL users, developers, and contributors together for technical discussion and local connection. We are continuing that momentum […] Fri, 02 Oct 2026 10:22:00 +0000 Oracle MySQL Group MySQL MySQL Community Facebook Status Box using Node.js and MySQL http://codeforgeek.com/?p=94 https://codeforgeek.com/facebook-status-box-using-node-js-expressjs-mysql/ A status with angle brackets appears as text in the feed rather than turning into page markup. I appreciate that line breaks also stay visible, so a longer update reads more like what was typed. A submitted status gives you a concrete path into the feed. See how the app saves that update so the […] Fri, 02 Oct 2026 00:21:29 +0000 Shahid shaikh Database Express tutorials Mysql Node Tutorials Tutorials Village News: MySQL News + Events (1 October 2026) 6abd536e7495d70001d83857 https://villagesql.com/blog/village-news-mysql-news-events-1-october-2026/ Welcome back to Village News, our curated roundup of MySQL and database news. If you want to get these updates, just subscribe to the blog. Enjoy! MySQL News Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated) MySQL 26.10.0 Early Access Is Here Oracle MySQL Group TL;DR - A preview build of the next Innovation release is on MySQL Labs. The headline change is MySQL REST Service as a server component, which serves HTTPS endpoints without a separate MySQL Router. It is not for production. Newsletter #1: What Happened in September 2026 Maria Nesterova, OurSQL Foundation TL;DR - The Foundation ran its first booth at Percona Live Amsterdam, launched the OurSQL Universe map of the MySQL ecosystem, and scheduled webinars on row-level recovery (15 Oct) and ProxySQL (3 Nov). Introducing the MySQL Analytical Replica (Parquet + DuckDB) dbtrail Blog TL;DR - An Apache 2.0 tool that follows the binlog, keeps a copy of your MySQL tables as Parquet files in local storage or S3, and lets you query them with DuckDB, so reporting queries stay off production. Resolving query plan regressions after a MySQL engine upgrade Chandana DA and Sravan Kumar Gogana, AWS Database Blog TL;DR - A three-step workflow for RDS and Aurora MySQL upgrades: refresh statistics, compare EXPLAIN ANALYZE plans, then fix each regression with optimizer_switch flags, index hints or composite indexes. MyVector v1.26.9: Keeping Up With MySQL Innovation Alkin Tezuysal TL;DR - The MyVector vector search plugin adds support for MySQL 26.7, the first Innovation release under Oracle's new calendar-based versioning. It also puts the Component architecture from v1.26.5 through heavier testing. Open FIPS Build, Vectors, and Binlog Server: Marco Tusa at Percona Live Percona Community TL;DR - Notes from Marco Tusa's Percona Live Amsterdam keynote. FIPS support moves from the Pro Build to the standard build, vector search lands in Percona Server for MySQL, and an early binlog server is shown. Amazon RDS for MySQL announces Extended Support minor versions 5.7.44-rds.20260902 and 8.0.46-rds.20260908 AWS What's New TL;DR - New CVE and bug-fix builds for RDS customers still on MySQL 5.7 and 8.0 under paid Extended Support. Join MySQL Public Discussion #6 and the November Contributor Summit Oracle MySQL Group TL;DR - Public Discussion #6 is on 22 October. The next MySQL Contributor Summit runs online on 4–5 November, continuing Oracle's series of open community sessions. Access MySQL over MCP with vsql-mcp VillageSQL TL;DR - A Rust VillageSQL Server extension that runs a Model Context Protocol (MCP) server inside the database. It offers six tools for schema browsing, read-only queries, query plans and optional writes, with bearer-token auth, table allowlists and row limits. Hardening EmergencyReparentShard in v25 Vitess TL;DR - Vitess v25 speeds up emergency failover by filtering relay-log apply by received GTIDs. It also orders candidates consistently by GTID dominance and adds explicit split-brain recovery options. How Atomic DDL Works in MySQL 8.0 Libing Song TL;DR - An internals walkthrough of how MySQL 8.0's transactional data dictionary replaced .frm files. It shows how this closed the crash windows that used to leave server metadata out of step with InnoDB's own metadata and data. PeakAIO contributes support of 8k API nodes to RonDB 26.02.11 Mikael Ronstrom TL;DR - RonDB, the NDB Cluster fork, goes from 2,039 to 8,191 API nodes in 26.02.11. The change was contributed by PeakAIO, which uses RonDB as the metadata server for its pNFS Lattice file system. Database News MySQL BR Conf 2026 Slides: Managed ProxySQL in Your Own AWS Account ProxySQL TL;DR - Slides from René Cannaò and Alkin Tezuysal's São Paulo session on ProxySQL.Cloud: bring-your-own-cloud architecture, configuration and day-to-day operations. A recap of the conference is also up. dbdeployer v2.4.2: Faster MariaDB Sandboxes, Better Cluster Checks, and Five Months of Progress ProxySQL Blog TL;DR - v2.4.2 removes a readiness wait that cuts MariaDB sandbox deployment from about eight minutes to 20 seconds and fixes Galera/PXC checks. Recent releases also added MariaDB downloads and PostgreSQL replication checks. Lakebase Search: State-of-the-art full text and vector search for Postgres Zhou Sun, Jinjing Zhou, Pranav Aurora et al., Databricks TL;DR - Two Lakebase Postgres extensions are now GA on AWS and Azure. lakebase_vector does ANN search with RaBitQ-compressed IVF indexes, and lakebase_text replaces GIN with a BM25 index. Both can be fused in one SQL query for hybrid search. MariaDB Foundation's Test Automation Framework (TAF) 4.0 Release Jonathan Miller, MariaDB Foundation TL;DR - The Foundation's performance test framework adds PostgreSQL alongside MariaDB and MySQL. It also generates database and test-case configuration at runtime and changes recovery mode to remove data skew. For MariaDB, LLMs are the new search engines to win over Richard Speed, The Register TL;DR - MariaDB Foundation chairman Kaj Arnö says MariaDB's vector search is ready, but AI coding assistants rarely recommend it. More documentation and real-world use are needed to change that. TideSQL 5 and MariaDB: the Tides Are Moving Fast Frédéric Descamps, MariaDB Foundation TL;DR - The TidesDB-backed storage engine reaches 5.0 with MVCC, secondary indexes, foreign keys, XA, partitioning and online DDL. It now installs from RPM/DEB packages via MariaDB Foundry, with no source build. Online enabling checksums in PostgreSQL 19 Franck Pachot TL;DR - PostgreSQL 19's pg_enable_data_checksums() turns on data checksums in the background without downtime. It needs superuser and two background workers, and a crash mid-run means starting over. Anatomy of a (Postgres) search engine Eric Ridge, PlanetScale TL;DR - A walkthrough of inverted indexes: term dictionaries, postings lists and segment merging. It covers how PlanetScale's TIN extension handles MVCC visibility and mixes full-text constraints with ordinary SQL in Postgres. A follow-up, TIN Postgres search is faster, better, and cheaper, reruns a popular Postgres search demo on TIN. Upcoming Database Events Postgres Summit US (formerly PGConf.NYC) September 30 – October 2, 2026 (PgUS) New York City, NY High Performance Transaction Systems (HPTS) October 4-7, 2026 Asilomar Conference Grounds Pacific Grove, CA Open Source Summit Europe October 7-9, 2026 Prague, Czechia TiDB SCaiLE 2026 October 15, 2026 Computer History Museum Mountain View, CA All Things Open 2026 October 19-20, 2026 Raleigh Convention Center Raleigh, NC PGConf.EU October 20–23, 2026 (PostgreSQL Europe) Valencia, Spain MySQL Public Discussion #6 October 22, 2026, 17:00–18:00 UTC Online Oracle AI World October 25–28, 2026 Las Vegas, NV MySQL Contributor Summit November 4-5, 2026 Online KubeCon + CloudNativeCon North America (VillageSQL is a sponsor) November 9-12, 2026 Salt Lake City, Utah AWS re:Invent 2026 November 30 – December 4, 2026 Las Vegas, NV Open Source Summit Japan December 7-9, 2026 Tokyo, Japan CIDR 2027 January 24-27, 2027 Mövenpick Hotel Amsterdam City Centre Amsterdam, The Netherlands FOSDEM 2027 January 30-31, 2027 ULB Solbosch Campus Brussels, Belgium PGConf India 2027 March 2-5, 2027 Sheraton Grand Hotel at Brigade Gateway Bengaluru, India KubeCon + CloudNativeCon Europe 2027 March 15-18, 2027 Barcelona, Spain Nordic PGDay 2027 March 16, 2027 Courtyard Kungsholmen Stockholm, Sweden SCaLE 24x April 1-4, 2027 Pasadena Convention Center Pasadena, CA PGConf.dev 2027 May 11-14, 2027 Plaza Centre-Ville Montréal, QC, Canada

Welcome back to Village News, our curated roundup of MySQL and database news.

If you want to get these updates, just subscribe to the blog.

Enjoy!

MySQL News

Note: Aggregated MySQL news can be found at Planet for MySQL Community and Planet MySQL (Oracle curated)

MySQL 26.10.0 Early Access Is Here
Oracle MySQL Group

TL;DR - A preview build of the next Innovation release is on MySQL Labs. The headline change is MySQL REST Service as a server component, which serves HTTPS endpoints without a separate MySQL Router. It is not for production.

Newsletter #1: What Happened in September 2026
Maria Nesterova, OurSQL Foundation

TL;DR - The Foundation ran its first booth at Percona Live Amsterdam, launched the OurSQL Universe map of the MySQL ecosystem, and scheduled webinars on row-level recovery (15 Oct) and ProxySQL (3 Nov).

Introducing the MySQL Analytical Replica (Parquet + DuckDB)
dbtrail Blog

TL;DR - An Apache 2.0 tool that follows the binlog, keeps a copy of your MySQL tables as Parquet files in local storage or S3, and lets you query them with DuckDB, so reporting queries stay off production.

Resolving query plan regressions after a MySQL engine upgrade
Chandana DA and Sravan Kumar Gogana, AWS Database Blog

TL;DR - A three-step workflow for RDS and Aurora MySQL upgrades: refresh statistics, compare EXPLAIN ANALYZE plans, then fix each regression with optimizer_switch flags, index hints or composite indexes.

MyVector v1.26.9: Keeping Up With MySQL Innovation
Alkin Tezuysal

TL;DR - The MyVector vector search plugin adds support for MySQL 26.7, the first Innovation release under Oracle's new calendar-based versioning. It also puts the Component architecture from v1.26.5 through heavier testing.

Open FIPS Build, Vectors, and Binlog Server: Marco Tusa at Percona Live
Percona Community

TL;DR - Notes from Marco Tusa's Percona Live Amsterdam keynote. FIPS support moves from the Pro Build to the standard build, vector search lands in Percona Server for MySQL, and an early binlog server is shown.

Amazon RDS for MySQL announces Extended Support minor versions 5.7.44-rds.20260902 and 8.0.46-rds.20260908
AWS What's New

TL;DR - New CVE and bug-fix builds for RDS customers still on MySQL 5.7 and 8.0 under paid Extended Support.

Join MySQL Public Discussion #6 and the November Contributor Summit
Oracle MySQL Group

TL;DR - Public Discussion #6 is on 22 October. The next MySQL Contributor Summit runs online on 4–5 November, continuing Oracle's series of open community sessions.

Access MySQL over MCP with vsql-mcp
VillageSQL

TL;DR - A Rust VillageSQL Server extension that runs a Model Context Protocol (MCP) server inside the database. It offers six tools for schema browsing, read-only queries, query plans and optional writes, with bearer-token auth, table allowlists and row limits.

Hardening EmergencyReparentShard in v25
Vitess

TL;DR - Vitess v25 speeds up emergency failover by filtering relay-log apply by received GTIDs. It also orders candidates consistently by GTID dominance and adds explicit split-brain recovery options.

How Atomic DDL Works in MySQL 8.0
Libing Song

TL;DR - An internals walkthrough of how MySQL 8.0's transactional data dictionary replaced .frm files. It shows how this closed the crash windows that used to leave server metadata out of step with InnoDB's own metadata and data.

PeakAIO contributes support of 8k API nodes to RonDB 26.02.11
Mikael Ronstrom

TL;DR - RonDB, the NDB Cluster fork, goes from 2,039 to 8,191 API nodes in 26.02.11. The change was contributed by PeakAIO, which uses RonDB as the metadata server for its pNFS Lattice file system.

Database News

MySQL BR Conf 2026 Slides: Managed ProxySQL in Your Own AWS Account
ProxySQL

TL;DR - Slides from René Cannaò and Alkin Tezuysal's São Paulo session on ProxySQL.Cloud: bring-your-own-cloud architecture, configuration and day-to-day operations. A recap of the conference is also up.

dbdeployer v2.4.2: Faster MariaDB Sandboxes, Better Cluster Checks, and Five Months of Progress
ProxySQL Blog

TL;DR - v2.4.2 removes a readiness wait that cuts MariaDB sandbox deployment from about eight minutes to 20 seconds and fixes Galera/PXC checks. Recent releases also added MariaDB downloads and PostgreSQL replication checks.

Lakebase Search: State-of-the-art full text and vector search for Postgres
Zhou Sun, Jinjing Zhou, Pranav Aurora et al., Databricks

TL;DR - Two Lakebase Postgres extensions are now GA on AWS and Azure. lakebase_vector does ANN search with RaBitQ-compressed IVF indexes, and lakebase_text replaces GIN with a BM25 index. Both can be fused in one SQL query for hybrid search.

MariaDB Foundation's Test Automation Framework (TAF) 4.0 Release
Jonathan Miller, MariaDB Foundation

TL;DR - The Foundation's performance test framework adds PostgreSQL alongside MariaDB and MySQL. It also generates database and test-case configuration at runtime and changes recovery mode to remove data skew.

For MariaDB, LLMs are the new search engines to win over
Richard Speed, The Register

TL;DR - MariaDB Foundation chairman Kaj Arnö says MariaDB's vector search is ready, but AI coding assistants rarely recommend it. More documentation and real-world use are needed to change that.

TideSQL 5 and MariaDB: the Tides Are Moving Fast
Frédéric Descamps, MariaDB Foundation

TL;DR - The TidesDB-backed storage engine reaches 5.0 with MVCC, secondary indexes, foreign keys, XA, partitioning and online DDL. It now installs from RPM/DEB packages via MariaDB Foundry, with no source build.

Online enabling checksums in PostgreSQL 19
Franck Pachot

TL;DR - PostgreSQL 19's pg_enable_data_checksums() turns on data checksums in the background without downtime. It needs superuser and two background workers, and a crash mid-run means starting over.

Anatomy of a (Postgres) search engine
Eric Ridge, PlanetScale

TL;DR - A walkthrough of inverted indexes: term dictionaries, postings lists and segment merging. It covers how PlanetScale's TIN extension handles MVCC visibility and mixes full-text constraints with ordinary SQL in Postgres. A follow-up, TIN Postgres search is faster, better, and cheaper, reruns a popular Postgres search demo on TIN.

Upcoming Database Events

Postgres Summit US (formerly PGConf.NYC)
September 30 – October 2, 2026 (PgUS)
New York City, NY

High Performance Transaction Systems (HPTS)
October 4-7, 2026
Asilomar Conference Grounds
Pacific Grove, CA

Open Source Summit Europe
October 7-9, 2026
Prague, Czechia

TiDB SCaiLE 2026
October 15, 2026
Computer History Museum
Mountain View, CA

All Things Open 2026
October 19-20, 2026
Raleigh Convention Center
Raleigh, NC

PGConf.EU
October 20–23, 2026 (PostgreSQL Europe)
Valencia, Spain

MySQL Public Discussion #6
October 22, 2026, 17:00–18:00 UTC
Online

Oracle AI World
October 25–28, 2026
Las Vegas, NV

MySQL Contributor Summit
November 4-5, 2026
Online

KubeCon + CloudNativeCon North America (VillageSQL is a sponsor)
November 9-12, 2026
Salt Lake City, Utah

AWS re:Invent 2026
November 30 – December 4, 2026
Las Vegas, NV

Open Source Summit Japan
December 7-9, 2026
Tokyo, Japan

CIDR 2027
January 24-27, 2027
Mövenpick Hotel Amsterdam City Centre
Amsterdam, The Netherlands

FOSDEM 2027
January 30-31, 2027
ULB Solbosch Campus
Brussels, Belgium

PGConf India 2027
March 2-5, 2027
Sheraton Grand Hotel at Brigade Gateway
Bengaluru, India

KubeCon + CloudNativeCon Europe 2027
March 15-18, 2027
Barcelona, Spain

Nordic PGDay 2027
March 16, 2027
Courtyard Kungsholmen
Stockholm, Sweden

SCaLE 24x
April 1-4, 2027
Pasadena Convention Center
Pasadena, CA

PGConf.dev 2027
May 11-14, 2027
Plaza Centre-Ville
Montréal, QC, Canada

]]> Thu, 01 Oct 2026 13:00:31 +0000 VillageSQL MySQL HeatWave Core Capabilities (September 2026) https://dasini.net/blog/?p=8978 https://dasini.net/blog/2026/09/30/mysql-heatwave-core-capabilities-september-2026/ OCI Open Source Service Update – Sept. 2026 Explore how MySQL HeatWave on Oracle Cloud Infrastructure helps optimize, protect, and scale MySQL applications. Covers Autopilot Indexing, high availability with automatic failover, managed read replicas, enterprise security, and fast DB system cloning for development, testing, and regional recovery. The post MySQL HeatWave Core Capabilities (September 2026) first appeared on Data Daz (dasini.net) - Data Systems, AI, and Real-World Insights. OCI Open Source Service Update – Sept. 2026
Explore how MySQL HeatWave on Oracle Cloud Infrastructure helps optimize, protect, and scale MySQL applications.
Covers Autopilot Indexing, high availability with automatic failover, managed read replicas, enterprise security, and fast DB system cloning for development, testing, and regional recovery.

The post MySQL HeatWave Core Capabilities (September 2026) first appeared on Data Daz (dasini.net) - Data Systems, AI, and Real-World Insights.]]> Wed, 30 Sep 2026 14:28:19 +0000 Olivier Dasini HeatWave MDS MySQL Cloud Heatwave MySQL Enterprise MySQL BR Conf 2026 Slides: Managed ProxySQL in Your Own AWS Account https://proxysql.com/blog/mysql-br-conf-2026-managed-proxysql-slides/ https://proxysql.com/blog/mysql-br-conf-2026-managed-proxysql-slides/ The final slides from René Cannaò and Alkin Tezuysal's session on ProxySQL.Cloud, BYOC architecture, configuration, and day-to-day operations. Wed, 30 Sep 2026 00:00:00 +0000 ProxySQL ProxySQL at MySQL BR Conf 2026: Together in São Paulo https://proxysql.com/blog/mysql-br-conf-2026-recap/ https://proxysql.com/blog/mysql-br-conf-2026-recap/ René Cannaò and Alkin Tezuysal presented managed ProxySQL in your own AWS account at MySQL BR Conf 2026 in São Paulo. Tue, 29 Sep 2026 00:00:00 +0000 ProxySQL When pt-online-schema-change “where” Meets Galera: Understanding Chunk Auto-Resize and Flow Control https://www.percona.com/?p=53966 https://www.percona.com/blog/when-pt-online-schema-change-where-meets-galera-understanding-chunk-auto-resize-and-flow-control/ Percona Toolkit’s pt-online-schema-change (pt-osc) has long been the preferred solution for performing online schema changes with minimal downtime. Its chunk-based copy algorithm is designed to adapt dynamically to the workload, making it suitable for very large tables in production environments. However, under specific conditions, one of its optimization mechanisms can become counterproductive. During a customer engagement involving Percona XtraDB Cluster (PXC), we investigated a case where running pt-online-schema-change with a selective –where clause caused severe Galera Flow Control and general overhead, effectively stalling the cluster for an extended period. The issue was not caused by the schema change itself, but by the interaction between: adaptive chunk auto-resizing highly selective copying of the newest rows clustered replication in Galera InnoDB checkpointing This article explains why it happens and proposes a set of practical mitigations that eliminate the root cause.   The Scenario Consider a large table containing hundreds of millions of rows. Instead of rebuilding the entire table, we only want to copy a subset of rows and apply a schema change like adding a new index. The command looks similar to:pt-online-schema-change \ --alter "ADD INDEX idx_status(status)" \ --where "created_at &amp;gt;= NOW() - INTERVAL 30 DAY" \ D=test,t=events \ --execute --forceThis is a common approach when: historical data can be ignored only active records require modification performing staged migrations reducing migration time At first glance this seems perfectly reasonable. Unfortunately, it exposes an interesting corner case in pt-online-schema-change.   How Chunk Auto-Resize Works During the copy phase, pt-osc continuously adjusts the chunk size in order to keep each chunk copy close to the configured execution time (–chunk-time), 0.5 seconds by default. Simplified, the algorithm works like this: if chunks execute too quickly, increase the next chunk size if chunks become slower, decrease the size repeat until reaching a stable value For ordinary table scans this works remarkably well because row density is relatively uniform. The problem appears when most scanned rows are discarded by –where.   The Hidden Effect of a Selective WHERE Clause Imagine a table like this with 1 billion rows: Rows Match WHERE 990 million No 10 million Yes Suppose the matching rows are located at the end of the clustered index — and that was exactly the case here. Keep in mind that chunk selection always relies on the Primary Key boundaries starting from minimum value, but in this case those boundaries were combined (AND) with the condition passed via –where. Since the created_at values increased in step with the Primary Key, all the matching rows ended up clustered at the end of the index. For a long time pt-osc scans chunks where almost every row is rejected. The copy operation becomes extremely fast. The adaptive algorithm interprets this as: “Chunks are too small.” It therefore keeps increasing the chunk size. Eventually, when the scan reaches the first matching rows, the chunk size has already grown dramatically. Instead of copying a few thousand rows, pt-osc may suddenly copy hundreds of thousands—or even a million—of rows in a single transaction. Nothing is technically wrong. But Galera now has a completely different problem.   Why Galera Suffers Every copied chunk is replicated as a write set. Very large transactions produce: large write sets long certification phases increased apply queues longer replication latency Flow Control activation Once Flow Control starts, cluster throughput decreases dramatically. Application commits begin waiting. In extreme situations the cluster appears almost frozen until the oversized transaction has been fully applied by every node. Ironically, the adaptive algorithm—designed to optimize throughput—ends up creating exactly the workload pattern that Galera dislikes the most. Why InnoDB Suffers too As soon as the first large chunks began processing, the following messages appeared in the PXC writer node’s error log:[Warning] [MY-014084] [InnoDB] Threads are unable to reserve space in redo log which can't be reclaimed due to the 'log_checkpointer' consumer still lagging behind at LSN = 852886039088. Consider increasing innodb_redo_log_capacity.This indicates that InnoDB received a very large transaction, and the writing threads could not find enough free space in the redo logs because the log_checkpointer consumer was falling behind. In practice, this forces the server to trigger checkpoints more frequently and on a larger scale, which increases overall overhead. After Galera Replication propagated the write set to the other nodes, the same type of message was observed on them as well, during the apply phase of the replicated transaction. Symptoms Typical symptoms include: Flow Control percentage rapidly increasing Growing wsrep_local_recv_queue High replication lag between nodes Application commits waiting CPU utilization increasing Large spikes in transaction apply time More frequent InnoDB checkpoints on all the nodes The schema change itself may still complete successfully, but the impact on production traffic can be severe. In the worst scenario this can lead to a complete stall of the cluster due to excessive flow control. This actually happened in production, and the cluster had to be restarted after remaining locked for dozens of minutes.   The Proposed Fixes Here are a few options, starting with the simplest and most effective one: disable adaptive chunk resizing whenever a selective –where clause is used. Instead of continuously increasing chunk sizes, keep them fixed throughout the operation. Simply include the following options in the command:--chunk-time = 0 --max-flow-ctl = 0If –chunk-time option is set to zero, the chunk size doesn’t auto-adjust, so query times will vary, but query chunk sizes will not. Another way to do the same thing is to specify a value for –chunk-size explicitly, instead of leaving it at the default, and omit the option –chunk-time. The adaptive algorithm is bypassed. Chunk execution becomes predictable. Transaction size remains bounded. Galera never receives unexpectedly massive write sets. The overall schema change may take slightly longer, but cluster stability improves dramatically. For production environments this is usually the preferable trade-off.   Another parameter you should consider is –max-flow-ctl. With –max-flow-ctl=PCT, pt-osc periodically checks wsrep_flow_control_paused (the percentage of time the node has spent paused for flow control) and automatically pauses row copying whenever it exceeds the threshold, resuming once the cluster drops back below it. Setting –-max-flow-ctl to zero (or very close to it, e.g. 1) makes pt-online-schema-change stop copying rows at the very first sign that any node has entered flow control, rather than tolerating some pause time before backing off — which is the safest possible stance for a production cluster, especially one that’s already showing signs of stress (like the checkpointer-lag warnings, where large transactions were pushing InnoDB’s redo log reclamation to its limit): any additional write pressure from the migration’s chunked copy could tip an already struggling node into a much bigger stall, so a near-zero threshold trades migration speed for safety, letting the tool fully defer to the cluster’s real-time capacity and only advance when there’s genuinely no contention, which is generally the right call for critical, latency-sensitive production workloads where a slower-but-safe schema change beats a fast one that risks a cluster-wide write freeze. A related lever worth mentioning is –chunk-index. By default, pt-online-schema-change chunks the table using the primary key (or the most appropriate unique index it can find), which is precisely why, in our case, chunk boundaries tracked the same monotonically increasing column as created_at and produced the long stretch of near-empty chunks described above. If the table has a secondary index whose ordering is not correlated with created_at, forcing the tool to chunk on that index instead (–chunk-index=idx_name) can spread matching and non-matching rows more evenly across chunks, so the adaptive algorithm never sees the extended low-density run that drives chunk size to unsafe levels in the first place. That said, this is not a substitute for –chunk-time=0 and –max-flow-ctl: pt-osc’s own documentation warns that a poorly chosen chunking index can hurt performance, since it forces query plans, and few tables have a convenient index that is both selective enough to chunk on and uncorrelated with the –where condition. In practice, we recommend treating –chunk-index as a secondary, table-specific optimization, with fixed chunk sizes and flow-control awareness remaining the primary safeguard.   A third option is to avoid the –where clause altogether and split the task into two phases instead. First, run pt-archiver solely to delete the rows you no longer need; then run pt-online-schema-change to alter the table. This two-step procedure avoids any risk stemming from the non-uniform distribution of the matching rows.   Conclusion pt-online-schema-change remains one of the best tools available for online schema migrations. Nevertheless, when using –where to copy only a small subset of rows on Percona XtraDB Cluster, adaptive chunk resizing can unintentionally generate oversized transactions that trigger excessive Galera Flow Control and temporarily stall the cluster. Keeping chunk sizes fixed eliminates this feedback loop, producing smoother replication behaviour and reducing the operational risk of online schema changes. Other fixes are also available. Sometimes, predictability is a better optimization than aggressiveness. The post When pt-online-schema-change “where” Meets Galera: Understanding Chunk Auto-Resize and Flow Control appeared first on Percona. Percona Toolkit’s pt-online-schema-change (pt-osc) has long been the preferred solution for performing online schema changes with minimal downtime. Its chunk-based copy algorithm is designed to adapt dynamically to the workload, making it suitable for very large tables in production environments.

However, under specific conditions, one of its optimization mechanisms can become counterproductive.

During a customer engagement involving Percona XtraDB Cluster (PXC), we investigated a case where running pt-online-schema-change with a selective –where clause caused severe Galera Flow Control and general overhead, effectively stalling the cluster for an extended period. The issue was not caused by the schema change itself, but by the interaction between:

  • adaptive chunk auto-resizing
  • highly selective copying of the newest rows
  • clustered replication in Galera
  • InnoDB checkpointing

This article explains why it happens and proposes a set of practical mitigations that eliminate the root cause.

 

The Scenario

Consider a large table containing hundreds of millions of rows.

Instead of rebuilding the entire table, we only want to copy a subset of rows and apply a schema change like adding a new index.

The command looks similar to:

pt-online-schema-change \
--alter "ADD INDEX idx_status(status)" \
--where "created_at &gt;= NOW() - INTERVAL 30 DAY" \
D=test,t=events \
--execute --force

This is a common approach when:

  • historical data can be ignored
  • only active records require modification
  • performing staged migrations
  • reducing migration time

At first glance this seems perfectly reasonable.

Unfortunately, it exposes an interesting corner case in pt-online-schema-change.

 

How Chunk Auto-Resize Works

During the copy phase, pt-osc continuously adjusts the chunk size in order to keep each chunk copy close to the configured execution time (–chunk-time), 0.5 seconds by default.

Simplified, the algorithm works like this:

  • if chunks execute too quickly, increase the next chunk size
  • if chunks become slower, decrease the size
  • repeat until reaching a stable value

For ordinary table scans this works remarkably well because row density is relatively uniform.

The problem appears when most scanned rows are discarded by –where.

 

The Hidden Effect of a Selective WHERE Clause

Imagine a table like this with 1 billion rows:

Rows Match WHERE
990 million No
10 million Yes

Suppose the matching rows are located at the end of the clustered index — and that was exactly the case here. Keep in mind that chunk selection always relies on the Primary Key boundaries starting from minimum value, but in this case those boundaries were combined (AND) with the condition passed via –where. Since the created_at values increased in step with the Primary Key, all the matching rows ended up clustered at the end of the index.

For a long time pt-osc scans chunks where almost every row is rejected. The copy operation becomes extremely fast.

The adaptive algorithm interprets this as: “Chunks are too small.” It therefore keeps increasing the chunk size.

Eventually, when the scan reaches the first matching rows, the chunk size has already grown dramatically. Instead of copying a few thousand rows, pt-osc may suddenly copy hundreds of thousands—or even a million—of rows in a single transaction.

Nothing is technically wrong. But Galera now has a completely different problem.

 

Why Galera Suffers

Every copied chunk is replicated as a write set.

Very large transactions produce:

  • large write sets
  • long certification phases
  • increased apply queues
  • longer replication latency
  • Flow Control activation

Once Flow Control starts, cluster throughput decreases dramatically.

Application commits begin waiting.

In extreme situations the cluster appears almost frozen until the oversized transaction has been fully applied by every node.

Ironically, the adaptive algorithm—designed to optimize throughput—ends up creating exactly the workload pattern that Galera dislikes the most.

Why InnoDB Suffers too

As soon as the first large chunks began processing, the following messages appeared in the PXC writer node’s error log:

[Warning] [MY-014084] [InnoDB] Threads are unable to reserve space in redo log which can't be reclaimed 
due to the 'log_checkpointer' consumer still lagging behind at LSN = 852886039088. 
Consider increasing innodb_redo_log_capacity.

This indicates that InnoDB received a very large transaction, and the writing threads could not find enough free space in the redo logs because the log_checkpointer consumer was falling behind. In practice, this forces the server to trigger checkpoints more frequently and on a larger scale, which increases overall overhead.

After Galera Replication propagated the write set to the other nodes, the same type of message was observed on them as well, during the apply phase of the replicated transaction.

Symptoms

Typical symptoms include:

  • Flow Control percentage rapidly increasing
  • Growing wsrep_local_recv_queue
  • High replication lag between nodes
  • Application commits waiting
  • CPU utilization increasing
  • Large spikes in transaction apply time
  • More frequent InnoDB checkpoints on all the nodes

The schema change itself may still complete successfully, but the impact on production traffic can be severe. In the worst scenario this can lead to a complete stall of the cluster due to excessive flow control. This actually happened in production, and the cluster had to be restarted after remaining locked for dozens of minutes.

 

The Proposed Fixes

Here are a few options, starting with the simplest and most effective one: disable adaptive chunk resizing whenever a selective –where clause is used.

Instead of continuously increasing chunk sizes, keep them fixed throughout the operation.

Simply include the following options in the command:

--chunk-time = 0
--max-flow-ctl = 0

If –chunk-time option is set to zero, the chunk size doesn’t auto-adjust, so query times will vary, but query chunk sizes will not. Another way to do the same thing is to specify a value for –chunk-size explicitly, instead of leaving it at the default, and omit the option –chunk-time.

The adaptive algorithm is bypassed. Chunk execution becomes predictable. Transaction size remains bounded. Galera never receives unexpectedly massive write sets. The overall schema change may take slightly longer, but cluster stability improves dramatically. For production environments this is usually the preferable trade-off.

 

Another parameter you should consider is –max-flow-ctl.

With –max-flow-ctl=PCT, pt-osc periodically checks wsrep_flow_control_paused (the percentage of time the node has spent paused for flow control) and automatically pauses row copying whenever it exceeds the threshold, resuming once the cluster drops back below it.

Setting –-max-flow-ctl to zero (or very close to it, e.g. 1) makes pt-online-schema-change stop copying rows at the very first sign that any node has entered flow control, rather than tolerating some pause time before backing off — which is the safest possible stance for a production cluster, especially one that’s already showing signs of stress (like the checkpointer-lag warnings, where large transactions were pushing InnoDB’s redo log reclamation to its limit): any additional write pressure from the migration’s chunked copy could tip an already struggling node into a much bigger stall, so a near-zero threshold trades migration speed for safety, letting the tool fully defer to the cluster’s real-time capacity and only advance when there’s genuinely no contention, which is generally the right call for critical, latency-sensitive production workloads where a slower-but-safe schema change beats a fast one that risks a cluster-wide write freeze.

A related lever worth mentioning is –chunk-index. By default, pt-online-schema-change chunks the table using the primary key (or the most appropriate unique index it can find), which is precisely why, in our case, chunk boundaries tracked the same monotonically increasing column as created_at and produced the long stretch of near-empty chunks described above. If the table has a secondary index whose ordering is not correlated with created_at, forcing the tool to chunk on that index instead (–chunk-index=idx_name) can spread matching and non-matching rows more evenly across chunks, so the adaptive algorithm never sees the extended low-density run that drives chunk size to unsafe levels in the first place. That said, this is not a substitute for –chunk-time=0 and –max-flow-ctl: pt-osc’s own documentation warns that a poorly chosen chunking index can hurt performance, since it forces query plans, and few tables have a convenient index that is both selective enough to chunk on and uncorrelated with the –where condition. In practice, we recommend treating –chunk-index as a secondary, table-specific optimization, with fixed chunk sizes and flow-control awareness remaining the primary safeguard.

 

A third option is to avoid the –where clause altogether and split the task into two phases instead. First, run pt-archiver solely to delete the rows you no longer need; then run pt-online-schema-change to alter the table. This two-step procedure avoids any risk stemming from the non-uniform distribution of the matching rows.

 

Conclusion

pt-online-schema-change remains one of the best tools available for online schema migrations.

Nevertheless, when using –where to copy only a small subset of rows on Percona XtraDB Cluster, adaptive chunk resizing can unintentionally generate oversized transactions that trigger excessive Galera Flow Control and temporarily stall the cluster.

Keeping chunk sizes fixed eliminates this feedback loop, producing smoother replication behaviour and reducing the operational risk of online schema changes. Other fixes are also available.

Sometimes, predictability is a better optimization than aggressiveness.

The post When pt-online-schema-change “where” Meets Galera: Understanding Chunk Auto-Resize and Flow Control appeared first on Percona.

]]> Mon, 28 Sep 2026 13:56:53 +0000 Corrado Pandiani Insight for DBAs Insight for Developers MySQL Open Source Percona Software XtraDB Cluster (PXC) pt-onine-schema-change pxc toolkit MyVector v1.26.9: Keeping Up With MySQL Innovation https://askdba.net/?p=2986 https://askdba.net/myvector-v1-26-9-keeping-up-with-mysql-innovation/ MySQL 26.7 Innovation Support Lands September 22, 2026 · ⁠GitHub Release MySQL just changed the rules. With MySQL 26.7, Oracle has moved to a new calendar-based versioning model for Innovation releases. For MyVector, that means one thing: We need to keep up. MyVector v1.26.9 adds MySQL 26.7 Innovation support, but the more interesting story is what happened beneath the surface. This release puts the Component architecture introduced in v1.26.5 through a much more serious test. From architecture to reliability When we introduced the MySQL Component architecture in v1.26.5, the goal was straightforward: build MyVector in the direction MySQL itself is taking. Now we are testing what happens when things don’t go perfectly. v1.26.9 adds proper Component lifecycle testing, including installation, uninstallation, failed removal, and deinitialization rollback. Because installing something is easy. Installing, removing, failing, restarting, and recovering correctly is the real test. HNSW gets harder to kill There is also some important HNSW work in this release. A number of failure paths around HNSW index creation and persistence have been fixed, including a case where an index build could crash mysqld. That obviously isn’t acceptable. We also fixed configuration handling so type=hnsw is correctly honored, and persistence failures are now surfaced instead of disappearing silently. If MyVector cannot save an index, you should know about it. Does it survive a restart? This became an explicit test in v1.26.9. It is one thing to create an index and run a vector search. It is another thing to restart MySQL and have that index come back. The new lifecycle tests verify that persisted indexes are actually reloaded from disk. That is an important distinction as MyVector moves from an interesting MySQL extension toward something people can consider for real workloads. The release process got better too v1.26.9 also expands the testing and release pipeline around: MySQL 26.7 compatibility Component lifecycle HNSW Persistent indexes Online updates Stress testing Stanford dataset smoke tests Benchmark failure detection Docker builds What’s next? The original MyVector idea hasn’t changed: Why move your data to another database just because your application needs vector search? The answer is becoming more interesting as MySQL evolves. With MyVector, the goal is to bring: SQL + transactional data + vector search + full-text search + AI workloads into the same database environment. MySQL 26.7 brings a new Innovation release. MyVector 1.26.9 makes sure we’re ready to follow it. v1.26.5 was about introducing the Component architecture. v1.26.9 is about proving it. Get MyVector ⁠MyVector v1.26.9 ⁠GitHub Repository ⁠Documentation MyVector is open source. Feedback, issues, benchmarks, and contributions are welcome.

MySQL 26.7 Innovation Support Lands


September 22, 2026 · ⁠GitHub Release

MySQL just changed the rules.

With MySQL 26.7, Oracle has moved to a new calendar-based versioning model for Innovation releases. For MyVector, that means one thing:

We need to keep up.

MyVector v1.26.9 adds MySQL 26.7 Innovation support, but the more interesting story is what happened beneath the surface.

This release puts the Component architecture introduced in v1.26.5 through a much more serious test.

From architecture to reliability

When we introduced the MySQL Component architecture in v1.26.5, the goal was straightforward: build MyVector in the direction MySQL itself is taking.

Now we are testing what happens when things don’t go perfectly.

v1.26.9 adds proper Component lifecycle testing, including installation, uninstallation, failed removal, and deinitialization rollback.

Because installing something is easy.

Installing, removing, failing, restarting, and recovering correctly is the real test.

HNSW gets harder to kill

There is also some important HNSW work in this release.

A number of failure paths around HNSW index creation and persistence have been fixed, including a case where an index build could crash mysqld.

That obviously isn’t acceptable.

We also fixed configuration handling so type=hnsw is correctly honored, and persistence failures are now surfaced instead of disappearing silently.

If MyVector cannot save an index, you should know about it.

Does it survive a restart?

This became an explicit test in v1.26.9.

It is one thing to create an index and run a vector search.

It is another thing to restart MySQL and have that index come back.

The new lifecycle tests verify that persisted indexes are actually reloaded from disk.

That is an important distinction as MyVector moves from an interesting MySQL extension toward something people can consider for real workloads.

The release process got better too

v1.26.9 also expands the testing and release pipeline around:

  • MySQL 26.7 compatibility
  • Component lifecycle
  • HNSW
  • Persistent indexes
  • Online updates
  • Stress testing
  • Stanford dataset smoke tests
  • Benchmark failure detection
  • Docker builds

What’s next?

The original MyVector idea hasn’t changed:

Why move your data to another database just because your application needs vector search?

The answer is becoming more interesting as MySQL evolves.

With MyVector, the goal is to bring:

SQL + transactional data + vector search + full-text search + AI workloads

into the same database environment.

MySQL 26.7 brings a new Innovation release.

MyVector 1.26.9 makes sure we’re ready to follow it.

v1.26.5 was about introducing the Component architecture.

v1.26.9 is about proving it.

Get MyVector

⁠MyVector v1.26.9

⁠GitHub Repository

⁠Documentation

MyVector is open source. Feedback, issues, benchmarks, and contributions are welcome.

]]> Fri, 25 Sep 2026 13:36:50 +0000 Alkin Tezuysal mysql Technical ai/ml artificial-intelligence hnsw llm vector Open FIPS Build, Vectors, and Binlog Server: Marco Tusa at Percona Live https://percona.community/blog/2026/09/25/open-mysql-fips-build-percona-live/ https://percona.community/blog/2026/09/25/open-mysql-fips-build-percona-live/ This post collects the main things Marco Tusa covered in his keynote at Percona Live 2026 in Amsterdam: where Percona stands on MySQL, which features landed in the server, and what holds the MySQL community together. Watch it here: “I Was Going to Show You a Roadmap”, about 18 minutes. Start with the community, because the rest of it follows from there. In 2011, after the Sun and Oracle acquisitions, O’Reilly stopped the MySQL Conference, and Percona kept a conference going when nobody knew what came next. The community scattered again more recently, and Peter Zaitsev and others regrouped it in the OurSQL Foundation. MySQL, MariaDB, and Percona ship competing solutions. The people running and building them are one community, with one shared interest: this environment stays usable and open. Oppose the decision, not the people MariaDB moved later Galera development inside an enterprise circle. The MariaDB Foundation pushed back. MariaDB keeps its own version, and others can still improve it. Percona keeps shipping Percona XtraDB Cluster (PXC). PXC has been in production since about 2012, and it was not built as a reply to that dispute. The argument is with the decision, not with the people who made it. The same rule applies inside Percona. A FIPS implementation in the Pro Build program was available to customers and not to everyone else. That broke the mandate to stay open. The code is open now: it is the default, and everyone can get it. What is already there This is work in progress, not a list of future dates. A binlog server, shown in Yura Sorokin’s talk “Beyond the Dumb Pipe.” It is early: a bicycle that should become a full server. Vector search inside Percona Server for MySQL, with no separate system beside it. Monitoring and observability an agent can use to see a query more clearly. MyRocks and PXC, still in active development. They are first-class parts of the server, not items added to fill a list. MySQL Enterprise features carried from version to version. Each version needs merges and adjustments. That work stays in the freely available server. The written vision is at percona.com/mysql/vision. The roadmap follows it, in public. Answer the MariaDB questionnaire from Percona Live, and share the answer. MySQL Calculator is an open source project you can try through the operator. If something is wrong, open an issue. More sessions from the event are in the Percona Live 2026 Amsterdam playlist. The same recordings are on the conference site, in Videos. This post collects the main things Marco Tusa covered in his keynote at Percona Live 2026 in Amsterdam: where Percona stands on MySQL, which features landed in the server, and what holds the MySQL community together.

Watch it here: “I Was Going to Show You a Roadmap”, about 18 minutes.

Start with the community, because the rest of it follows from there. In 2011, after the Sun and Oracle acquisitions, O’Reilly stopped the MySQL Conference, and Percona kept a conference going when nobody knew what came next. The community scattered again more recently, and Peter Zaitsev and others regrouped it in the OurSQL Foundation. MySQL, MariaDB, and Percona ship competing solutions. The people running and building them are one community, with one shared interest: this environment stays usable and open.

Oppose the decision, not the people

MariaDB moved later Galera development inside an enterprise circle. The MariaDB Foundation pushed back. MariaDB keeps its own version, and others can still improve it. Percona keeps shipping Percona XtraDB Cluster (PXC). PXC has been in production since about 2012, and it was not built as a reply to that dispute. The argument is with the decision, not with the people who made it.

The same rule applies inside Percona. A FIPS implementation in the Pro Build program was available to customers and not to everyone else. That broke the mandate to stay open. The code is open now: it is the default, and everyone can get it.

What is already there

This is work in progress, not a list of future dates.

  • A binlog server, shown in Yura Sorokin’s talk “Beyond the Dumb Pipe.” It is early: a bicycle that should become a full server.
  • Vector search inside Percona Server for MySQL, with no separate system beside it.
  • Monitoring and observability an agent can use to see a query more clearly.
  • MyRocks and PXC, still in active development. They are first-class parts of the server, not items added to fill a list.
  • MySQL Enterprise features carried from version to version. Each version needs merges and adjustments. That work stays in the freely available server.

The written vision is at percona.com/mysql/vision. The roadmap follows it, in public.

Answer the MariaDB questionnaire from Percona Live, and share the answer. MySQL Calculator is an open source project you can try through the operator. If something is wrong, open an issue.

More sessions from the event are in the Percona Live 2026 Amsterdam playlist. The same recordings are on the conference site, in Videos.

]]> Fri, 25 Sep 2026 12:00:00 +0000 Percona Community MySQL Percona Live Open Source Percona Server PXC MySQL 26.10.0 Early Access Is Here 67736b8de833261306fce23bbfa7b47a https://blogs.oracle.com/mysql/mysql-26-10-0-early-access-is-here The MySQL 26.10.0 Early Access Release is now available, giving developers and database teams an early opportunity to preview, evaluate, and provide feedback on the next evolution of MySQL. Early Access releases are where curiosity becomes contribution. Whether you’re testing an application stack, validating upgrade plans, or simply eager to see what’s coming, this is […] Fri, 25 Sep 2026 11:58:21 +0000 Oracle MySQL Group MySQL MySQL Community early access mysql mysqlcommunity