This proposal is authored by the AWS RDS Aurora MySQL team. In this discussion we propose an optimization to improve binlog write and recovery performance for large transactions. We are looking to collaborate with the community and implement this performance optimization in upstream MySQL. The tech contact is Xueting Wu (@ams-binlog-replication).
Introduction
Large transactions present a significant performance challenge for MySQL's binary logging subsystem. When a transaction exceeds the in-memory binlog cache, events spill to a local temporary file. At commit time, the entire contents of that file must be copied to the final binlog file, a process that grows linearly with transaction size, inflating commit latency and increasing the window for crash-related recovery costs.
Aurora MySQL has implemented an internal optimization that eliminates the data-copy bottleneck for large transactions by writing events directly to a dedicated remote binlog file and finalizing the commit with a rename operation. This feature has been running in production since 2020, significantly improving write performance by 6x, and bringing our P99 crash recovery time to under one minute across all Aurora MySQL customers.
Inspired by Aurora MySQL, MariaDB has since implemented a similar optimization. We believe MySQL would benefit from an equivalent capability and are prepared to contribute our implementation to upstream MySQL.
Problem Statement
MySQL's current binlog architecture handles large transactions as follows:
- Memory cache fills up: Transaction events are buffered in a per-session binlog cache. When the cache exceeds
binlog_cache_size, events spill to a local temporary file.
- Commit copies all data: At commit time, the engine copies the entire contents of the local temp file (potentially many gigabytes) into the active binlog file. This makes commit latency proportional to transaction size and increases group commit stall time for concurrent transactions.
- Recovery scans the full file: If the engine crashes after partially writing a large transaction into the binlog file, recovery must scan the entire binlog file to identify incomplete transactions and determine GTID state. For multi-gigabyte files, this can take tens of minutes.
Solution
We propose an optimization where large transactions write binlog events directly to a dedicated temporary file on persistent storage and finalize the commit with a rename, eliminating the data-copy bottleneck entirely.
Write Protocol
When a transaction's binlog cache exceeds a configurable threshold (default: 128 MB), the engine switches strategies:
- Spill to a dedicated temp file: All events accumulated so far are flushed to a new temporary binlog file on persistent storage. Subsequent events for this transaction are written directly to this file. We propose a system variable analogous to MariaDB's
binlog_large_commit_threshold to determine when to spillover. Transactions whose binlog cache exceeds this threshold trigger the direct-write path.
- Other transactions proceed normally: Concurrent transactions continue writing to the original active binlog file, which may even rotate independently.
- Commit via rename: As part of the large transaction commits, the temporary file is renamed to the next binlog file in the sequence and added to the binlog index. Commit is an O(1) metadata operation regardless of transaction size.
- Rollback via discard: If the transaction is rolled back (partially or fully), the temp file is truncated or deleted. No trace remains in the binlog.
Recovery
The design ensures that a large transaction's temp file either becomes a complete, valid binlog file or is discarded entirely:
| Crash Point |
Recovery Action |
Cost |
| Before temp file is created |
Standard rollback (online) |
Negligible |
| After spill, before rename |
Temp file is discarded; rollback is online |
Negligible |
| After rename, before final commit |
Transaction is rolled forward |
Fast — XID is at a known position near the start of the isolated file |
| After commit, before rotation |
File is already valid |
No action needed |
Because the large transaction is isolated in its own binlog file, recovery never needs to scan gigabytes of data. The engine retrieves the XID from a known position near the beginning of the file.
Collaboration Proposal
We are prepared to contribute the implementation of this optimization to upstream MySQL. The code has been battle-tested in production for over five years, and the design has been validated independently by MariaDB's adoption.
We welcome discussion on integration approach, coding standards alignment, and any design modifications Oracle considers necessary for upstream inclusion.
This proposal is authored by the AWS RDS Aurora MySQL team. In this discussion we propose an optimization to improve binlog write and recovery performance for large transactions. We are looking to collaborate with the community and implement this performance optimization in upstream MySQL. The tech contact is Xueting Wu (@ams-binlog-replication).
Introduction
Large transactions present a significant performance challenge for MySQL's binary logging subsystem. When a transaction exceeds the in-memory binlog cache, events spill to a local temporary file. At commit time, the entire contents of that file must be copied to the final binlog file, a process that grows linearly with transaction size, inflating commit latency and increasing the window for crash-related recovery costs.
Aurora MySQL has implemented an internal optimization that eliminates the data-copy bottleneck for large transactions by writing events directly to a dedicated remote binlog file and finalizing the commit with a rename operation. This feature has been running in production since 2020, significantly improving write performance by 6x, and bringing our P99 crash recovery time to under one minute across all Aurora MySQL customers.
Inspired by Aurora MySQL, MariaDB has since implemented a similar optimization. We believe MySQL would benefit from an equivalent capability and are prepared to contribute our implementation to upstream MySQL.
Problem Statement
MySQL's current binlog architecture handles large transactions as follows:
binlog_cache_size, events spill to a local temporary file.Solution
We propose an optimization where large transactions write binlog events directly to a dedicated temporary file on persistent storage and finalize the commit with a rename, eliminating the data-copy bottleneck entirely.
Write Protocol
When a transaction's binlog cache exceeds a configurable threshold (default: 128 MB), the engine switches strategies:
binlog_large_commit_thresholdto determine when to spillover. Transactions whose binlog cache exceeds this threshold trigger the direct-write path.Recovery
The design ensures that a large transaction's temp file either becomes a complete, valid binlog file or is discarded entirely:
Because the large transaction is isolated in its own binlog file, recovery never needs to scan gigabytes of data. The engine retrieves the XID from a known position near the beginning of the file.
Collaboration Proposal
We are prepared to contribute the implementation of this optimization to upstream MySQL. The code has been battle-tested in production for over five years, and the design has been validated independently by MariaDB's adoption.
We welcome discussion on integration approach, coding standards alignment, and any design modifications Oracle considers necessary for upstream inclusion.