Feature #684: In-memory Relaylog by yangxt-aws · Pull Request #743 · mysql/mysql-server

yangxt-aws · 2026-09-03T21:08:38Z

Primary issue: In-Memory Relaylog #684
Design: design/replication/684-in-memory-relaylog

What does this change do?

Why is it needed?

How was it tested?

Added/updated MTR tests under mysql-test/
scripts/ci/mtr.sh passes locally
Ran the relevant full suite (name it): rpl, rpl_nogtid, rpl_gtid.

Contributor checklist

Code is formatted (scripts/ci/format.sh)
Commits are focused with descriptive messages

AI assistance

I did not use AI assistance for this contribution
I used AI assistance for this contribution

If AI assistance was used, describe the tool(s) and extent of use:

AI is used for

code generation
test generation
refactoring
review

Areas touched

replication

oracle-contributor-agreement · 2026-09-03T21:08:44Z

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

PR author: yangxt-aws
yangxt@amazon.com (@yangxt-aws)

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

- Primary issue: mysql#684 - Design: `design/replication/684-in-memory-relaylog`

NOTE: This is a test level commit for high level design proposal. What's implemented in this commit: 1. Add IMR CRST command with configurations. 2. Implement end-to-end memory-path relaylog. 3. Implement end-to-end spill-path for large transaction. 4. Add IO/SQL thread recovery (single thread stop/start) 5. Add unittests for introduced class/functions. 6. Add end to end MTR tests.

wxueting-aws · 2026-09-04T20:54:07Z

-
-ER_BULK_READER_PATH_IS_DIRECTORY
-  eng "Bulk Reader: Path is a directory, expected a file: %s"
+ER_CRST_IN_MEMORY_RELAYLOG_ONLY_FOR_CSA


There is merge conflicts on this file, please resolve

Thank you, already resolved.

yangxt-aws · 2026-09-04T21:06:10Z

To whom it may concern, the published code change in this PR is for demonstration and testing only.
It includes:

CRST for IMR.
IMR normal memory path.
IMR large transaction spill path.
IMR IO/SQL recovery.
Unittests + MTR tests

The code is for validation and map the design to the code. More changes may be submitted in the following development process.

oracle-contributor-agreement · 2026-09-07T14:12:33Z

Thank you for signing the OCA.

karolina-szczepankiewicz

The design is in good shape. I reviewed the design and used the provided code only for validation. I requested clarification where the implementation and design diverge. I recommend resolving some comments through discussion, but most can be addressed during implementation and code review. Thank you!

karolina-szczepankiewicz · 2026-09-08T08:18:01Z

  assert(m_fde_ptr != nullptr);
 }

+Cached_event_payload::~Cached_event_payload() {


Great. Would you like to report this as a bug and submit the fix as a separate contribution / a separate PR?

karolina-szczepankiewicz · 2026-09-08T08:19:05Z

+| ---------- | -----------------: | ------------------------------: |
+| Receiver   |                 80 |                             230 |
+| Applier    |                 66 |                              67 |
+| End-to-end |                 44 |                              65 |


Great improvement!

karolina-szczepankiewicz · 2026-09-08T08:33:29Z

+The coordinator consumes transactions through the existing `Transaction_provider` interface, so the `Csa_service::run` loop, scheduler, dependency tracking, and worker pool are all reused unchanged. The changes on the coordinator side are limited to the `Reader` interface.
+A new reader `Queued_transaction_reader` is introduced as a memory path counterpart of `Relay_log_adaptive_reader`. Its `read()` takes the envelope at `dispatch_seqno` from the `Trx_envelope_queue`, advances the cursor, and wraps the envelope's `Fetchable_transaction` into a fresh `Job_applier` using the same constructor the relaylog reader uses today.
+Everything downstream is untouched. The coordinator obtains the next transaction from the provider, computes scheduling dependencies from the logical clock and commit-order inputs, and dispatches it to the worker thread pool exactly as today.
+The coordinator also tries to advance `commit_seqno` in the queue over a contiguous run of committed transactions at the head and dequeues each. A transaction that commits behind an uncommitted head is only marked committed and swept later when the head commits.


I had a comment about the previous approach, but you changed it to lazy advancement of the commit LWM by the coordinator, which is great. The only remaining item is a short explanation of why the queue maintains a committed prefix separately from the CSA scheduler's existing Clock_lwm_registry, which also computes a contiguous completed task prefix regardless of the value of replica_preserve_commit_order.

karolina-szczepankiewicz · 2026-09-08T08:35:41Z

@@ -0,0 +1,81 @@
+### Functional Requirements
+
+- FR1. The feature must provide a per-channel option, introduced as a new `CHANGE REPLICATION SOURCE TO` option `IN_MEMORY_RELAYLOG_ENABLED`, that controls whether the feature is enabled. It is `ON` by default for eligible channels. When `OFF`, replication must fall back to the standard disk path relaylog.


The implementation defaults IN_MEMORY_RELAYLOG_ENABLED to OFF. Please update the design to use OFF consistently. With this default, IMR is enabled only through an explicit CHANGE REPLICATION SOURCE TO statement, and no separate default is needed for upgraded channels.

IMHO we should keep the default as OFF, given that users expect durability of relay logs by default.
User stories on 667fcff#diff-8bcbe8983b9d223e1cb768c4a672500fb8844b4b9e8638d6d8fb1023d6fef248 also need to be updated.

karolina-szczepankiewicz · 2026-09-08T08:43:07Z

+- FR38. An error diagnostic file created by the feature must have a name matching the pattern `imr_err_<channel>_<unique_id>`, where `<channel>` is a lowercase channel identifier and `<unique_id>` is a lowercase identifier unique within `#in_memory_relaylog_temp_files` (imr_err stands for "In-Memory Relaylog Error").
+- FR39. Startup cleanup must not delete temp error diagnostic files, contrary to FR25.
+- FR40. If the feature encounters a non-recoverable error, the user must be able to fall back to the standard disk path relaylog and resume replication without data loss by stopping the channel, setting `CHANGE REPLICATION SOURCE TO` with `IN_MEMORY_RELAYLOG_ENABLED = OFF`, and restarting it.
+


It's good that FR30-FR30.2 now cover the partial transaction on STOP REPLICA IO_THREAD. The remaining thing is disabling IMR through CRST. Please add one functional requirement stating that the threads must be stopped, the IMR queue is removed, and the next start uses the disk relay log. Please also state what happens to spill and diagnostic files.

karolina-szczepankiewicz · 2026-09-08T11:04:01Z

+
+### Configuration / Knobs — New configuration clauses or options
+
+All In-Memory relaylog settings are per channel. They are exposed as `CHANGE REPLICATION SOURCE TO` clauses. Each clause is a per-channel value held in the channel's in-memory `Master_info` and backed by the persisted replication metadata repository. They are effective only on a CSA channel with asynchronous replication. Setting them on any other channel type is rejected with an error.


Please change Master_info to Relay_log_info in this paragraph

karolina-szczepankiewicz · 2026-09-08T11:11:48Z

+Recovery
+
+The queue is purely an in-memory struct that cannot persist through replication stop, server restart, or crash. The recovery process therefore relies only on the durably recorded `gtid_executed`. Any uncommitted transaction can be re-obtained by GTID, either re-dispatched from the queue if it still holds the transaction (in the case of `STOP REPLICA SQL_THREAD`), or re-fetched from the source by auto-positioning. Duplicates are harmless because a worker skips any transaction whose GTID is already in `gtid_executed` (`is_already_logged_transaction`), and an interrupted transaction is always rolled back and re-fetched during recovery.
+`STOP REPLICA SQL_THREAD` (applier only; receiver keeps running). The queue and all uncommitted envelopes still stay in memory. In-flight jobs are driven to a terminal state (commit, or roll back), so no transaction is left half-applied. With no advance of `commit_seqno`, all uncommitted transactions remain in the queue. The IO thread keeps enqueuing new envelopes until the memory usage counter reaches the `IN_MEMORY_RELAYLOG_LIMIT`. On next `START REPLICA SQL_THREAD`, the coordinator rewinds `dispatch_seqno` to `commit_seqno` and re-dispatches all the retained uncommitted transactions in order; committed-but-not-yet-swept envelopes are skipped because their payload is freed.


The implementation performs one final sweep_committed() after the SQL thread and workers stop. Please change this paragraph if applicable.

karolina-szczepankiewicz · 2026-09-08T11:14:46Z

+
+In-memory relaylog keeps the transaction in memory instead. As a new transaction arrives, the receiver wraps the transaction metadata and incoming encoded events into a data structure, pushes it to an in-memory queue. The coordinator reads from it and dispatches the transaction to workers, without relaylog file write/read during the process (referred to as the memory path in the document).
+
+The queue has a hard memory limit per channel. A transaction larger than a configured threshold is re-routed to a temporary file in standard relaylog format (referred to as the spill path in the document). On replication stop, server restart, or crash, the queue is emptied and any uncommitted transactions are re-fetched by GTID auto-positioning.


Please change On replica stop to On full replication channel stop

karolina-szczepankiewicz · 2026-09-08T11:19:24Z

+
+**NAME**: `IN_MEMORY_RELAYLOG_LIMIT`
+
+**VALUES:** unsigned integer, bytes. Range [33554432 (32 MB), 4294967296 (4 GB)]; values outside the range are rejected.


Please note that this range will require a 64 bit integer. If you would like to keep a 32 bit value, please change the maximum to 4 GiB - 1.

karolina-szczepankiewicz · 2026-09-08T11:21:59Z

+
+`SHOW RELAYLOG EVENTS` is not applicable
+
+`FLUSH RELAY LOGS` is a no-op.


Please note that the implementation must block standard relay log rotation to keep this operation a no-op.

nacarvalho · 2026-09-08T16:04:42Z

+otherwise                             -> block until commits free space, then memory path
+```
+
+Reserving the exact `trx_length` up front guarantees a memory-path transaction fits before streaming begins. A transaction larger than the `IN_MEMORY_RELAYLOG_SPILL_THRESHOLD` always takes the spill path.


It would be good to list the reasons to have both IN_MEMORY_RELAYLOG_SPILL_THRESHOLD and IN_MEMORY_RELAYLOG_LIMIT.

nacarvalho

Thank you for the contribution.
I only looked into the design, please see my comments inline.

nacarvalho

Pending comment.

nacarvalho · 2026-09-08T16:13:24Z

+
+**Out of Scope / Limitations**
+
+The feature is effective only on CSA channels. It is not supported for `Group Replication` channels, or `Semisynchronous` replication. Enabling this feature on such channels shall be rejected.


Need to be clarified that for Group Replication, only group_replication_applier channel cannot enable it.
group_replication_recovery channel can enable it, same way it can enable CSA.
This also impacts FR2.

oracle-contributor-agreement Bot added the OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. label Sep 3, 2026

github-actions Bot added Docs Changes touching documentation Replication Changes touching replication or binlog code Tests Changes touching test code or test data labels Sep 3, 2026

yangxt-aws changed the title ~~Design review for in-memory relaylog~~ In-memory Relaylog Sep 4, 2026

yangxt-aws added 3 commits September 4, 2026 20:53

Design review for in-memory relaylog

667fcff

- Primary issue: mysql#684 - Design: `design/replication/684-in-memory-relaylog`

Fix Cached_event_payload possible mem leak.

c64c307

yangxt-aws force-pushed the in-memory-relaylog-design branch from f826ea3 to ca86326 Compare September 4, 2026 20:55

wxueting-aws reviewed Sep 4, 2026

View reviewed changes

yangxt-aws changed the title ~~In-memory Relaylog~~ In-memory Relaylog #684 Sep 4, 2026

yangxt-aws changed the title ~~In-memory Relaylog #684~~ In-memory Relaylog Sep 4, 2026

yangxt-aws changed the title ~~In-memory Relaylog~~ Feature #684: In-memory Relaylog Sep 4, 2026

yangxt-aws marked this pull request as ready for review September 4, 2026 21:11

yangxt-aws requested a review from a team September 4, 2026 21:11

yangxt-aws requested review from gopshank and seemasundara as code owners September 4, 2026 21:11

nacarvalho requested review from karolina-szczepankiewicz and nacarvalho September 7, 2026 09:52

oracle-contributor-agreement Bot added OCA Verified All contributors have signed the Oracle Contributor Agreement. and removed OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. labels Sep 7, 2026

github-actions Bot added the Review Requested Review requested from code owners label Sep 7, 2026

karolina-szczepankiewicz reviewed Sep 8, 2026

View reviewed changes

nacarvalho reviewed Sep 8, 2026

View reviewed changes

		@@ -0,0 +1,81 @@
		### Functional Requirements

		- FR1. The feature must provide a per-channel option, introduced as a new `CHANGE REPLICATION SOURCE TO` option `IN_MEMORY_RELAYLOG_ENABLED`, that controls whether the feature is enabled. It is `ON` by default for eligible channels. When `OFF`, replication must fall back to the standard disk path relaylog.


		### Configuration / Knobs — New configuration clauses or options

		All In-Memory relaylog settings are per channel. They are exposed as `CHANGE REPLICATION SOURCE TO` clauses. Each clause is a per-channel value held in the channel's in-memory `Master_info` and backed by the persisted replication metadata repository. They are effective only on a CSA channel with asynchronous replication. Setting them on any other channel type is rejected with an error.


		In-memory relaylog keeps the transaction in memory instead. As a new transaction arrives, the receiver wraps the transaction metadata and incoming encoded events into a data structure, pushes it to an in-memory queue. The coordinator reads from it and dispatches the transaction to workers, without relaylog file write/read during the process (referred to as the memory path in the document).

		The queue has a hard memory limit per channel. A transaction larger than a configured threshold is re-routed to a temporary file in standard relaylog format (referred to as the spill path in the document). On replication stop, server restart, or crash, the queue is emptied and any uncommitted transactions are re-fetched by GTID auto-positioning.


		NAME: `IN_MEMORY_RELAYLOG_LIMIT`

		VALUES: unsigned integer, bytes. Range [33554432 (32 MB), 4294967296 (4 GB)]; values outside the range are rejected.


		`SHOW RELAYLOG EVENTS` is not applicable

		`FLUSH RELAY LOGS` is a no-op.


		Out of Scope / Limitations

		The feature is effective only on CSA channels. It is not supported for `Group Replication` channels, or `Semisynchronous` replication. Enabling this feature on such channels shall be rejected.

Uh oh!

Conversation

yangxt-aws commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this change do?

Why is it needed?

How was it tested?

Contributor checklist

AI assistance

Areas touched

Uh oh!

oracle-contributor-agreement Bot commented Sep 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yangxt-aws commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oracle-contributor-agreement Bot commented Sep 7, 2026

Uh oh!

karolina-szczepankiewicz left a comment

Choose a reason for hiding this comment

Uh oh!

karolina-szczepankiewicz Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nacarvalho left a comment

Choose a reason for hiding this comment

Uh oh!

nacarvalho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yangxt-aws commented Sep 3, 2026 •

edited

Loading

yangxt-aws commented Sep 4, 2026 •

edited

Loading

karolina-szczepankiewicz Sep 8, 2026 •

edited

Loading