Skip to content

Rollback Prepared Transactions Asynchronously During Binlog Crash Recovery - #711

Open
SongLibing wants to merge 1 commit into
mysql:trunkfrom
SongLibing:binlog_recovery_async_rollback
Open

Rollback Prepared Transactions Asynchronously During Binlog Crash Recovery#711
SongLibing wants to merge 1 commit into
mysql:trunkfrom
SongLibing:binlog_recovery_async_rollback

Conversation

@SongLibing

@SongLibing SongLibing commented Aug 4, 2026

Copy link
Copy Markdown

Problem:

Binlog recovery can block server startup for a long time when it must roll back a large prepared internal XA transaction. The rollback runs synchronously before the server becomes available.

Solution:

Add an optional storage engine callback, recover_rollback_by_xid, for recovery-time rollback. During binlog recovery, InnoDB uses the callback to persistently convert a recovered prepared DML transaction back to ACTIVE state. The background recovery rollback thread then performs the expensive undo work asynchronously. DDL transactions still fall back to the normal synchronous rollback_by_xid path.

Add a binlog MTR test that covers prepared DML rollback, background rollback handoff, and repeated-crash safety before the rollback state change is flushed.

Copyright (c) 2026, Oracle and/or its affiliates.

What does this change do?

mysql/mysql-community#100

How was it tested?

  • Added/updated MTR tests under mysql-test/
  • scripts/ci/mtr.sh passes locally
  • Ran the relevant full suite (name it): ______

Contributor checklist

  • I have signed the OCA with the email on these commits
  • Code is formatted (scripts/ci/format.sh)
  • Commits are focused with descriptive messages

AI assistance

  • I did not use AI assistance for this contribution
  • I used AI assistance for this contribution

If AI assistance was used, describe the tool(s) and extent of use: codex, only merge code from AliSQL to trunk.

Areas touched

replication, innodb

@oracle-contributor-agreement

Copy link
Copy Markdown

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

  • PR author: SongLibing

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

@oracle-contributor-agreement oracle-contributor-agreement Bot added the OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. label Aug 4, 2026
@github-actions github-actions Bot added MTR Failed MTR suite failed InnoDB Changes touching InnoDB storage engine code Tests Changes touching test code or test data Build Failed PR build failed labels Aug 4, 2026
@oracle-contributor-agreement

Copy link
Copy Markdown

Thank you for signing the OCA.

@oracle-contributor-agreement oracle-contributor-agreement Bot added OCA Verified All contributors have signed the Oracle Contributor Agreement. and removed OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. labels Aug 5, 2026
@github-actions
github-actions Bot requested a review from gopshank August 5, 2026 12:02
@github-actions github-actions Bot added the Review Requested Review requested from code owners label Aug 5, 2026
@gopshank
gopshank requested review from nacarvalho and removed request for gopshank and seemasundara August 5, 2026 14:45
@SongLibing
SongLibing force-pushed the binlog_recovery_async_rollback branch from e6d0129 to c85ef66 Compare August 6, 2026 12:36
@SongLibing
SongLibing requested a review from gopshank as a code owner August 6, 2026 12:36
@SongLibing
SongLibing requested a review from a team August 6, 2026 12:36
@github-actions
github-actions Bot requested a review from seemasundara August 6, 2026 12:36
@SongLibing
SongLibing force-pushed the binlog_recovery_async_rollback branch from c85ef66 to 6d6e94a Compare August 6, 2026 21:35
@henrikingo-oracle

Copy link
Copy Markdown

Thank you SongLibing for this.

We've uncovered that internally in Oracle this same issue and solution(s) have been discovered a couple years ago, discussed and documented, but not prioritized. We've made https://bugs.mysql.com/bug.php?id=114053 public so that it may benefit the review of this PR.

The review will take its time, we will probably want both a member from replication team as well as InnoDB team to look at this and this is obviously very critical part of the database, so they will take their time. Other than that everything seems to be in order now, oca signed and tests green.

@SongLibing
SongLibing force-pushed the binlog_recovery_async_rollback branch from 6d6e94a to 5be807a Compare September 3, 2026 12:04
@github-actions github-actions Bot added Build Changes touching build or GitHub automation Docs Changes touching documentation and removed Build Failed PR build failed MTR Failed MTR suite failed labels Sep 3, 2026
@SongLibing
SongLibing force-pushed the binlog_recovery_async_rollback branch 2 times, most recently from 23d5eee to 23b5313 Compare September 4, 2026 05:39
@github-actions github-actions Bot removed the Build Changes touching build or GitHub automation label Sep 4, 2026
Crash Recovery

Description
===========
Binary log crash recovery resolves internal transactions
that were prepared in a storage engine before the server
stopped. If the binary log does not contain the transaction's
commit decision (Xid_log_event), the transaction must be rolled
back. Historically, the server invokes `rollback_by_xid()` in
the startup thread and waits for the complete rollback. Undoing
a large transaction row-by-row can therefore keep the server
unavailable for hours.

This change adds an optional recovery-specific storage-engine
callback to perform a fast rollback handoff. InnoDB uses it to
durably change the state of a prepared DML transaction back to
active. The existing InnoDB recovery rollback thread then does
the expensive row-by-row undo work in the background while
server startup continues. The final transaction outcome does
not change.

detail: design/server/100-async-rollback-during-binlog-recovery.md
@SongLibing
SongLibing force-pushed the binlog_recovery_async_rollback branch from 23b5313 to a2ea313 Compare September 4, 2026 07:40

@jujose-1 jujose-1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @SongLibing,

Thank you for the contribution. I reviewed the design from the replication and transaction-coordinator recovery perspective (the InnoDB-specific aspects would still need review from the InnoDB team).

The overall approach looks reasonable to me and appears to preserve the existing transaction outcome. I’ve left a few inline comments for clarification. Please take a look when you have a chance.

Thank you for working on this.

Regards,
Justin Jose

rollback thread then does the expensive row-by-row undo work in the background
while server startup continues. The final transaction outcome does not change.

## Functional Requirements

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for documenting the requirements. Could you please consider expanding and regrouping this section? The current FR1 appears to describe a performance property, while NFR1 describes functional correctness.

Would it make sense to cover the following requirement areas?

Functional requirements

  • Transaction outcome and eligible transaction scope
  • Asynchronous handoff and synchronous fallback
  • Callback error handling and multi-engine transactions
  • Crash, restart, and shutdown behavior

Non-functional requirements

  • Startup-time improvement and its measurable boundary
  • Replication, GTID, point-in-time recovery (PITR), and upgrade/downgrade compatibility
  • Resource usage and observability

Comment on lines +169 to +173
There is no change to external XA recovery: the new callback is selected for
internal transactions by the existing binlog recovery decision path. There is
also no change to security checks, SQL interfaces, replication protocols, or
user-visible configuration. The interaction with the existing
`innodb_force_recovery` setting is described above.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please expand the impact assessment slightly beyond protocol compatibility?

It would be useful to confirm that GTID recovery, the recovered binlog position, and point-in-time recovery preserve the same final transaction outcome. It would also help to describe whether asynchronous replication or Group Replication can start while rollback is still running and what happens when an applier transaction conflicts with locks retained by the recovered transaction.

There is no change to external XA recovery: the new callback is selected for
internal transactions by the existing binlog recovery decision path. There is
also no change to security checks, SQL interfaces, replication protocols, or
user-visible configuration. The interaction with the existing

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design states that there is “no change to … user-visible configuration,” which appears consistent with the “Configuration / Knobs: No change” section in Issue #100.

However, the issue also refers to synchronous fallback “when the feature is disabled,” requests coverage for “disabled-feature behavior,” and shows a recovery path where the “option is ON.”

Could you please clarify the intended behavior and align the issue and design accordingly? If “enabled” refers only to whether a storage engine implements the optional callback, perhaps the option and disabled-feature wording in the issue could be updated. Otherwise, it would be helpful to describe the configuration mechanism in the design.

- `storage/innobase/trx/trx0trx.cc`: scheduling tables for metadata-lock
acquisition before background rollback.

## Alternatives Considered

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the proposal adds a new storage-engine callback, could you please briefly explain why a recovery-specific callback is preferred over changing the behavior of the existing rollback_by_xid() callback?

It may also be useful to mention that adding a separate rollback worker was not selected because the existing InnoDB recovery rollback thread can already perform this work. A few sentences should be sufficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Docs Changes touching documentation InnoDB Changes touching InnoDB storage engine code OCA Verified All contributors have signed the Oracle Contributor Agreement. Review Requested Review requested from code owners Tests Changes touching test code or test data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants