Skip to content

[fix][test] Cover scheduler shutdown membership-lock deadlock - #26557

Merged
merlimat merged 3 commits into
apache:masterfrom
lhotari:fix-flaky-SchedulerManager-close-deadlock
Sep 13, 2026
Merged

[fix][test] Cover scheduler shutdown membership-lock deadlock#26557
merlimat merged 3 commits into
apache:masterfrom
lhotari:fix-flaky-SchedulerManager-close-deadlock

Conversation

@lhotari

@lhotari lhotari commented Sep 12, 2026

Copy link
Copy Markdown
Member

Motivation

PR #26556 fixed the lock-order deadlock in SchedulerManager.close() and added bounded shutdown for stuck scheduling rounds. Add regression coverage to ensure membership queries can still complete while shutdown waits for the scheduler lock.

The deadlock was observed during PulsarFunctionTlsTest.tearDown in Broker Group 2 in PR #26547: the test worker held the manager monitor while waiting for schedulerLock, while the scheduler held schedulerLock and waited for the monitor to read membership.

Modifications

Add a bounded regression test that holds the exposed scheduler lock, waits until shutdown queues for it, and checks that a membership query can still finish. Verify that the producer remains open until the lock is released and then closes successfully. Cleanup releases the lock even on failure so the test detects the inversion without hanging the test JVM.

This PR retains the shutdown implementation from #26556 and adds only test coverage; it makes no production-code changes.

Verifying this change

  • Make sure that the change passes the CI checks.

  • After merging upstream master, all 15 tests in SchedulerManagerTest pass with retries disabled, including the membership-lock regression test and the upstream stuck-scheduling-round test.

  • ./gradlew quickCheck passes.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Preserve the upstream bounded scheduler shutdown implementation and retain the membership-lock regression test.
@lhotari lhotari changed the title [fix][fn] Avoid scheduler shutdown deadlock [fix][test] Cover scheduler shutdown membership-lock deadlock Sep 13, 2026
@merlimat
merlimat merged commit 6075c2b into apache:master Sep 13, 2026
42 checks passed
@lhotari lhotari added this to the 5.0.0 milestone Sep 14, 2026
lhotari added a commit that referenced this pull request Sep 14, 2026
lhotari added a commit that referenced this pull request Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants