Skip to content

Add convergence visibility - #211

Open
LZD-PratyushBhatt wants to merge 1 commit into
devfrom
lzd/convergence-tracker
Open

Add convergence visibility#211
LZD-PratyushBhatt wants to merge 1 commit into
devfrom
lzd/convergence-tracker

Conversation

@LZD-PratyushBhatt

@LZD-PratyushBhatt LZD-PratyushBhatt commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Compute bounded convergence reports asynchronously and expose typed cluster and resource REST
views without adding controller-path or metadata-store pressure.

Issues

  • N/A. This PR is not linked to an Apache Helix GitHub issue.

Description

  • This PR adds opt-in, customer-facing visibility into whether Helix assignments have
    converged.

The controller compares CurrentState with BestPossibleState and uses pending, selected, throttled,
successfully dispatched, and failed transition messages to classify each partition as:

  • CONVERGED
  • IN_PROGRESS
  • BLOCKED
  • UNKNOWN
  • PAUSED

The resulting typed ConvergenceStatus records include bounded partition diagnostics, aggregate
counts, actionable reasons, WAGED target freshness, optimizer health, report age, and controller
session metadata.

Reports are persisted under:

/{cluster}/PROPERTYSTORE/HELIX_CONVERGENCE_STATUS
/{cluster}/PROPERTYSTORE/HELIX_CONVERGENCE_STATUS/{resource}

The following REST endpoints expose the reports:

GET /clusters/{cluster}/convergence
GET /clusters/{cluster}/resources/{resource}/convergence

Performance and safety properties:

  • Monitoring is disabled by default through CONVERGENCE_MONITORING_ENABLED.
  • Full partition calculation and persistence run on a dedicated asynchronous worker.
  • Pending controller events are deduplicated.
  • Controller pipeline submission measured 0 ms in normal live testing, occasionally 1 ms.
  • Persistence is rate-limited to at most once per second.
  • Unchanged reports are not rewritten.
  • Existing convergence records are loaded once per controller session and cached.
  • Diagnostic output is bounded to 20 partitions and 64 assignment entries per side.
  • A 10,000-partition bounded calculation completed in approximately 54 ms off-thread.
  • Writes are controller-session fenced and use an incomplete-root/complete-root protocol.
  • REST rejects missing, incomplete, unsupported, or stale-controller reports.

There are no UI changes.

Tests

  • The following unit and integration tests cover this change:

  • TestConvergenceStatusCalculator

  • TestConvergenceStatusPersistStage

  • TestWagedRebalanceStatus

  • TestConvergenceStatusResponseMapper

  • TestConvergenceStatusAccessor

  • TestPropertyPathBuilder

  • TestPropertyKeyGetPath

Core focused test result:

mvn -pl helix-core \
  -Dtest='TestPropertyPathBuilder,TestConvergenceStatusPersistStage,TestConvergenceStatusCalculator' \
  test

Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

REST end-to-end test result:

mvn -pl helix-rest -am \
  -Dtest='TestConvergenceStatusAccessor,TestConvergenceStatusResponseMapper' \
  -Dsurefire.failIfNoSpecifiedTests=false test

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

Repository build result:

mvn clean install -Dmaven.test.skip.exec=true

BUILD SUCCESS

Live validation was also performed against a standalone ZooKeeper server with real controller,
participant, and REST processes. The scenarios covered:

  • Initial assignment convergence
  • Dynamic resource addition
  • Participant scale-out, loss, and rejoin
  • Cluster pause and resume
  • Maintenance mode entry and exit
  • Controller loss and recovery with stale-report detection
  • REST restart using persisted reports
  • WAGED enablement and instance movement
  • Resource deletion and PropertyStore cleanup
  • A deliberately long-running participant transition that remains IN_PROGRESS

Changes that Break Backward Compatibility (Optional)

  • None.

All changes are additive. Convergence monitoring is disabled by default, existing clusters do not
perform convergence calculation or persistence unless explicitly enabled, and existing REST APIs
are unchanged.

Documentation (Optional)

  • N/A. No wiki page was added.

The REST contract and configuration behavior are documented by the model, endpoint implementation,
and test coverage in this PR.

Commits

  • The commit follows the project commit-message guidelines:
    1. Subject is separated from the body by a blank line.
    2. Subject is under 50 characters.
    3. Subject does not end with a period.
    4. Subject uses the imperative mood.
    5. Body explains what and why.

No issue reference is included because this PR is not linked to an Apache Helix issue.

Code Quality

  • The diff follows existing Helix Java formatting conventions.
  • git diff --check passes.
  • The code compiles for the repository's JDK 8 and JDK 11 build targets.

Compute bounded convergence reports asynchronously and expose typed cluster and resource REST views without adding controller-path or metadata-store pressure.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant