Skip to content

Add partition-weight guard rail for addWagedResource - #219

Open
sjainit wants to merge 4 commits into
linkedin:devfrom
sjainit:sarjain/helix-rest-guardrail-weight-rule
Open

Add partition-weight guard rail for addWagedResource#219
sjainit wants to merge 4 commits into
linkedin:devfrom
sjainit:sarjain/helix-rest-guardrail-weight-rule

Conversation

@sjainit

@sjainit sjainit commented Aug 4, 2026

Copy link
Copy Markdown

Summary

Adds a concrete guard rail rule on top of the now-merged guard rail framework (#213): PartitionWeightCapacityGuardrailRule, wired into the addWagedResource REST endpoint. It rejects — before the write reaches ZooKeeper — a WAGED resource whose per-partition weight is larger than any single instance could ever host, which would otherwise be accepted and only surface later as a WAGED rebalance failure.

This is the first rule added after the framework landed, so it also serves as a worked example of the framework's extension points (author a rule, add an additive context field, guard another endpoint).

The rule: PartitionWeightCapacityGuardrailRule

What it checks: WAGED places each partition replica on exactly one instance, so a partition can only ever be placed if — for every capacity dimension dsome instance has capacity_d >= weight_d. This rule rejects addWagedResource when a partition's effective weight in any dimension exceeds the largest single instance's capacity in that dimension, because such a partition is permanently unplaceable no matter how the cluster is arranged.

Gap it closes: today addWagedResource only validates that the required weight keys are present (WagedValidationUtil.validateAndGetPartitionCapacity checks key coverage, never magnitude). So a resource whose weight is larger than any instance can hold is accepted, the write succeeds, and it only surfaces later as a WAGED rebalance failure. This rule compares magnitude against real instance capacity on the write path, so the impossible resource is rejected before it reaches ZooKeeper.

It is deliberately a necessary (not sufficient) condition — it compares each dimension independently against the best instance in that dimension, so it never blocks a resource that could plausibly be placed; it only fails the provably-impossible cases. It is a no-op for non-WAGED clusters (no capacity keys), honors cluster-level default instance/partition weights (merged in the same way the rebalancer does), and evaluates the DEFAULT partition so resources relying purely on default weights are still checked.

How it uses the framework's extension points

  1. Author a rule — implement GuardrailRule: null-guard → read what it needs from the context (through the read-only ReadOnlyDataAccessor) → reuse the same weight/capacity primitives WAGED uses → return a ValidationResult (feasible, or an infeasible verdict with an actionable message). Mirrors the framework's LiveInstanceGuardrailRule.
  2. Add an additive context field — the rule needs the object the mutation would write, not just current cluster state, so GuardrailContext.proposedResourceConfig is added via the Builder. Existing rules/endpoints are untouched — this is the intended extensibility seam.
  3. Guard another endpoint — wired into ResourceAccessor.addResource (the addWagedResource command) with the same force / dryRun preflight the instance-delete endpoint already uses. No registry, no dispatch — the endpoint just builds new GuardrailPipeline(new PartitionWeightCapacityGuardrailRule()).

Behavior

PUT /clusters/{cluster}/resources/{resource}?command=addWagedResource (body = map of IdealState + ResourceConfig ZNRecords):

  • enforce (no query param): a partition weight exceeding max instance capacity → 400 + JSON verdict, resource not created; within-capacity → 200, created.
  • dry-run (dryRun=true): always 200 + verdict, resource never created (validation-probe semantics, consistent with the framework).
  • force (force=true): proceeds even for an over-weight resource → 200, created, with the overridden verdict logged.

Tests

  • Rule unit tests (TestPartitionWeightCapacityGuardrailRule, 8): null proposed config, null cluster config, no capacity keys (non-WAGED), no instance capacity, weight within capacity, weight exceeding capacity (DEFAULT → unscoped), per-partition override exceeding, and max-across-instances used. Mocked accessor + real ClusterConfig / InstanceConfig / ResourceConfig.
  • REST integration test (TestResourceAccessor#testAddWagedResourceWeightGuardrail): full Jersey + embedded ZK, exercising enforce / dryRun / force / within-capacity with status codes, the JSON verdict, and actual resource presence/absence. Saves and restores the cluster + instance capacity configuration in try/finally so it does not disturb sibling tests.

Validated on JDK 11: rule unit tests 8/8 and the full TestResourceAccessor suite 18/18 pass.

Not in scope

  • The stricter exact check ("does any single instance fit the whole weight vector"); this PR implements the per-dimension necessary condition by design.
  • Additional rules from the design catalog.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Block adding a WAGED resource whose per-partition weight exceeds the
largest single instance's capacity in any dimension, which would make
the partition permanently unplaceable. Existing addWagedResource
validation only checks that weight keys are present, never their
magnitude, so today such a resource is accepted into ZooKeeper and only
fails later at rebalance time. This rule closes that gap by
pre-validating the mutation on the REST endpoint.

- New PartitionWeightCapacityGuardrailRule computes, per capacity
  dimension, the maximum capacity advertised by any single instance and
  fails the mutation when a partition's effective weight exceeds it.
- GuardrailContext carries the proposed ResourceConfig so rules can read
  the to-be-written weights before the object exists in ZK.
- ResourceAccessor.addResource wires the rule into the addWagedResource
  path with force/dryRun, mirroring the existing instance-drop guard rail.
- Unit tests for the rule plus an integration test for the endpoint
  (enforce, dry-run, force bypass, within-capacity happy path).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sarthak Jain and others added 3 commits August 14, 2026 16:37
The PARTITION_CAPACITY_MAP is operator-supplied and can carry stale or
mistyped entries naming partitions the resource does not actually have
(e.g. leftovers after lowering NUM_PARTITIONS). WAGED ignores such ghost
entries at placement time and ZKHelixAdmin.validateWeightForResourceConfig
tolerates them on the write path, so blocking on them made the guard rail
stricter than the operation it fronts and produced false positives on
valid resources.

Thread the proposed IdealState through GuardrailContext and skip any
weight-map key that is neither DEFAULT nor a real partition of the
resource. Real partition names come from the ideal state's partition set
when populated, otherwise from NUM_PARTITIONS via Helix's canonical
<resource>_<index> naming (a freshly-proposed WAGED ideal state has no
assignment yet, so its partition set is empty at pre-validation time).
When no ideal state is supplied the rule falls back to evaluating every
key, preserving prior behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…acity, no misblame

Fixes three reviewer findings on the partition-weight guard rail:

1. force/dryRun were on the shared addResource dispatch method but only
   honored in the addWagedResource branch, so dryRun=true on a plain
   addResource silently performed a real write. Reject both flags with a
   400 for any command other than addWagedResource so a "simulation" can
   never mutate ZK.

3. The rule used getOrDefault(dimension, 0), so when the cluster declared
   a capacity key the instances did not advertise, a resource weight in
   that dimension was reported as "exceeds capacity 0" and the resource
   author was told to lower a weight that cannot go below 0. That missing
   capacity key is an instance-side misconfiguration already reported by
   WagedValidationUtil, so skip the dimension instead (mirroring the
   existing weight == null skip) rather than misblaming the resource.

5. maxInstanceCapacity folded over every InstanceConfig, including
   non-assignable ones (EVACUATE / SWAP_IN / UNKNOWN). WAGED only places
   on getAssignableInstanceConfigMap() instances, so counting a decommissioning
   instance's capacity let the rule certify a resource WAGED can never
   place. Filter to InstanceConfig.isAssignable() instances, matching the
   rebalancer. Also documented the fail-closed behavior of the
   getChildValues(..., true) instance-config read.

Adds unit tests for the assignable-only and missing-dimension cases and an
integration test asserting force/dryRun are rejected for a non-waged command.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tier 2 (linkedin#4): Document the cluster-wide blast radius of an unplaceable
WAGED resource (CAPACITY_DEFICIT stalls the global rebalance so nothing
added afterward gets placed) and stop suggesting force=true, which is the
exact action that triggers the deficit. The force capability itself is
retained.

Tier 3:
- linkedin#6: Collect and report every over-capacity partition/dimension in a
  single verdict (deterministic order: DEFAULT first then natural), so a
  caller sees all problems at once instead of fixing one and resubmitting.
- linkedin#7: Run cheap local structural checks (IdealState/ResourceConfig name
  match, non-negative weights) before building the guard rail context, so
  a dry-run reflects them and a structurally invalid request never reaches
  ZooKeeper. Negative-weight validation is explicit because the endpoint
  builds ResourceConfig from a raw ZNRecord that bypasses the setter's own
  check.
- linkedin#8: Order the guard rail integration tests after testAddResourceWithWeight
  via dependsOnMethods.

Adds a unit test for multi-violation aggregation/ordering and an
integration test asserting structural checks are applied before the guard
rail (including the raw-record negative-weight path).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sjainit

sjainit commented Aug 14, 2026

Copy link
Copy Markdown
Author

@LZD-PratyushBhatt thanks for the thorough review — all eight comments are addressed and each thread has an inline reply with details. Summary:

Must-fix

  • Reject force/dryRun for non-addWagedResource commands (6990e089d)
  • Ghost-partition false positive — only evaluate the IdealState's real partitions (0e47b04bf)
  • Skip (not zero-fill) capacity dimensions missing from the weight map (6990e089d)
  • Fold capacity only over assignable instances; documented fail-closed getChildValues (6990e089d)

Tier 2 / polish (1249ec570)

  • Dropped the force=true hint and documented the cluster-wide CAPACITY_DEFICIT blast radius
  • Aggregate all over-capacity violations in a deterministic order
  • Run cheap structural checks (name match, non-negative weights) before the guard rail so dry-run reflects them and invalid requests never hit ZK
  • Pinned the guard rail integration test with dependsOnMethods; added multi-violation + structural-check tests

Validation: rule unit tests 13/13, TestResourceAccessor 20/20. Resolving the threads; happy to reopen any you want to revisit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants