feat(client): spawn the builder preferences service

[shane-moore]

Note

Stacked on #1282 (feat/1278-sign-request-auth); the diff shows its commits until it merges. Only the head commit is this PR. Merge/deploy order for the stack: #1282 -> this PR -> #1283, with no soak or release image from an intermediate head.

Problem, Evidence, and Context

Closes #1280. Last piece of the direct-builder-connections milestone: #1282 wired the BuilderStore and RequestAuthCache into the block service (both are required by BlockServiceBuilder::build() at the Lighthouse pin), but nothing publishes builder preferences ahead of time and nothing prunes the cache. RequestAuthCache::prune has exactly one caller in the entire pin, inside BuilderPreferencesService, so until that service runs the block service's cache is insert-only.

One premise of the issue text went stale at the pin: there is no BuilderConfig setter on the BlockServiceBuilder. The block service resolves the builder config per proposal itself, and #1282 already shipped that produce-time half. What remained here is the service spawn plus single-instance sharing.

Change Overview

One file, anchor/client/src/lib.rs. A single RequestAuthCache is hoisted next to the BuilderStore construction and both are clone-shared (both types are Arc-backed) between the block service and a newly spawned BuilderPreferencesService, started inside the existing is_gloas_scheduled block beside ProposerPreferencesService. This mirrors Lighthouse's own VC wiring one for one.

Read the diff top to bottom: the import, the hoist plus comment rewrite at the block-service builder, then the spawn in the Gloas block.

Not changed: the produce-time path (shipped in #1282), the builder definitions validation (#1283), and Gloas-unscheduled networks, where the service does not spawn and the cache has no insert path either.

Risks, Trade-offs, and Mitigations

The diff adds no Anchor logic but activates a pinned Lighthouse task. Known degraded modes at this pin, all confined to a feature that is off by default (empty builders file):

• Elapsed-duty re-resolution: the service revisits passed current-epoch proposers each tick while the cache prunes them, so Anchor returns DeclinePastSlot for every elapsed proposal slot for the rest of its epoch. The decline is free by design (feat(validator_store): sign builder request auth #1282: no collection, no broadcast, no partial-signature traffic, excluded from the failure reporter), so the cost is repeated per-slot resolution work rather than anything operators see. Re-pin ask: elapsed-slot filter in the service loop. CORRECTED after the live run: an earlier version of this line claimed the decline produced a per-builder error log from Lighthouse's builder_store every slot. It does not. Anchor's log filter (anchor/logging/src/utils.rs) allowlists workspace crates plus five Lighthouse crates and builder_store is not among them, so that crate's output is dropped entirely. Measured: 17 debug decline lines and zero error or warn lines mentioning builders, at --debug-level debug.
• Head-of-line blocking: the service resolves proposers serially and only POSTs after scanning both epochs, so a sub-quorum entry (rolling config change window, or the byte-divergence misconfig the docs warn about) delays other builders' preferences by the 2-slot future signing bound. Block production is unaffected (independent produce-path resolve with a current-slot 1s bound). Conditional re-pin asks (per-proposer incremental publish, bounded concurrency), to be judged from soak evidence.
• Whole-chunk resubmit: any builder failure makes the BN return an indexed error, the VC marks none of the chunk sent, and the chunk is re-POSTed each tick. Stock Lighthouse behavior; builders must tolerate replay regardless (the service re-sends on restart).

None of these can reach a release: this PR stays draft until the Lighthouse re-pin, where the asks above get resolved or the residual modes get explicitly accepted.

Validation

• make cargo-fmt, make cargo-fmt-check, make lint: clean.
• cargo check --workspace --all-targets: clean.
• The issue's spawn-gating and produce-body unit tests are not implementable from Anchor: the spawn sits in the monolithic client start flow (no test harness; PayloadAttestationService and ProposerPreferencesService shipped identically) and the produce body is pinned Lighthouse code from feat(validator_store): sign builder request auth #1282, unchanged here.

Validated live on ssv-mini 2026-08-27, the first end-to-end exercise of the Gloas direct-builder preference flow. Profile: 4 Anchor operators (quorum 3), 10 managed validators, Gloas at epoch 2, a beacon node built from Anchor's exact Lighthouse pin 44f442479 (replacing Lodestar, which has no such route), and a builder receiving the submissions.

• Ahead-of-time publication: at head slot 45, still pre-Gloas, the builder had accepted preferences for slots 72, 79 and 91. Cross-checked against the beacon node's own /eth/v1/validator/duties/proposer/2, those are exactly this cluster's three proposer duties in epoch 2 out of 32 slots, with no misses and nothing spurious. The per-epoch fork gate held: no pre-Gloas slot was ever published.
• Service spawn: "Builder preferences service started" once per operator on all four; async_tasks_count{async_task_count="builder_preferences_service"} 1.
• Shared cache works: anchor_signed_request_auth_total{status="success"} held at 9, one per duty, while the service re-resolved the whole builder config every slot. Repeat resolutions are cache hits, not re-signings, which is the property the single hoisted RequestAuthCache exists to provide.
• Publish-once dedup: exactly one submission per operator per duty, stable across ticks.
• Auth data derivation: the submitted bytes decode to the configured URL exactly.
• Post-fork continuation: entering epoch 2 the service published six further duties for epoch 3; all three epoch-2 proposals produced blocks.
• Zero publish failures across the run.

Rollback

Revert the commit; no config, data, or wire format changes. With the service unspawned, behavior returns to the #1282 state.

Blockers / Dependencies

Undraft gates, identical to the stack's plus soak scenarios: Lighthouse builder-API stack merges to unstable and Anchor re-pins (resolving or explicitly accepting the risk items above), migration probe rerun, ssv-spec kind-9 constant, go-ssv parity reverification, and an ssv-mini soak covering: a 2-of-4 sub-quorum entry plus a common builder across consecutive future proposer duties (common preference reaches the builder before each proposal slot), restart with past/current/future duties, empty config producing zero preference POSTs, with builder-side receipt as the observable.

Additional Info / Next Steps

After this merges, #1283 resolves a small known conflict in the same lib.rs region (its earlier open_and_validate construction plus this PR's clones and spawn).

[shane-moore]

@claude, please take a look

[claude-code-actions-sigp]

Claude finished @shane-moore's task in 3m 0s —— View job

---

Review: PR #1285 — spawn builder preferences service

• Read the diff against origin/epbs
• Review anchor/client/src/lib.rs (spawn + shared instances)
• Review validator_store changes (request_auth signing + tests)
• Review new test file testing/request_auth.rs
• Post review feedback

Scope note: This is a stacked PR. The only commit owned here is 89d01ca (anchor/client/src/lib.rs, +23/-7). The validator_store signing code, metrics, data_dir, and the 568-line test file are #1282's (e4f1fea), showing in the diff until it merges — I reviewed them for context but they're not this PR's to change.

Head commit (lib.rs) — the actual change

The wiring is correct:

• Move/clone semantics are sound. configured_builders and request_auth_cache are constructed once (lib.rs:732–735), .clone()d into the BlockServiceBuilder (lib.rs:743–744), and the originals moved into BuilderPreferencesService (lib.rs:884–885). No use-after-move; both types are Arc-backed so the clones genuinely share one store and one cache — which is the whole point (a second RequestAuthCache::default() would leave the block service's cache prune-less, i.e. insert-only). The rewritten comment at lib.rs:725–731 states this accurately.
• Gating is consistent with siblings. The spawn sits in the same is_gloas_scheduled() block as PayloadAttestationService and ProposerPreferencesService (lib.rs:854–890) and follows their new(...).start_update_service().map_err(...) shape one-for-one.
• No leak on non-Gloas networks. When Gloas is unscheduled, the two originals are simply dropped at scope end (the block service still got its clones). Clean.

One minor observation, not a blocker: configured_builders/request_auth_cache are constructed unconditionally (lib.rs:732–735) even on networks where is_gloas_scheduled() is false and neither the service nor any cache-insert path exists. This matches #1282's existing placement (the block service builder needs them regardless) and BuilderStore::open_or_create is cheap, so hoisting them into the Gloas gate would be churn for no real gain. Fine as-is.

I could not statically verify the BuilderPreferencesService::new argument order against the Lighthouse pin (the vendored source isn't in this checkout), but the PR reports cargo check --workspace --all-targets clean, which exercises that signature. Argument order there is Lighthouse's, not Anchor's to choose.

Stacked #1282 content (context only)

The sign_request_auth_v1 implementation and its tests are high quality and I have no correctness concerns:

• The slot-aware collection bound (request_auth_collection_bound, validator_store/src/lib.rs) uses a clean RequestAuthCollectionBound enum rather than a nested Option<Result<..>>, matching the repo's control-flow-clarity rule.
• Opaque request_auth.data is logged only by length (auth_data_len), never raw — good, honors the no-sensitive-payload rule.
• The known-answer vector test (request_auth.rs:531–568) independently re-derives the signing root with a hand-rolled Python SSZ script rather than round-tripping Lighthouse's own hashing, which makes it a genuine cross-check of the domain constant and field layout.
• The metric-reading tests correctly serialize on a tokio::sync::Mutex (METRIC_TEST_LOCK) because they assert deltas on the process-global prometheus counter — the right call to avoid flakiness.

Verdict

The head commit is a minimal, correct wiring change and is safe to review on its own merits. No changes requested. The degraded modes the description enumerates (elapsed-duty log noise, head-of-line blocking, whole-chunk resubmit) are all upstream/pin behavior confined to a feature that is off by default (empty builders file), and the PR correctly stays draft behind the Lighthouse re-pin gate. I did not run make lint/make test here; the PR reports fmt + lint + cargo check clean, and the wiring-level ACs are verified by reading the pin as the description states.
· feat/1280-wire-builder-preferences

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.87309% with 28 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (epbs@7c428cf). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
anchor/client/src/lib.rs	0.00%	18 Missing ⚠️
anchor/common/global_config/src/data_dir.rs	0.00%	3 Missing ⚠️
anchor/validator_store/src/testing/request_auth.rs	98.95%	3 Missing ⚠️
anchor/validator_store/src/lib.rs	97.77%	2 Missing ⚠️
...alidator_store/src/testing/proposer_preferences.rs	95.74%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             epbs    #1285   +/-   ##
=======================================
  Coverage        ?   78.50%           
=======================================
  Files           ?      176           
  Lines           ?    39092           
  Branches        ?        0           
=======================================
  Hits            ?    30689           
  Misses          ?     8403           
  Partials        ?        0           

Flag	Coverage Δ
rust	78.50% <93.87%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shane-moore]

Reviewed only the top commit, 89d01ca5, against its direct parent, d1a56cdd.

The Anchor wiring itself is correct: the block service and builder-preferences service share one Arc-backed store and request-auth cache, and the new service starts under the existing Gloas gate. I found one upstream behavior worth noting during soak:

1. The pinned service derives one current_fork, gathers duties from both the current and next epochs, then submits every entry with that current fork. In epoch gloas_fork_epoch - 1, it skips the current Fulu duties, gathers next-epoch Gloas duties, and sends their Gloas-only bodies with Eth-Consensus-Version: fulu. The merged validator API says the header names the version the submitted preferences belong to, while the builder spec defines a Gloas request body and says the header names the fork of the body's type. A builder that dispatches decoding by this required header can reject every pre-fork attempt, so the first Gloas epoch loses its intended ahead-of-time publication window. This behavior is in the pinned dependency, not the Anchor wiring reviewed here.

All CI checks, including run-local-testnet, pass at this head. The existing draft and soak gates remain.

Reviewed by gpt-5.6-sol max.