chaos harness: oracle & fault-injection can report a false PASS on the fork/halt conditions under test

[Richard1048576]

Context

I reviewed the codex/gravity-chaos-simple chaos toolkit (cluster/chaos/*) together with its design/status doc. The foundation is genuinely good — stake-weighted quorum math, real docker-backend network partitions, separate block-hash and state-root comparison, a panic/fatal log scan, receipt re-fetch, scoped heal, and an honest gap list.

The concern below is narrow but matters for a consensus-safety test tool: several defects let the harness report PASS on exactly the fork/halt conditions it exists to detect. A false PASS in a safety test is worse than no test — it manufactures confidence. Practical takeaway today: trust the FAIL signals (docker partition scenarios), don't yet trust a PASS until the items below are addressed.

Findings come from a multi-lens adversarial read; the #1 oracle bug was independently reproduced by two reviewers.

✅ Keep (working well)

• Stake-weighted BFT quorum (stake*3 > total*2) used consistently — not node count.
• Docker-backend splits are genuine partitions (fresh bridge nets, disconnect members from all original nets, reject host-network); correctly die on non-docker backends.
• State root read from block.stateRoot and compared separately from block hash.
• receipt_checker.verify_one re-fetches receipt and canonical block per node and compares blockHash/blockNumber.
• Heal is scoped (only the GRAVITY_CHAOS chain / the specific qdisc), no global iptables -F.

🔴 Correctness bugs — can report a false PASS (fix before trusting a PASS)

• Oracle no-fork / state-root guard is a dead tautology — cluster/chaos/lib/cluster.py:812-825. pass = len(hashes)==1 and len(common_block_samples)==len(nodes); the second clause is always true (one sample is appended per node regardless of RPC success, :800-801), and hashes/state_roots are sets built only from samples where rpc_ok and a truthy hash (:802-811). So a node on a divergent chain whose common-height requery times out / returns null is silently dropped, and the agreeing remainder passes. Worst case: a single responder ⇒ len(hashes)==1 ⇒ PASS. Fix: responders = [s for s in common_block_samples if s.get('rpc_ok') and s.get('hash')]; pass = len(responders)==len(nodes) and len({s['hash'] for s in responders})==1 (same for state_root); parallelize + lengthen the requery timeout so a slow-but-honest node isn't dropped.

• Oracle only compares one deep block (min_height - common_depth), never the tip — cluster/chaos/lib/cluster.py:797-825. Divergence first appears at the tip; the single post-heal oracle run every scenario fires evaluates a height below the fork point ⇒ identical pre-fork hash ⇒ PASS while forked at the tip. height_spread <= HEIGHT_DIFF_MAX(10) compares only heights, never hashes, so it doesn't compensate. Fix: also diff hash + state_root at the highest common height (min tip) and across a band of recent heights, in addition to the deep finality sample. (Note the --common-depth default is 2 at :1117, even shallower.)

• wait_oracle returns on the FIRST passing run — cluster/chaos/scenarios.sh:190-215 (used by every scenario via timed_wait_oracle). Combined with the tautology above, retry-until-pass actively selects the transient false-pass and records "recovered" for a fork that never reconciled. Fix: require N consecutive all-present passes; never count a run where a node was unreachable at common-height.

• receipt_checker passes on any 2 nodes, and auto-downgrades to 1 during partitions — cluster/chaos/lib/receipt_checker.py:197, 226. ok = len(ok_nodes) >= min_nodes with min_nodes defaulting to 2 and min_nodes = min(args.min_nodes, len(nodes)), so a 3rd forked/halted node is ignored exactly when the check should be strictest. Fix: require all selected nodes agree (len(failures)==0 and len(ok_nodes)==len(nodes)); if fewer than intended nodes are reachable ⇒ inconclusive/fail, never pass; record how many were actually compared.

• Local delay/loss/throttle apply netem to the default-route interface, not loopback — cluster/chaos/lib/net.sh:16-22,121-148 (detect_iface returns the default-route dev). On a 127.0.0.1 cluster all inter-node traffic goes over lo, so the qdisc lands on an interface carrying zero inter-node traffic — the fault injects nothing, yet the scenario passes while claiming to validate latency/loss. Fix: target lo via iptables -t mangle MARK on the listener ports + tc filter fw, or die on the local backend like partition-split already does.

• Local symmetric partition doesn't block the victim's outbound dials — cluster/chaos/chaos.sh:148-167. At 127.0.0.1 peer_hosts is always empty, so only the victim's own listener ports are dropped; its outbound TCP to peer listener ports survives, keeping a persistent bidirectional consensus connection alive → not a real cut (weaker than partition-asym out). This surfaces as misleading "victim advanced while partitioned" reports. Fix: make symmetric partition the union of asym-in + asym-out, or gate it to the docker backend.

• In-flight tx at workload shutdown → downgraded to inconclusive/pass, never checked on-chain — cluster/chaos/lib/tx_workload.py:200, 303-313; receipt_checker.py. On STOP, still-pending txs are written as tx_interrupted (a warning ⇒ inconclusive ⇒ overall pass by default) and are never verified against the chain. A tx a fault actually dropped is reported PASS. Fix: on STOP, grace re-poll, then verify each remaining tx_hash on all nodes (mined-but-forked vs truly absent).

• panic_log_scan gates on file mtime — cluster/chaos/lib/cluster.py:678; oracle.sh:10. A node that panicked and went silent has a frozen mtime (< oracle start) ⇒ its log is skipped ⇒ the crash signature the scan exists to catch is discarded for the dead node. Fix: don't gate crash detection on mtime; scan candidate log tails unconditionally; treat a stopped-growing log as suspect, not clean.

• (secondary) validator_stake_advancing increments on a node advancing its own (possibly forked/minority) chain, so it isn't a fork detector; and --skip-advancing disables the only halt detector — cluster/chaos/lib/cluster.py:768-795.

🟡 Coverage gaps (fault classes the oracle currently can't observe)

• No clock-skew / time-manipulation fault. Timestamp-gated hardfork-boundary behavior is out of the test envelope. Highest-value single addition: a clock-skew primitive (e.g. libfaketime preload in the node image) + a scenario that skews one node across a configured activation timestamp while the cluster produces blocks, then runs the state-root oracle.
• No hardfork-activation-under-chaos scenario (crossing a fork-activation timestamp while partitioned / under load / with a laggard). The strict "no init / no clean data dir" design principle also structurally prevents starting one node with a differing chain-spec.
• Transfer-only sentinel workload ⇒ the state-root oracle can't observe execution nondeterminism. Add contract creation / reverts / storage-contention under parallel workers, and EIP-7702 (type-4 / SetCode) transactions, so the oracle actually has divergence to catch.
• No laggard / slow-but-connected node (execution lag via CPU/IO throttle, not partition) with a bounded catch-up + post-catch-up state-root-equality assertion.

Suggested order (please don't reorder)

1. Fix the oracle (tautology guard + tip-band compare) and make receipt_checker require all-agree with no silent min_nodes downgrade. Acceptance test: a fixture that deliberately forks ONE node in a 4-node cluster and asserts oracle and receipt_checker report FAIL — this fixture is the gate for the whole fix.
2. Make faults verifiable — a post-injection effectiveness probe (partition ⇒ victim can't TCP-connect a peer listener; netem ⇒ measured RTT/loss delta); fix or die on the local no-op faults. Any round where the fault didn't measurably bite ⇒ inconclusive, never pass.
3. Harden recovery judgement — require stable/consecutive all-present passes; fix the mtime log gate.
4. Add clock-skew + fork-boundary scenario.
5. Enrich the workload (contracts + type-4).
6. Then laggard + continuous per-height state-root history.

Adding scenarios (4–6) before fixing the oracle (1) and fault verification (2) just produces more confident false PASSes.

cc @Lchangliang — thanks for building this; the skeleton is solid and the direction is right. These are about making a PASS trustworthy.

[Lchangliang]

Thanks for the detailed review. I went through the false-PASS items in order and addressed the P0/P1 harness issues in the current chaos branch.

Fixed so far:

• Oracle agreement is no longer a responder-subset tautology. hash / state_root checks now require every selected RPC node to respond at the target height, and failures report missing/responding nodes explicitly.
• Oracle now checks the deep common height, the min-tip height, and a recent tip band for both block hash and state root.
• Recovery judgement now requires consecutive all-present oracle passes instead of returning on the first transient pass.
• receipt_checker now requires all selected RPC nodes to agree. The old min_nodes downgrade is kept only as reported config metadata and no longer makes a PASS weaker.
• tx_interrupted / inconclusive histories are strict by default now, so interrupted in-flight txs do not silently pass unless explicitly relaxed.
• Panic/fatal scanning no longer skips candidate logs by file mtime; it scans log tails unconditionally to avoid missing panics appended to old log files.
• Local no-op network faults are gated: localhost multi-node delay/loss/netem/throttle refuses to run unless CHAOS_NET_IFACE=lo is explicit, and same-host local partition / outbound partition-asym refuses by default because it cannot precisely isolate one node's outbound traffic.
• Docker delay/loss/netem/throttle now verifies the injected tc qdisc after applying the fault; if the expected netem / tbf rule is absent, the injection fails instead of being recorded as a valid fault.
• The 4-validator + 2-VFN + 2-PFN Docker packet-loss-jitter case is covered as a sustained/random case and can include validators.

Validation run locally:

• python3 -m py_compile cluster/chaos/lib/cluster.py cluster/chaos/lib/receipt_checker.py cluster/chaos/lib/tx_workload.py
• bash -n cluster/chaos/scenarios.sh cluster/chaos/oracle.sh cluster/chaos/chaos.sh cluster/chaos/lib/net.sh cluster/chaos/lib/docker.sh cluster/chaos/loop.sh
• git diff --check
• Docker qdisc smoke: CHAOS_BACKEND=docker ... netem pfn1 1% 20ms 10ms, verified with net-status, then healed back to qdisc noqueue.
• Strict Docker oracle on the 8-node bridge cluster passed with the new common/tip/recent-band checks and skipped_old_files=0.
• Short Docker packet-loss-jitter regression passed: randomly injected loss+delay+jitter into node4,node3, fault-period oracle passed, heal passed, and recovery oracle passed after consecutive stable checks. Report: cluster/chaos/reports/chaos-20260728-170838.jsonl.

Still intentionally left as follow-up work:

• A deterministic acceptance fixture that deliberately forks one node and asserts oracle/receipt-checker FAIL.
• Stronger post-injection effectiveness probes beyond qdisc presence, e.g. TCP reachability / measured RTT-loss delta.
• Clock-skew / fork-boundary scenarios.
• Richer workload coverage: contracts, storage contention, revert paths, type-4/EIP-7702 style txs.

So the immediate false-PASS risks called out here are substantially reduced, but I agree the deliberate-fork fixture should be the next gate before calling the harness fully trustworthy.