Make cross-task promise sharing work instead of hanging

NEW-ARCH.md

@@ -0,0 +1,122 @@
+# All-quiescence scheduling: a simpler architecture, not (yet) adopted
+
+This document writes up a simplification of the scheduler that came out of
+fixing cross-task promise sharing (a task awaiting a promise settled by
+another task used to hang the simulation silently). It describes the
+current two-path architecture, the proposed single-path alternative, and
+the trade-offs that kept the alternative out of the codebase for now.
+
+## Background: what the bookkeeping can and cannot know
+
+The scheduler tracks each task in one of three states (`TaskInfo.resolve`):
+
+| State | Meaning | Nature |
+| --- | --- | --- |
+| a function | parked at a checkpoint/failpoint; the function is its continuation | **fact** — the scheduler holds the wake mechanism |
+| `false` | blocked (sleeping on a timer, waiting on a mutex/CV) | **fact** — a timer fire or a notify routes the wake back through the scheduler |
+| `undefined` | "running" | **prediction** — "this task will re-enter the scheduler at its next park or finish" |
+
+The first two states are exact because nothing outside the scheduler can
+invalidate them. The third is a bet, and JavaScript provides no hook at
+the moment a task awaits something, so the scheduler cannot observe
+suspension — it can only assume re-entry. The bet is wrong in exactly one
+situation: the task awaited a promise the scheduler doesn't manage
+(another task's promise, a bare deferred). And the wrongness is
+observable at exactly one kind of moment: *quiescence*, when the
+microtask queue has drained and the predicted re-entry has demonstrably
+not happened.
+
+## The current architecture: two paths
+
+1. **Synchronous path** (`unlockIfNecessary` → `scheduleNext`): runs
+   inside park/finish calls. If any task is marked running, it does
+   nothing (someone will re-enter). Otherwise every state is a fact, and
+   it is safe to make scheduling decisions immediately and to fire
+   several timers in a row until a task becomes schedulable.
+2. **Quiescence probe** (`armQuiescenceProbe` → `onQuiescence`): armed
+   whenever the scheduler hands control to user code; fires as a
+   macrotask, i.e. strictly after the whole microtask queue has drained.
+   A task still marked running at that point is provably suspended on an
+   unmanaged promise, and the probe treats it as blocked. Unlike the
+   synchronous path it fires at most **one** timer per probe and then
+   re-arms: a fired timer may settle promises via deadline abort
+   listeners, and those wake-ups sit in the microtask queue until the
+   probe returns — firing further timers would advance time past a
+   wake-up already in flight, or misreport a deadlock. Errors raised in a
+   probe have no task stack, so they fail `runTasks` through an
+   out-of-band rejection channel (`Promise.race`).
+
+One-line summary: **bookkeeping where it's fact, observation where it's
+prediction.**
+
+## The proposed simplification: quiescence as the only scheduling point
+
+Delete the synchronous path. Park calls only register their continuation
+and arm the probe; *every* scheduling decision — picking a parked task,
+firing a timer, declaring deadlock — happens in a probe, at quiescence.
+
+What this removes:
+
+- `unlockIfNecessary` and its someone-is-running early return. The
+  "running" state stops mattering entirely: at quiescence nobody is
+  running, so the scheduler never needs to ask.
+- The dual firing rule. Fire-one-then-yield becomes the only rule, and
+  it is the safer one everywhere.
+- `unlockIfNecessaryAfterPark` and its rollback contract. It exists only
+  because the synchronous path can throw *through* a parking task's
+  stack, which corrupts the park slot unless carefully rolled back. With
+  no synchronous scheduling, park calls that don't themselves consume
+  entropy cannot throw, and the subtlest exception-path code in the
+  scheduler disappears. (`sleep`'s rollback survives in reduced form:
+  timer registration can still raise a trace-divergence error in task
+  context.)
+- The two-channel error story. All scheduling errors go through the
+  out-of-band channel; only task-attributable errors (failpoint draws,
+  `task.random`, timer-creation divergence) still throw in task context.
+
+Scheduling decisions would depend on the same scheduler states in the
+same order — a decision made "immediately when the last task parks"
+observes the same candidates and draws the same entropy as the same
+decision deferred to the quiescence that immediately follows — so
+recorded traces are expected to replay unchanged for closed-world
+workloads. This must be verified (golden traces recorded on the current
+scheduler, replayed on the new one) before adopting.
+
+## Why it hasn't been adopted
+
+1. **It changes the error-injection contract.** Today an entropy source
+   that throws from a scheduling pick (a DST resource guard tripping)
+   throws synchronously into whichever task happened to park last, and
+   that task can catch it and recover — documented, tested behavior. In
+   the all-quiescence design the pick happens in a macrotask, so every
+   scheduling error fails the run out-of-band; no task can intercept it.
+   Arguably cleaner (the current attribution — "whoever parked last eats
+   the throw" — is accidental), but it is a deliberate breaking change,
+   not a refactor, and it deletes a feature: recoverable entropy-guard
+   trips.
+2. **Every scheduling step costs a macrotask hop.** The current
+   synchronous path schedules thousands of steps per run purely in
+   microtasks. Deferring every decision to `setImmediate` adds a real
+   constant factor to step-dense simulations, and DST workloads run many
+   iterations.
+3. **The win is smaller than it looks.** The probe machinery (arming,
+   run tokens, the out-of-band channel) is needed in both designs; the
+   synchronous path that would be deleted is the simple, well-tested
+   part. The genuinely subtle deletion — the park rollback contract —
+   only shrinks, because timer-registration divergence still throws in
+   task context.
+
+## When to adopt it
+
+In a major version, if/when breaking the "tasks can catch scheduler
+throws from park calls" contract is acceptable. The migration should:
+
+- move all scheduling-pick entropy draws and deadlock/budget errors to
+  the out-of-band channel, keeping task-context throws only for
+  failpoint draws, `task.random`, and timer-creation divergence;
+- verify trace compatibility with golden traces recorded on the old
+  scheduler (same scenario, replay must fully consume);
+- benchmark step-dense simulations to size the macrotask-hop cost;
+- re-run the existing suite expecting failures *only* in tests that
+  assert the sync error-injection contract, and rewrite those as
+  out-of-band assertions.

README.md

@@ -75,6 +75,10 @@ Returns `Result<T[], Error>` — either the array of results (in spec order) or
 
 **Scheduling algorithm**: When all running tasks have reached a checkpoint or blockpoint, the scheduler picks one of the checkpointed tasks using `sample()` (entropy-driven). Blocked tasks are excluded. If no tasks are checkpointed and all are blocked, a deadlock error is raised.
 
+**Cross-task promises**: A task may await a promise that another task will settle — the singleflight/coalescing shape, or a `Promise.all` spanning tasks. The scheduler's task states are facts except one: "parked" and "blocked" are exact because the scheduler itself holds the wake mechanism, but "running" is a *prediction* — "this task will re-enter the scheduler at its next park or finish" — and awaiting a promise the scheduler doesn't manage breaks exactly that prediction. So whenever the scheduler hands control to user code, it arms a one-shot *quiescence probe* (a macrotask, which runs only once the entire microtask queue has drained). A task still marked running when the probe fires is provably suspended on an unmanaged promise, and the probe treats it as blocked: parked tasks keep running, pending timers fire one at a time — yielding after each fire so that settling cascades (e.g. a deadline abort listener resolving a deferred) drain before the next decision — and virtual time advances until the awaited promise settles. If nothing can progress — a task awaits a promise nobody will ever settle and no timers are pending — the run fails with a deadlock report naming the task as awaiting a promise not managed by the simulation, instead of hanging silently.
+
+Two consequences to be aware of: tasks resumed by a promise settling run in native promise-reaction order, not entropy order — deterministic and replayable, but that resumption order is not part of the explored schedule space; and the quiescence inference assumes a closed world — a task awaiting real async work (I/O, real timers) looks identical to one awaiting an unmanaged promise, so the scheduler may advance virtual time while that work is still in flight. Real async work inside simulation tasks is outside the library's contract regardless, since it breaks determinism on its own.
+
 #### Deterministic time
 
 The simulation has a virtual monotonic clock, starting at 0 per run. Time never passes for real: `task.sleep(3_600_000, "one hour")` completes instantly in real time.
@@ -381,9 +385,10 @@ npm run typecheck
 
 See [TIMERS-SPEC.md](./TIMERS-SPEC.md) for the full requirements and semantics of deterministic time.
 
-- All concurrency is cooperative, not preemptive. Tasks only yield control at explicit `checkpoint`, `failpoint`, `blockpoint`, or `sleep` calls.
+- All concurrency is cooperative, not preemptive. Tasks only yield control at explicit `checkpoint`, `failpoint`, `blockpoint`, or `sleep` calls — or by awaiting a promise another task settles (see "Cross-task promises" above).
 - A task is a single sequential coroutine: never race two parking operations (checkpoints, failpoints, sleeps, mutex/condition-variable waits) within one task via `Promise.race`/`Promise.all`. The scheduler models exactly one park per task. Concurrency is expressed with multiple tasks; bounding work with a deadline is expressed with `withTimedSignal` and cooperative cancellation.
 - Timer durations are lower bounds, and firing order is entropy-controlled — never depend on the relative firing order of pending timers.
-- The simulation runs in a single JS event loop turn between scheduling decisions. There is no actual parallelism.
+- The simulation makes scheduling decisions synchronously (in microtasks) while its bookkeeping is provably exact, and defers to a macrotask-based quiescence probe only when a task might be suspended on an unmanaged promise. There is no actual parallelism. [NEW-ARCH.md](./NEW-ARCH.md) writes up a simpler all-quiescence alternative and why it hasn't been adopted.
+- Scheduling errors (a deadlock, an exhausted step budget, an entropy source that throws from a scheduling pick) normally propagate synchronously into the task whose park call triggered the decision — which is what lets a task catch and recover from a transient entropy-guard trip. Errors raised from a quiescence probe have no task stack, so they fail `runTasks` directly through an out-of-band channel.
 - `SimulationImpl` should be treated as single-use per `runTasks` call. After a failed run (error or deadlock), the instance is permanently poisoned (`abortedWithError` is never reset) and subsequent `runTasks` calls will immediately fail.
 - The `sample()` function's "no entropy for single item" optimization is critical for replay correctness — it ensures the entropy consumption sequence doesn't depend on transient pool sizes.

package.json

@@ -1,6 +1,6 @@
 {
     "name": "determined",
-    "version": "0.4.0",
+    "version": "0.4.1",
     "type": "module",
     "license": "MIT",
     "repository": {