Repository navigation
fix(drivers): a hung legacy poll blocks autonomous default mode #1132
Description
A failed or stale driver must receive driver_default_mode. On bundled and other non-control-v2 drivers, a stuck driver_poll can prevent that.
Found while reviewing safety invariants on master 6eadec3f (after v3.1.3-beta.1).
What happens
Each driver actor handles poll and default on the same loop (go/internal/drivers/registry.go). Poll holds LuaDriver.mu for the whole Lua call (go/internal/drivers/lua.go). setLifecycleContext is a no-op unless the runtime policy is control v2, so a legacy driver_poll has no VM deadline.
SendDefault cancels an in-flight command and queues default on defaultCh. It cannot preempt a poll that is already inside http_get (15s client timeout) or a multi-register Modbus sequence (5s per request). sendDriverDefault in go/cmd/ftw/main.go waits only 2s, logs the timeout, and continues. The last setpoint can stay on the inverter.
Control v2 is better: poll gets a 10s Lua context. Preemption tests in registry_runtime_preemption_test.go cover a wedged command, not a wedged poll.
Reproduction sketch
- Load a legacy Lua driver whose
driver_pollblocks inhttp_getorhost.sleeplonger than the watchdog. - Stop fresh telemetry so the watchdog marks it stale.
- Observe
sendDriverDefaultreturn on the 2s deadline whilePollis still running. - Confirm no
driver_default_modereaches the device until poll returns.
What a fix needs
- Give every poll a cancellable Lua context, not only control v2.
- Make
SendDefaultcancel that poll context the same way it cancels a command. - Tests: HTTP or Modbus stub that blocks longer than
driverDefaultTimeout; assert default still runs (or poll is cancelled) and the device is not left on the last setpoint. - Keep the existing wedged-command tests.
Do not treat a successful poll with no host.emit as watchdog success; that split already exists after the Ferroamp cache outage.
Open PR overlap: none of the current drafts are the right place for this. #1129 is DuckDB storage.
Activity
frahlg commentedon Sep 8, 2026
Extra detail from the same pass:
host.sleepalready selects onluaCallContext(L).Done()(go/internal/drivers/lua.go). Control v2 also rejectsms > 100.On a legacy poll
setLifecycleContextis a no-op, so that context isBackground.host.sleep(60000)then sleeps the full minute and holds both the actor loop andLuaDriver.mu.registry_runtime_preemption_test.goalready asserts that a context-canceled sleep returns early — that only helps once poll itself has a context.The 2s deadline on
sendDriverDefaultis on the waiter, not onmu.Lock()insidecall(). Even after poll returns, default's Lua context issetLuaCallContext(ctx, 5s)with a ctx that may already be expired.