fix(connectors): bound and renew ephemeral connector secrets by their TTL - #2110
fix(connectors): bound and renew ephemeral connector secrets by their TTL#2110yiboyasss wants to merge 3 commits into
Conversation
The process-local per-turn secret store had no expiry: an entry whose turn never reached a terminal settlement (lease lost, DB pool exhaustion, unhealthy heartbeat at shutdown) stayed referenced forever. Record a store timestamp and reap entries older than the interaction TTL on each new store, so the leak is bounded instead of permanent.
…tcome Pop from inside the settlement that decides the status (finish_turn, settle_task_lease_isolated, _finalize_resumed_task) rather than from a later separate read, and only for COMPLETED/FAILED. A WAITING_FOR_USER or PAUSED outcome is the same turn resuming later under the same turn_id, so it keeps its values; a resume reads that id back off the cached tool config via the new get_connector_runtime_turn_id.
…ause The TTL only took effect via the opportunistic prune in store_ephemeral_runtime_values, so a quiet process (no turn ever starts a new store after this one goes stale) kept serving an entry past its advertised lifetime - the bound was never actually observable from get/pop themselves. Check age on every read and pop. Bounding by time alone then created a new problem: a turn that pauses again under the same turn_id gets a fresh interaction lifetime, but its stored_at timestamp never moved, so a later unrelated store's prune could reap an active turn's still-needed secrets out from under it. Renew the timestamp from finish_turn's and _finalize_resumed_task's non-terminal (PAUSED/WAITING_FOR_USER) branches, the same places that already decide not to pop.
There was a problem hiding this comment.
Code Review
This pull request introduces a TTL-based expiration and renewal mechanism for ephemeral per-turn connector secrets to prevent memory leaks in deferred settlement paths. It updates connector_runtime.py to track storage timestamps, enforce TTL on reads/writes, and allow TTL renewal for paused turns. Additionally, it integrates turn_id propagation across task lease settlement and finalization paths in websocket.py and task_orchestrator.py. A review comment correctly identifies a critical issue in websocket.py where an undefined task_id variable is referenced in exception handlers, which would raise a NameError and mask the original exception.
| if turn_id is not None: | ||
| if final_task_status in TERMINAL_TASK_STATUSES: | ||
| try: | ||
| from ..services.connector_runtime import ( | ||
| pop_ephemeral_runtime_values, | ||
| ) | ||
|
|
||
| pop_ephemeral_runtime_values(turn_id) | ||
| except Exception: | ||
| logger.warning( | ||
| "connector runtime cleanup failed for task %s turn %s", | ||
| task_id, | ||
| turn_id, | ||
| exc_info=True, | ||
| ) | ||
| else: | ||
| # WAITING_FOR_USER/PAUSED: the same turn resuming again later | ||
| # under this same turn_id, with a fresh interaction lifetime - | ||
| # keep whatever ephemeral secrets it may still need from | ||
| # expiring on the original pause's clock (see | ||
| # connector_runtime.renew_ephemeral_runtime_values). | ||
| try: | ||
| from ..services.connector_runtime import ( | ||
| renew_ephemeral_runtime_values, | ||
| ) | ||
|
|
||
| renew_ephemeral_runtime_values(turn_id) | ||
| except Exception: | ||
| logger.warning( | ||
| "connector runtime secret renewal failed for task %s turn %s", | ||
| task_id, | ||
| turn_id, | ||
| exc_info=True, | ||
| ) |
There was a problem hiding this comment.
The variable task_id is not defined in the scope of _finalize_resumed_task. This function receives task_lease: TaskLease as a parameter, but does not define task_id locally. Referencing task_id in the exception handlers will raise a NameError, which would mask the original exception and make debugging difficult. Please use task_lease.task_id instead.
if turn_id is not None:
if final_task_status in TERMINAL_TASK_STATUSES:
try:
from ..services.connector_runtime import (
pop_ephemeral_runtime_values,
)
pop_ephemeral_runtime_values(turn_id)
except Exception:
logger.warning(
"connector runtime cleanup failed for task %s turn %s",
task_lease.task_id,
turn_id,
exc_info=True,
)
else:
# WAITING_FOR_USER/PAUSED: the same turn resuming again later
# under this same turn_id, with a fresh interaction lifetime -
# keep whatever ephemeral secrets it may still need from
# expiring on the original pause's clock (see
# connector_runtime.renew_ephemeral_runtime_values).
try:
from ..services.connector_runtime import (
renew_ephemeral_runtime_values,
)
renew_ephemeral_runtime_values(turn_id)
except Exception:
logger.warning(
"connector runtime secret renewal failed for task %s turn %s",
task_lease.task_id,
turn_id,
exc_info=True,
)
Summary
Part of splitting #1967 into smaller, independently-reviewable pieces (secret lifecycle → identity contract → eligibility gate → frontend answer contract → runtime enable). This is the first slice: the process-local per-turn ephemeral connector secret store's TTL.
_EPHEMERAL_RUNTIME_STORED_AT+ TTL (referencing_MAX_INTERACTION_TTL_SECONDS) with an opportunistic reaper, plus a terminal-status-gated atomic pop fromfinish_turn/settle_task_lease_isolated/_finalize_resumed_task— a turn's secrets are freed the moment its outcome is decided as COMPLETED/FAILED, not on a later separate read.turn_id(WAITING_FOR_USER/PAUSED) renews the TTL instead of leaving the original store's timestamp in place, so a later unrelated store's prune can't reap an active turn's still-needed secrets.No behavior change for a turn that reaches a terminal outcome normally; this only bounds/protects the deferred-to-TTL-recovery paths (lease lost, DB pool exhaustion, unhealthy heartbeat at shutdown) that can never safely pop or renew themselves.
Test plan
tests/web/services/test_connector_runtime_ephemeral.py)finish_turnandexecute_resume_background/_finalize_resumed_taskconfirming a WAITING_FOR_USER outcome renews rather than popstests/web+tests/core/tools/adapters/vibesuite green; only pre-existing environment failures (missinglibcairo, docker-dependent sandbox tests) and 2 order-dependent flakes untouched by this diff