Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ dependencies = [
"deadline-job-attachments == 0.1.3",
# Pinned to patch version due to Host Config Script runner usage of private OpenJD Sessions API.
"openjd-sessions == 0.10.14",
"openjd-model >= 0.11.3, < 0.12",
"openjd-model >= 0.11.4, < 0.12",
# tomli became tomllib in standard library in Python 3.11
"tomli == 2.0.* ; python_version<'3.11'",
"tomlkit >= 0.13,< 0.16",
Expand Down
5 changes: 5 additions & 0 deletions src/deadline_worker_agent/api_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,9 @@ class StepDetailsData(StepDetailsIdentifierFields):
dependencies: NotRequired[list[str]]
"""A list of step identifiers that this step depends on"""

resolvedSymbolTable: NotRequired[str]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New wire field, but session_events.py is untouched — worth checking whether that is the intended outcome for the agent log.

LOGGING_ALLOW_LIST["deadline:BatchGetJobEntity"]["res_log_body"] (session_events.py:161) enumerates the loggable stepDetails / environmentDetails keys, and _get_loggable_parameters (session_events.py:206) replaces any key not in that map with *REDACTED*. Since resolvedSymbolTable was not added there, it currently logs as resolvedSymbolTable: "*REDACTED*" — which is the safe default, and correct given the table holds resolved job/task parameter values (the same reasoning the existing comments give for excluding template and jobDetails.parameters).

So no leak today. The ask is just to make it deliberate rather than incidental: adding an explicit "resolvedSymbolTable": False, # resolved parameter values; sensitive entry alongside the existing exclusions documents the decision and prevents a future "*": True-style change or allow-list refactor from silently opting this field in.

"""Pre-resolved symbol table as a JSON string, forwarded to the Rust session runtime."""

extensions: NotRequired[list[str]]
"""The extensions enabled for the job, as supplied by the service"""

Expand Down Expand Up @@ -401,6 +404,8 @@ class EnvironmentDetailsData(EnvironmentDetailsIdentifierFields):
"""The Open Job Description schema version"""
template: dict[str, Any]
"""The template of the environment."""
resolvedSymbolTable: NotRequired[str]
"""Pre-resolved symbol table as a JSON string, forwarded to the Rust session runtime."""
extensions: NotRequired[list[str]]
"""The extensions enabled for the job, as supplied by the service"""

Expand Down
3 changes: 3 additions & 0 deletions src/deadline_worker_agent/scheduler/scheduler.py
Original file line number Diff line number Diff line change
Expand Up @@ -1227,6 +1227,8 @@ def _create_new_sessions(
region=self._boto_session.region_name,
)

resolved_symbol_table_json = queue.peek_resolved_symbol_table_json()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This peek performs a blocking BatchGetJobEntity service call on the scheduler thread, and that is a new behaviour for this code path — worth weighing before merge.

The docstring on the new accessor says "Entity resolution results are cached by JobEntities, so the later dequeue issues no additional service request." That is true of the second call, but the first one is this one, and at this point in _create_new_sessions the cache is empty for the entity in question. JobEntities.request() (job_entities.py:124-146) finds no entity_record.data, calls cache_entities([identifier]) synchronously, and blocks on the API round trip with retries. So the cost is not moved — it is moved earlier, onto a thread that previously never did entity resolution.

Two consequences:

  1. The warming step is bypassed. Entity resolution deliberately happens on the session thread: Session.run() calls _warm_job_entities_cache() first (session.py:319-364), which batches every action identifier into one request and swallows failures with an INFO log. That design keeps per-entity latency off the scheduler. This peek resolves one entity ahead of it, on the wrong thread, in its own single-identifier request.

  2. It is inside the for session_spec in ... loop. With N new sessions assigned in one UpdateWorkerSchedule response, this serialises N blocking calls before any session starts. The scheduler thread is what heartbeats — _sync is called from the run() loop at scheduler.py:315 and drives updated_session_actions/cancellation/shutdown. A slow or retrying BatchGetJobEntity here delays the next heartbeat for all sessions, not just the new one.

Since JobEntities is constructed a few lines above (scheduler.py:927) and is passed to both the queue and the session, one alternative that keeps the current thread model: have the adapter pull the table lazily at first use on the session thread, rather than the scheduler pre-resolving it at construction. If the value really must be known at Session.__init__ time (it must, for job_name= on the v0 session — see python.py:66), it is worth saying so here in a comment, and bounding the cost explicitly, because the blocking call is not obvious at this call site.


try:
session = Session(
id=new_session_id,
Expand All @@ -1244,6 +1246,7 @@ def _create_new_sessions(
session_root_dir=self._session_root_dir,
farm_id=self._farm_id,
region=self._boto_session.region_name,
resolved_symbol_table_json=resolved_symbol_table_json,
)
except (ValueError, NotImplementedError, OSError) as e:
# Runtime construction can fail per-session (e.g. the selected runtime's
Expand Down
49 changes: 49 additions & 0 deletions src/deadline_worker_agent/scheduler/session_queue.py
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,7 @@ def dequeue(self) -> SessionActionDefinition | None:
next_action = ExitEnvironmentAction(
id=action_id,
environment_id=environment_id,
details=environment_details,
)
else:
raise ValueError(f'Unknown action type "{action_type}".')
Expand Down Expand Up @@ -557,3 +558,51 @@ def dequeue(self) -> SessionActionDefinition | None:
f'Unknown action type "{action_type}". Complete action = {action_definition}'
)
return next_action

def peek_resolved_symbol_table_json(self) -> str | None:
"""Inspect the first queued action's resolved symbol table without consuming it.

This accessor is non-consuming: the queue state is not mutated, and a
subsequent ``dequeue`` call will still yield the same front action.

Entity resolution results are cached by ``JobEntities``, so the later

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things about this docstring claim are worth correcting, because both invite a wrong conclusion at the call site.

"Entity resolution results are cached by JobEntities, so the later dequeue issues no additional service request." This holds only when the peek succeeds. On failure it is the opposite of true. cache_entities creates an EntityRecord up front (_create_entity_records, job_entities.py:209-215) and only assigns .data on success; on the DeadlineRequestWorkerNotFound / DeadlineRequestUnrecoverableError path it continues (job_entities.py:231-235), leaving the record with data is None and error is None. request() then falls through both checks and raises the "Should be impossible" RuntimeError (job_entities.py:157-160). That is swallowed by the new except Exception below — so the peek returns None, and the later dequeue re-enters request(), finds data is None, and calls cache_entities again. The failing case therefore costs two blocking service calls, not one, both on the scheduler thread.

"a failure must not break session creation. The subsequent dequeue surfaces the real error through the normal action-failure path." The second sentence is only true for the terminal-error case. If the peek failed for a transient reason and the dequeue retry succeeds, no error is ever surfaced — the session runs to completion with job_name=None, and the only trace is this logger.warning. That is the intended graceful degradation, but the comment describes it as though the error always resurfaces, which would lead a future reader to assume silent-None sessions cannot happen.

Also, the warning drops the exception entirely. Given that this swallows except Exception — including UnsupportedSchema and validation ValueErrors from validate_entity_data, which are permanent and will fail the action moments later — recording at least type(e).__name__ would make the two situations distinguishable in logs without echoing free text.

``dequeue`` issues no additional service request for the same entity.

Returns
-------
str | None
The ``resolved_symbol_table_json`` from the first action's entity,
or None when the queue is empty or the action type has no table.
"""
if not self._actions:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Peeking only self._actions[0] makes the session-scoped job_name depend on which action the service happens to put first, and there is a common case where it silently comes back None.

The else: return None branch covers SYNC_INPUT_JOB_ATTACHMENTS, and that action type is routinely the first action in a session — it is the input-sync that precedes task runs, and list_all_action_identifiers (line 167) treats it as a first-class queue member. So for any queue whose front action is an attachment download/upload, peek returns None, and PythonSessionRuntime is constructed with job_name=None for the entire session — including all the later TASK_RUN actions whose stepDetails did carry a resolvedSymbolTable. The value is not merely late; it is never picked up, because _config.resolved_symbol_table_json is consumed once at construction (python.py:66) and Session.__init__ runs before any action executes.

The result is a feature that works or does not work based on action ordering, with no signal either way: the logger.warning in the except block does not fire, since this is the non-exceptional else path.

Job.Name is job-scoped, so any queued action that has an associated entity can supply it. Two options that remove the ordering dependence:

  • Scan for the first action in self._actions that has a table rather than inspecting only index 0 (still one entity resolution in the common case, since ENV_ENTER/TASK_RUN are usually at the front).
  • Or source it from jobDetails instead, which the scheduler has already resolved at this point (job_details at scheduler.py:940) and which is unambiguously job-scoped — no peeking, no extra request, no ordering dependence.

At minimum, worth a logger.debug on the else branch so "no table because the front action was an attachment sync" is distinguishable in logs from "the service sent no table."

return None

action_queue_entry = self._actions[0]
action_type = action_queue_entry.definition["actionType"]

try:
if action_type.startswith("ENV_"):
action_queue_entry = cast(EnvironmentQueueEntry, action_queue_entry)
environment_id = action_queue_entry.definition["environmentId"]
environment_details = self._job_entities.environment_details(
environment_id=environment_id
)
return environment_details.resolved_symbol_table_json
elif action_type == "TASK_RUN":
action_queue_entry = cast(TaskRunQueueEntry, action_queue_entry)
step_id = action_queue_entry.definition["stepId"]
step_details = self._job_entities.step_details(step_id=step_id)
return step_details.resolved_symbol_table_json
else:
return None
except Exception:
# This accessor only seeds session-scoped symbols (e.g. Job.Name),
# so a failure must not break session creation. The subsequent
# dequeue surfaces the real error through the normal action-failure
# path.
logger.warning(
"Failed to prefetch resolved symbol table for the first queued action "
"(type=%s); proceeding without it.",
action_type,
)
return None
1 change: 1 addition & 0 deletions src/deadline_worker_agent/sessions/actions/enter_env.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,4 +109,5 @@ def start(
job_env_id=self._job_env_id,
environment=self._details.environment,
os_env_vars={"DEADLINE_SESSIONACTION_ID": self._id},
resolved_symbol_table_json=self._details.resolved_symbol_table_json,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EnvironmentDetails now carries resolved_symbol_table_json, and it is forwarded on env enter — but ExitEnvironmentAction (actions/exit_env.py:60) and Session.exit_environment are untouched, so the environment’s onExit script resolves its expressions without the pre-resolved table. If the Rust session does not persist the symtab from enter_environment for the lifetime of that environment, onEnter and onExit for the same environment will resolve references from different contexts — an asymmetry that will be hard to debug.

Worth confirming which is true: if the session caches it, a brief comment saying so would prevent a future reader from “fixing” this; if it does not, exit_environment needs the same threading.

)
13 changes: 12 additions & 1 deletion src/deadline_worker_agent/sessions/actions/exit_env.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from .openjd_action import OpenjdAction

if TYPE_CHECKING:
from ..job_entities import EnvironmentDetails
from ..session import Session


Expand All @@ -20,26 +21,32 @@ class ExitEnvironmentAction(OpenjdAction):
A unique identifier for the session action
environment_id : str
The job environment identifier
details : EnvironmentDetails | None
Optional environment details carrying the pre-resolved symbol table
"""

_environment_id: str
_details: EnvironmentDetails | None

def __init__(
self,
*,
id: str,
environment_id: str,
details: EnvironmentDetails | None = None,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making details optional here is worth a second look, because the only production caller always has the value and the default silently disables the feature.

There is exactly one non-test construction site — session_queue.py:452 — and by that point environment_details has already been resolved (it is fetched a few lines above for both ENV_ENTER and ENV_EXIT, and EnterEnvironmentAction takes it as a required kwarg). So the = None default is not serving any real caller; it exists only for the tests in test_exit_env.py that construct the action bare.

The cost of the default is that the failure is silent. Forget the kwarg at some future call site and exit_environment receives resolved_symbol_table_json=None, the onExit script resolves against a different symbol context than onEnter did, and nothing logs or raises. Contrast EnterEnvironmentAction, where omitting details is a TypeError at construction.

Given EnterEnvironmentAction._details is required and non-optional, making this one required too (details: EnvironmentDetails) would keep the two symmetric, let start() drop the if self._details is not None else None ternary, and turn a silent degradation into an immediate error. The three tests that build the action without details would each need the kwarg added — a small change that also makes them exercise the shape production actually uses.

If the optionality is deliberate — e.g. anticipating a future caller that genuinely has no details — a comment naming that caller would help, since today the signature suggests a case that does not exist.

) -> None:
super(ExitEnvironmentAction, self).__init__(
id=id, action_log_kind=SessionActionLogKind.ENV_EXIT
)
self._environment_id = environment_id
self._details = details

def __eq__(self, other: Any) -> bool:
return (
type(self) is type(other)
and self._id == other._id
and self._environment_id == other._environment_id
and self._details == other._details
)

def start(
Expand All @@ -58,5 +65,9 @@ def start(
An executor for running futures
"""
session.exit_environment(
job_env_id=self._environment_id, os_env_vars={"DEADLINE_SESSIONACTION_ID": self._id}
job_env_id=self._environment_id,
os_env_vars={"DEADLINE_SESSIONACTION_ID": self._id},
resolved_symbol_table_json=self._details.resolved_symbol_table_json
if self._details is not None
else None,
)
Original file line number Diff line number Diff line change
Expand Up @@ -85,4 +85,5 @@ def start(self, *, session: Session, executor: Executor) -> None:
task_parameter_values=self._task_parameter_values,
os_env_vars=env_vars,
step_name=self._details.step_template.name,
resolved_symbol_table_json=self._details.resolved_symbol_table_json,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scope mismatch worth confirming: the resolved table arrives on stepDetails, but it is consumed once per task.

JobEntities caches the raw stepDetails payload in _entity_record_map keyed by step id (job_entities.py:132), and step_details() rebuilds StepDetails from that same cached dict on every call (job_entities.py:372). So all tasks in a step — including tasks dispatched in later UpdateWorkerSchedule cycles within the same session — share one resolved_symbol_table_json, sourced from whenever the step was first fetched.

That is fine if the service only ever puts step-and-above-scoped symbols in it (Job.*, Param.*). It is a correctness bug if the table can contain task-scoped entries such as Task.Param.<name>, since task_parameter_values on line 86 varies per task while the table beside it would not — task 2 would resolve {{Task.Param.Frame}} to task 1's value, silently, and the two sources would disagree.

Worth confirming which contract the service guarantees. If task-scoped symbols are excluded by design, a short comment here saying the table is step-scoped would protect the invariant; if they are not, the table needs to be per-task rather than carried on the cached step entity.

)
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ class EnvironmentDetails:
environment: EnvironmentModel
"""The environment"""

resolved_symbol_table_json: str | None = None
"""Pre-resolved symbol table JSON from the service, forwarded to the Rust session runtime."""

@classmethod
def from_boto(cls, environment_details_data: EnvironmentDetailsData) -> EnvironmentDetails:
"""Converts an environmentDetails entity received from BatchGetJobEntity API response into
Expand Down Expand Up @@ -59,7 +62,10 @@ def from_boto(cls, environment_details_data: EnvironmentDetailsData) -> Environm
else:
raise UnsupportedSchema(schema_version.value)

return EnvironmentDetails(environment=environment)
return EnvironmentDetails(
environment=environment,
resolved_symbol_table_json=environment_details_data.get("resolvedSymbolTable", None),
)

@classmethod
def validate_entity_data(cls, entity_data: dict[str, Any]) -> EnvironmentDetailsData:
Expand Down Expand Up @@ -90,6 +96,7 @@ def validate_entity_data(cls, entity_data: dict[str, Any]) -> EnvironmentDetails
Field(key="environmentId", expected_type=str, required=True),
Field(key="jobId", expected_type=str, required=True),
Field(key="schemaVersion", expected_type=str, required=True),
Field(key="resolvedSymbolTable", expected_type=str, required=False),
Field(key="extensions", expected_type=list, required=False),
),
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ class StepDetails:
dependencies: list[str] = field(default_factory=list)
"""The dependencies (a list of IDs) that the step depends on"""

resolved_symbol_table_json: str | None = None
"""Pre-resolved symbol table JSON from the service, forwarded to the Rust session runtime."""

@classmethod
def from_boto(cls, step_details_data: StepDetailsData) -> StepDetails:
"""Converts an stepDetails entity received from BatchGetJobEntity API response into a
Expand Down Expand Up @@ -86,6 +89,7 @@ def from_boto(cls, step_details_data: StepDetailsData) -> StepDetails:
step_template=step_template,
step_id=step_details_data["stepId"],
dependencies=step_details_data["dependencies"],
resolved_symbol_table_json=step_details_data.get("resolvedSymbolTable", None),
)

@classmethod
Expand Down Expand Up @@ -118,6 +122,7 @@ def validate_entity_data(cls, entity_data: dict[str, Any]) -> StepDetailsData:
Field(key="template", expected_type=dict, required=True),
Field(key="stepId", expected_type=str, required=True),
Field(key="dependencies", expected_type=list, required=False),
Field(key="resolvedSymbolTable", expected_type=str, required=False),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Field line is the highest-blast-radius change in the PR, and nothing tests it.

validate_object rejects unknown keys wholesale (validation.py:42-47): any key not in the declared set raises ValueError: Unexpected fields: .... So before this line existed, the mere presence of resolvedSymbolTable in a stepDetails response would fail validation and take down every step in the job — not degrade, fail. This line is what makes the whole feature safe to roll out, and it is load-bearing in the deployment direction that actually happens first: the service starts sending the field to workers already in the field.

The gap is that the test_input_validation_success parametrization right above (test_step_details.py:14-63) was not extended — it still covers only dependencies and extensions permutations. The two new tests added in this PR both exercise from_boto, not validate_entity_data, and from_boto uses a plain .get() that never consults the field list. So if this line were dropped or its key misspelled, every new test in this PR still passes while the agent hard-rejects every step entity from a field-sending service.

Two cheap parametrize cases would close it, in both this file and environment_details.py:

  • a valid payload including "resolvedSymbolTable": "[]" in test_input_validation_success
  • "resolvedSymbolTable": 123 (or a dict) in test_input_validation_failure, pinning the expected_type=str half

The second is worth having on its own: expected_type=str is the only guard rejecting a non-string here, and a dict payload would otherwise flow into _parse_resolved_symtab and hit from_json_str with a non-str.

Field(key="extensions", expected_type=list, required=False),
),
)
Expand Down
3 changes: 3 additions & 0 deletions src/deadline_worker_agent/sessions/runtime/_abc.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ def enter_environment(
environment: EnvironmentModel,
identifier: Optional[EnvironmentIdentifier] = None,
os_env_vars: Optional[dict[str, str]] = None,
resolved_symbol_table_json: str | None = None,
) -> EnvironmentIdentifier:
"""Enter an environment; returns its identifier."""
...
Expand All @@ -68,6 +69,7 @@ def exit_environment(
identifier: EnvironmentIdentifier,
os_env_vars: Optional[dict[str, str]] = None,
keep_session_running: bool = False,
resolved_symbol_table_json: str | None = None,
) -> None:
"""Exit a previously entered environment."""
...
Expand All @@ -81,6 +83,7 @@ def run_task(
os_env_vars: Optional[dict[str, str]] = None,
log_task_banner: bool = True,
step_name: str | None = None,
resolved_symbol_table_json: str | None = None,
) -> None:
"""Run a task within the session's active environment(s)."""
...
Expand Down
6 changes: 6 additions & 0 deletions src/deadline_worker_agent/sessions/runtime/_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,9 @@ class SessionRuntimeConfig:
session_root_directory: Path
spec_revision: str = "2023-09"
supported_extensions: tuple[str, ...] = ()
resolved_symbol_table_json: str | None = None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RustSessionRuntime.__init__ never reads this field — I checked every reference, and the only consumer is PythonSessionRuntime (python.py:66). The Rust adapter gets its table per-action via the resolved_symbol_table_json kwarg on enter_environment / exit_environment / run_task instead.

That makes the config-level plumbing dead weight on the path the feature is actually built for. Because queue.peek_resolved_symbol_table_json() at scheduler.py:1230 is called unconditionally — before the Session constructor and with no reference to the runtime_kind resolved twenty lines earlier — every RUST session pays a blocking BatchGetJobEntity round trip on the scheduler thread for a value that is then threaded into SessionRuntimeConfig and dropped on the floor.

So the cost lands on precisely the sessions that do not benefit, while the sessions that do benefit (PYTHON) are the ones where the v0 runtime cannot use the table for anything except the Job.Name display string.

Cheapest fix is to gate the peek on the selected kind, since runtime_kind is already in scope at that point:

resolved_symbol_table_json = (
    queue.peek_resolved_symbol_table_json()
    if runtime_kind is SessionRuntimeKind.PYTHON
    else None
)

That removes the extra request from the Rust path entirely, and makes the narrow purpose of this field — which this docstring already describes accurately as "Adapters that seed session-scoped symbols at construction (e.g. job_name for the classic Python session)" — visible at the call site too.

"""Pre-resolved symbol table JSON observed on the session's first queued action.

Adapters that seed session-scoped symbols at construction (e.g. job_name for
the classic Python session) consume this at build time.
"""
35 changes: 35 additions & 0 deletions src/deadline_worker_agent/sessions/runtime/python.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@
from __future__ import annotations

from datetime import timedelta
from logging import getLogger
from pathlib import Path
from typing import TYPE_CHECKING, Any, Optional

from openjd.expr import SerializedSymbolTable

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This module-level from openjd.expr import SerializedSymbolTable couples the Python adapter importability to the Rust binding, which the surrounding code goes out of its way to treat as optional.

openjd.expr is the Rust extension namespace — rust.py:72 says so explicitly ("openjd.expr.PathFormat is a non-constructable Rust enum"), and rust.py imports it alongside openjd._openjd_rs. Meanwhile _factory.py imports each adapter lazily inside its own try/except ImportError, with the documented rationale: "the adapter module for kind cannot be imported (e.g. the Rust binding is unavailable on this platform)."

That guard exists for PythonSessionRuntime too. So on any host where the openjd._openjd_rs extension will not load — an unsupported arch/libc, a wheel that fell back to sdist, a stripped-down install — the effect of this line is:

  • before: RUST fails with NotImplementedError, PYTHON still works, and the no-hint default in select_runtime() (SessionRuntimeKind.PYTHON, _select.py:47) keeps the worker running.
  • after: both adapters raise NotImplementedError: PythonSessionRuntime adapter is not available, create_session_runtime fails for every kind, and the except (ValueError, NotImplementedError, OSError) in scheduler.py fails every action of every session. The worker stays up but runs nothing.

So the fallback path now inherits the exact failure mode it was structured to survive, and it does so to obtain one optional display string.

Worth confirming which is actually true here, since the fix differs:

  1. If openjd.sessions 0.10.14 already imports openjd.expr transitively (plausible given the "pythonexpr" hint naming in _select.py), then the binding is already a hard requirement of the v0 path and this line changes nothing — in which case a one-line note saying so would stop a future reader from worrying, and the _factory.py comment is the misleading part.
  2. If it does not, move the import inside _extract_job_name (or guard it), so a missing binding degrades to job_name=None instead of taking down the Python runtime.

Note the new tests cannot distinguish these: patch.object(python_module, "OpenJDSession", autospec=True) runs only after python.py has already imported successfully.

from openjd.model import RevisionExtensions, SpecificationRevision
from openjd.sessions import Session as OpenJDSession

Expand All @@ -22,6 +24,29 @@

__all__ = ["PythonSessionRuntime"]

logger = getLogger(__name__)


def _extract_job_name(json_str: str | None) -> str | None:
"""Extract the Job.Name value from a resolved symbol table JSON string.

Returns None when the input is None, the table lacks a Job.Name entry, the
entry's value is not a string, or parsing fails (graceful degradation —
mirrors _parse_resolved_symtab in the Rust adapter).
"""
if json_str is None:
return None
try:
symtab = SerializedSymbolTable.from_json_str(json_str).to_symtab()
entry = symtab.get("Job.Name")
if entry is None:
return None
value = entry.item()
return value if isinstance(value, str) else None
except Exception as e:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This %s-of-exception can leak resolved symbol-table contents into the agent log, and the exposure here is broader than the equivalent line in the Rust adapter.

The table holds resolved job/task parameter values — the same data session_events.py deliberately redacts from BatchGetJobEntity response logging, and the reason template / jobDetails.parameters are excluded from LOGGING_ALLOW_LIST. So it should not reappear verbatim in a warning.

What makes this line riskier than _parse_resolved_symtab in rust.py:249 is that it wraps two calls, not one:

  • from_json_str(json_str) — raises on malformed JSON; such errors conventionally quote the offending input fragment.
  • .to_symtab() — this is the content validation step, and its errors are about specific entries. An error naming the entry it rejected is very likely to carry that entry name and value, i.e. exactly a resolved parameter value.

Because to_symtab() is inside the try here, the higher-signal-for-leaking failure mode is the one this line reports. And the warning fires precisely when the payload is unexpected — the case whose contents you control least.

Suggest logging only type(e).__name__ (plus payload length or entry count if useful for diagnosis) rather than the exception text. The unit test at test_construction_passes_none_when_resolved_table_is_malformed_json only asserts warning was called once, so tightening the message will not break it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This except Exception is the wrong width for what the try actually calls, and the consequence is worse here than in the Rust adapter because there is no decorator behind it.

to_symtab() is the Rust content-validation step (the binding rejects invalid entries there, not in from_json_str). This repo already has a documented policy for Rust code that can fail abnormally — convert_runtime_crashes in _abc.py:30, whose own docstring exists because of "a Rust panic crossing the PyO3 boundary as a BaseException". Every openjd.sessions._v1 call in rust.py is wrapped in it, including the two _parse_resolved_symtab call sites.

This one is not, and it cannot be:

  • _extract_job_name runs from PythonSessionRuntime.__init__ (line 66), before any adapter method exists to decorate.
  • A BaseException from to_symtab() therefore escapes __init__, escapes create_session_runtime, and escapes Session.__init__.
  • The scheduler catches only except (ValueError, NotImplementedError, OSError) at scheduler.py:1251, so it propagates out of _create_new_sessions — which is called from the scheduler run() loop. That is the thread that heartbeats, so the failure mode is not "this session degrades to job_name=None", it is the worker stopping.

The trigger is service-controlled and is exactly the payload class the graceful-degradation story is meant to absorb: well-formed JSON whose entries are not valid symbol-table entries. from_json_str accepts those, and to_symtab() is where they blow up.

Two options:

  • Catch BaseException here (this helper is pure parsing with a None fallback, so there is nothing to lose by being total), or
  • do not call to_symtab() in __init__ at all — resolve job_name lazily, or add SessionRuntimeCrashError/BaseException to the scheduler's construction guard.

Worth noting the current tests cannot see this: test_construction_passes_none_when_resolved_table_is_malformed_json patches nothing on the Rust side and feeds "{not json", which fails in from_json_str — the one path that does raise a normal Exception. A case that reaches to_symtab() would pin the real behaviour.

logger.warning("Failed to extract Job.Name from resolvedSymbolTable: %s", e)
return None


class PythonSessionRuntime(SessionRuntime):
"""SessionRuntime backed by openjd.sessions (v0 Python implementation)."""
Expand All @@ -38,6 +63,7 @@ def __init__(self, config: SessionRuntimeConfig) -> None:
callback=config.action_callback,
os_env_vars=config.os_env_vars,
session_root_directory=config.session_root_directory,
job_name=_extract_job_name(config.resolved_symbol_table_json),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sourcing job_name from the resolved symbol table introduces an unvalidated string into the v0 session, and it is worth knowing where that value lands before shipping.

Job.Name is a user-controlled field — it is whatever the job submitter typed. In the v0 openjd.sessions.Session, job_name is not inert: it feeds the session log banner and, depending on version, working-directory/log naming. So the questions are the ordinary ones for any user string crossing into path or log construction:

  1. Length and content. Nothing here bounds the string or rejects control characters. _extract_job_name checks only isinstance(value, str). If the v0 session incorporates job_name into a filesystem path, a name containing /, .., or a NUL is worth confirming as handled upstream rather than assumed. If it only reaches log output, embedded newlines still allow log-line forgery in the agent log — a name containing \n followed by a plausible-looking log prefix.
  2. Where the value comes from now. Note this is a new trust path, not a relabelling of an existing one. Previously the worker never took Job.Name from the wire at all on this code path; now it does, via a field (resolvedSymbolTable) whose entries the worker does not otherwise inspect. The value bypasses the expected_type=str check that guards the outer field, because that check validates the JSON string, not the Job.Name entry inside it.

Given the peek already parses the table anyway, a length cap and a rejection of control characters (or at minimum a comment recording that upstream openjd.sessions sanitises job_name before any path or log use) would close this cheaply. If upstream already validates, saying so here is still useful — it is not evident from this call site, and the isinstance check suggests validation was considered the caller responsibility.

revision_extensions=RevisionExtensions(
# Currently for simplicity request that our session allow all extensions.
# This does not obey the spec. It should be changed at a later date to the
Expand All @@ -53,7 +79,10 @@ def enter_environment(
environment: EnvironmentModel,
identifier: Optional[EnvironmentIdentifier] = None,
os_env_vars: Optional[dict[str, str]] = None,
resolved_symbol_table_json: str | None = None,
) -> EnvironmentIdentifier:
# resolved_symbol_table_json: not forwarded — the v0 Python session does
# not support pre-resolved symbol tables.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python (v0) runtime accepts resolved_symbol_table_json and drops it with no runtime signal. If the service ever sends resolvedSymbolTable because it contains values the worker cannot derive locally, a worker on the v0 runtime resolves those references from its own context instead — silently producing different values rather than failing. Worth at least a one-time logger.debug/warning when a non-None table is discarded, so this is diagnosable from logs instead of only from reading the source comment. (Same applies to the run_task override below.)

return self._session.enter_environment(
environment=environment,
identifier=identifier,
Expand All @@ -66,7 +95,10 @@ def exit_environment(
identifier: EnvironmentIdentifier,
os_env_vars: Optional[dict[str, str]] = None,
keep_session_running: bool = False,
resolved_symbol_table_json: str | None = None,
) -> None:
# resolved_symbol_table_json: not forwarded — the v0 Python session does
# not support pre-resolved symbol tables.
self._session.exit_environment(
identifier=identifier,
os_env_vars=os_env_vars,
Expand All @@ -81,7 +113,10 @@ def run_task(
os_env_vars: Optional[dict[str, str]] = None,
log_task_banner: bool = True,
step_name: str | None = None,
resolved_symbol_table_json: str | None = None,
) -> None:
# resolved_symbol_table_json: not forwarded — the v0 Python session does
# not support pre-resolved symbol tables.
self._session.run_task(
step_script=step_script,
task_parameter_values=task_parameter_values,
Expand Down
Loading
Loading