Skip to content

BlockingIOError on large agent prompt logging in non-interactive / piped environments #543

Description

@hertznsk

Describe the bug

When running a workflow in a non-interactive environment (such as cron or when stderr/stdout is connected to a pipe), conductor run fails with BlockingIOError: [Errno 11] write could not complete without blocking if an agent's rendered prompt is large (e.g. > 64 KB).

Context & Reproduction Details

  • Workflow scenario: A multi-agent workflow where parallel collector agents gather data and pass aggregated outputs to a downstream synthesis agent.
  • Prompt size: The rendered prompt for the synthesis agent was 58,161 characters (~105 KB UTF-8).
  • Execution environment: Running via cron (0 7 * * * /bin/bash -lc "... conductor run workflow.yaml --no-interactive").
  • In cron/piped environments, standard streams are pipes without a TTY. When writing large continuous blocks exceeding the OS pipe buffer capacity (typically 64 KB on Linux) to a pipe descriptor in non-blocking mode, sys.stderr.write() / self.file.write() raises BlockingIOError.

Observed Behavior & Stack Trace

  1. Orchestrator completes collector agents, routes to synthesis, prepares the agent prompt, and attempts to log it via verbose_log_section.
  2. Logging to _verbose_console / Rich Console fails on self.file.write(text):
  File ".../site-packages/conductor/executor/agent.py", line 490, in execute
    _verbose_log_section(
        f"Prompt for '{agent.name}'",
        rendered_prompt,
    )
  File ".../site-packages/conductor/executor/agent.py", line 40, in _verbose_log_section
    verbose_log_section(title, content)
  File ".../site-packages/conductor/cli/run.py", line 401, in verbose_log_section
    _verbose_console.print(
        Panel(
            Text(content),
            title=styled("[cyan]{}[/cyan]", title),
            border_style="dim",
        )
    )
  ...
  File ".../site-packages/rich/console.py", line 2124, in _write_buffer
    self.file.write(text)
BlockingIOError: [Errno 11] write could not complete without blocking
  1. The engine catches the exception and attempts error recovery / saving checkpoint, but the teardown handlers (_print_resume_instructions and print_error) also invoke console printing, triggering repeated BlockingIOError exceptions.

Suspected Root Cause (Presumed)

  • verbose_log_section unconditionally prints the full rendered prompt Panel to console when in full verbose mode (which is default).
  • Diagnostic / verbose logging to standard output/error streams does not handle BlockingIOError / non-blocking pipe semantics.
  • A failure during verbose logging aborts the entire workflow execution rather than being handled gracefully or suppressed.

Expected Behavior

  • Verbose console logging should not crash the workflow execution if write() cannot complete immediately or if the pipe buffer is full.
  • Error recovery handlers (_print_resume_instructions, print_error) should survive stream write errors without crashing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions