Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Methods:

All paths are resolved inside the sandbox root. Attempts to escape (via `..` or absolute paths) raise errors immediately.

### TemplateCall
### TemplateCall / `llm_call`

Lets one template invoke another with controlled inputs. Example configuration:

Expand All @@ -79,6 +79,14 @@ run(
)
```

The public tool LLMs see is called `llm_call`, which maps its parameters onto `TemplateCall.run`:

- `task` → `template`
- `extra_context` → `fragments`
- `attachments`, `params`, and `expect_json` pass through unchanged

Think of `llm_call` as "delegate this subtask to a separate LLM worker with its own context and attachments," backed by the safety checks above.

This enforces allowlists, file size/type restrictions, and attachment limits. It also supports template locking (force all calls to use a specific vetted template) and structured outputs via `expect_json=True`. Only set `expect_json=True` if the target template defines `schema_object`; otherwise TemplateCall will error. When the child template omits its own `model`, TemplateCall automatically inherits the caller's model so sub-calls use the same provider/config by default.

## Why TemplateCall?
Expand Down Expand Up @@ -111,7 +119,7 @@ examples/pitchdeck_eval/
The orchestrator template:
1. Lists PDFs in `pipeline/` using `Files("ro:pipeline")`
2. Decides which files to process (could be all of them, or just a subset based on task description)
3. For each file, calls `pitchdeck-single.yaml` via `TemplateCall_run`, passing the PDF as an attachment and `PROCEDURE.md` as a fragment
3. For each file, calls `pitchdeck-single.yaml` via `llm_call`, passing the PDF as an attachment and `PROCEDURE.md` as a fragment
4. Writes the resulting Markdown evaluations to `evaluations/` using `Files("out:evaluations")`

Run it like this:
Expand Down
60 changes: 37 additions & 23 deletions docs/templatecall.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
# TemplateCall: Design and Motivation
# TemplateCall and llm_call: Design and Motivation

This document explains why `TemplateCall` exists and how it fits into the `llm-do` plugin architecture.
This document explains why `TemplateCall` exists, how it fits into the `llm-do` plugin architecture, and how the public tool `llm_call` exposes it to models.

## The Problem
## Programmer vs LLM Perspective

When automating tasks with LLM templates, you frequently encounter workflows that require multiple LLM calls. The simplest and most common case is a two-step pattern:

1. **Selection:** Examine a directory, triage files, choose what to process
2. **Execution:** For each chosen item, run a focused LLM call with specific attachments and context

You could write Python code to orchestrate these steps, but that's slow to iterate on. Templates are easier to modify, but they need a way to safely invoke other templates. That's the gap `TemplateCall` fills.
- **Programmers** work with `TemplateCall` directly in Python/YAML. Their mental model is "call another template in a safe, controlled way" with:
- template allowlists or locks
- attachment validation (count, size, suffix)
- fragment handling for extra context
- structured outputs via `expect_json` + `schema_object`
- model inheritance and security toggles
- **LLMs** only see a generic tool named `llm_call` that means "delegate a subtask to another LLM worker with its own context and attachments." The model chooses a `task`, provides `input`, optional `attachments` and `extra_context`, and (optionally) `params`/`expect_json`. The model does not need to know anything about templates—the configuration lives behind the tool.

## What TemplateCall Does

Expand All @@ -34,10 +35,10 @@ TemplateCall(
)
```

The `run` method mirrors the `llm -t` CLI interface:
Programmer-facing API call:

```python
result = TemplateCall_run(
result = TemplateCall().run(
template="templates/evaluate-one.yaml",
input="Evaluate this document",
attachments=[{"path": "file.pdf"}],
Expand All @@ -47,21 +48,36 @@ result = TemplateCall_run(
)
```

LLM-facing tool surface (wired to the same implementation):

```yaml
# Conceptual view of the tool the LLM sees
- name: llm_call
description: Call a separate LLM worker with its own context and optional file attachments.
params:
task: which worker/template to call
input: main text to process
attachments: list of files to attach
extra_context: optional snippets/rubrics
params: optional template parameters
expect_json: request structured output when available
```

`llm_call` maps parameters like this: `task` → `template`, `extra_context` → `fragments`, while `attachments`, `params`, and `expect_json` pass through with the same validation rules. (TODO: Consider adding additional tool aliases such as `delegate_task` or `call_subtask` if certain models respond better to alternate names.)

## Why Recursion Matters

When a template can call another template, the template language becomes recursively closed. This is useful for a couple of reasons:

1. **Composability:** Common patterns like "choose files → process files" or "triage → escalate" become reusable building blocks instead of bespoke scripts.

2. **Uniformity:** Sub-calls inherit the same auditing, logging, and security guarantees as top-level template invocations.

3. **Programmer ergonomics:** Clean recursion is easier to reason about than ad-hoc orchestration glue. You can build workflows that feel like composing functions.

This isn't just theoretical—it matters in practice. When your orchestrator needs to handle edge cases (retry logic, partial failures, dynamic template selection), having a consistent primitive for "call another template" makes those extensions straightforward.

## The Two-Step Pattern in Practice

Here's a simplified example showing how `TemplateCall` fits into a pitch deck evaluation workflow:
Here's a simplified example showing how `TemplateCall` and `llm_call` fit into a pitch deck evaluation workflow:

```yaml
# pitchdeck-orchestrator.yaml (pseudo-YAML, simplified)
Expand All @@ -71,7 +87,7 @@ system: |

tools:
- Files_list
- TemplateCall_run
- llm_call

params:
max_decks: 5
Expand All @@ -80,10 +96,10 @@ params:
# 1. Call Files_list to get pipeline/*.pdf
# 2. Choose up to max_decks files based on task description
# 3. For each chosen file:
# - Call TemplateCall_run with:
# template: "templates/pitchdeck-single.yaml" (locked)
# - Call llm_call with:
# task: "templates/pitchdeck-single.yaml" (locked)
# attachments: [the PDF]
# fragments: ["PROCEDURE.md"]
# extra_context: ["PROCEDURE.md"]
# expect_json: true
# 4. Write results to evaluations/ via Files_write_text
```
Expand All @@ -93,11 +109,8 @@ The locked `pitchdeck-single.yaml` template expects exactly one PDF, returns str
## Benefits

- **Tight context:** Each sub-call is scoped to a single unit of work (one file, one task) rather than batching everything into a single bloated prompt.

- **Guardrails by construction:** File size caps, suffix restrictions, and template locks are enforced in code, not by hoping the LLM respects instructions.

- **Reproducibility:** Sub-calls are explicit, loggable, and re-runnable. You can audit exactly which template processed which files with which parameters.

- **Iteration speed:** Refining the evaluation template doesn't require touching the orchestrator. They evolve independently.

## When to Use TemplateCall
Expand Down Expand Up @@ -129,7 +142,7 @@ The `examples/pitchdeck_eval` directory demonstrates this pattern. The orchestra

1. Lists PDFs in `pipeline/` using `Files("ro:pipeline")`
2. Chooses which ones to evaluate (based on the task description)
3. Calls the locked `pitchdeck-single.yaml` template once per PDF, passing:
3. Calls the locked `pitchdeck-single.yaml` template once per PDF via `llm_call`, passing:
- The PDF as an attachment
- `PROCEDURE.md` as a fragment (shared evaluation rubric)
- Parameters like `deck_id` derived from the filename
Expand All @@ -156,8 +169,9 @@ The `TemplateCall` toolbox is implemented in `llm_do/tools_template_call.py`. Ke
- If the child template does not declare a `model`, TemplateCall automatically inherits the parent template's model for the sub-call.
- Inline Python functions in sub-templates are ignored by default (security); override via `allow_functions=True` if needed
- Fragment files are read and passed as text to the template (useful for procedures, rubrics, or context snippets)
- From the LLM's point of view, all of this is exposed as the single `llm_call` tool. The model only decides which `task` to call, what `input` to send, which files to attach, and any extra snippets of context.

The API deliberately mirrors the `llm -t` CLI to keep the mental model consistent.
There is currently no external code depending on this API, so the naming shift from `TemplateCall_run` to `llm_call` is not a breaking change. We're optimizing names now for clarity of mental models (programmer vs LLM) before external adoption.

## Future Directions

Expand Down
8 changes: 5 additions & 3 deletions examples/pitchdeck_eval/templates/pitchdeck-orchestrator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ system: |
Only touch files via the provided Files toolboxes:
- Files_pipeline_ro_* is read-only and points at ./pipeline
- Files_evaluations_out_* is writable and points at ./evaluations
Call TemplateCall_run exactly once per deck.
You have a tool called llm_call for delegating to a pitch-deck worker with
its own context and attachments. Use llm_call exactly once per deck when you
need that worker to read the PDF and apply the rubric.
tools:
- Files("ro:pipeline")
- Files("out:evaluations")
Expand Down Expand Up @@ -47,9 +49,9 @@ prompt: |
2. Load the evaluation procedure if `pipeline/PROCEDURE.md` exists using
Files_pipeline_ro_read_text.
3. For each PDF (one at a time):
- Call TemplateCall_run with:
- Call llm_call with:
* attachments: a list containing the absolute path to the PDF
* fragments: include the procedure text if available
* extra_context: include the procedure text if available
* expect_json: true
- Parse the JSON response and convert it to Markdown via render_markdown.
- Write the Markdown to Files_evaluations_out_write_text using the slugified
Expand Down
8 changes: 6 additions & 2 deletions llm_do/templates/generic-orchestrator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,14 @@ system: |
Available tools:
- Files_workspace_ro_list() and Files_workspace_ro_read_text() for reading files
- Files_workspace_out_write_text() for writing/creating files
- TemplateCall_run() to execute sub-templates with tightly scoped context
- llm_call(task, input, attachments, extra_context) to delegate work to a
specialized template/LLM worker with its own context and attachments
- TemplateHelpers_make_template(prompt_text) to generate a standard template

Both Files toolboxes point to the current directory.
Use llm_call when you need a separate LLM worker to process a unit of work,
especially when that worker needs attachments or extra snippets of context
that shouldn't bloat your own prompt.
tools:
- 'Files({"mode": "ro", "path": ".", "alias": "workspace_ro"})'
- 'Files({"mode": "out", "path": ".", "alias": "workspace_out"})'
Expand All @@ -30,6 +34,6 @@ prompt: |
2. If no sub_template provided, create a `.yaml` template in `templates/generated/`:
- Use TemplateHelpers_make_template(prompt_text) to generate template content
- Write it with Files_workspace_out_write_text
3. For each unit: call TemplateCall_run with the template, attachments, and
3. For each unit: call llm_call with the template, attachments, and
any relevant fragments, then write results using Files_workspace_out_write_text
4. Report units processed, output location, and template path for reuse
37 changes: 37 additions & 0 deletions llm_do/tools_template_call.py
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,43 @@ def run(
return json.dumps(payload, sort_keys=True, indent=2)
return text

def llm_call(
self,
task: str,
input: str = "",
attachments: Optional[List[str]] = None,
extra_context: Optional[List[str]] = None,
params: Optional[dict] = None,
expect_json: bool = False,
) -> str:
"""
Public LLM-facing tool for delegating a subtask to another template.

TODO: Consider exposing additional aliases (for example, "delegate_task"
or "call_subtask") if specific models prefer alternate names.
"""

return self.run(
template=task,
input=input or "",
attachments=attachments,
fragments=extra_context,
params=params,
expect_json=expect_json,
)

def tools(self):
"""Expose the llm_call tool to LLMs."""

yield llm.Tool.function(
self.llm_call,
name="llm_call",
description=(
"Call a separate LLM worker with its own context and optional "
"file attachments."
),
)

# helpers ----------------------------------------------------------
def _display_tool_executions(self, content_blocks):
"""Display tool execution details similar to --tools-debug."""
Expand Down