Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .github/scripts/changelog-for-release.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -125,5 +125,23 @@ assert_contains 'frameworks path included' '- **frameworks/langchain**: new adap
assert_not_contains 'path outside the SDK excluded' 'outside the python tree' "$out"
assert_not_contains 'seed before previous tag excluded' 'seed core' "$out"

mkdir -p plugins/hermes
printf 'hermes\n' > plugins/hermes/file.txt
git add plugins/hermes/file.txt
git commit -q -m 'feat(hermes): import plugin'
git tag sdk-python/v9.0.0
out=$("$CHANGELOG" 0.10.1 plugins/hermes plugins/hermes)
assert_contains 'first Hermes release includes import without a baseline tag' '- **hermes**: import plugin' "$out"
assert_not_contains 'Hermes excludes SDK changes' 'core export change' "$out"
assert_not_contains 'Hermes excludes other plugin changes' 'repair login' "$out"

git tag plugins/hermes/v0.10.1
printf 'fix\n' >> plugins/hermes/file.txt
git add plugins/hermes/file.txt
git commit -q -m 'fix(hermes): repair export'
out=$("$CHANGELOG" 0.10.2 plugins/hermes plugins/hermes)
assert_contains 'Hermes uses its own previous tag' '- **hermes**: repair export' "$out"
assert_not_contains 'previous Hermes release excluded' 'import plugin' "$out"

echo "passed: ${pass}, failed: ${fail}"
[[ $fail -eq 0 ]]
6 changes: 6 additions & 0 deletions .github/scripts/changelog-top-version.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,12 @@ assert_eq 'first of several sections wins' '1.2.3' "$("$TOP_VERSION" "$FILE")"
printf '# Changelog\n\n## [Unreleased]\n\n## [0.9.0] - 2026-01-01\n\n- c\n' > "$FILE"
assert_eq 'non-semver heading skipped' '0.9.0' "$("$TOP_VERSION" "$FILE")"

printf '# Changelog\n\n## [Unreleased]\n\n## [0.10.0](https://example.com/releases/0.10.0) - 2026-09-17\n' > "$FILE"
assert_eq 'Hermes linked version heading' '0.10.0' "$("$TOP_VERSION" "$FILE")"

printf '# Changelog\n\n## [0.10.1] - 2026-09-18\n\n## [0.10.0](https://example.com/releases/0.10.0) - 2026-09-17\n' > "$FILE"
assert_eq 'plain version heading precedes linked heading' '0.10.1' "$("$TOP_VERSION" "$FILE")"

assert_eq 'missing file prints nothing' '' "$("$TOP_VERSION" "${TMP}/absent.md" 2>/dev/null)"
assert_eq 'missing file still exits 0' 0 "$("$TOP_VERSION" "${TMP}/absent.md" >/dev/null 2>&1; echo $?)"

Expand Down
38 changes: 38 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,44 @@ jobs:
- name: Build opencode plugin
run: pnpm --filter @grafana/agento11y-opencode run build

hermes-checks:
name: Hermes lint, types, and artifacts
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
python-version: '3.11'
- uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4
with:
install: false
- run: mise run lint:py:plugin-hermes
- run: mise run typecheck:py:plugin-hermes
- run: mise run build:py:plugin-hermes

hermes-test:
name: Hermes Python ${{ matrix.python-version }}
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
python-version: ['3.11', '3.12', '3.13', '3.14']
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
python-version: ${{ matrix.python-version }}
- uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4
with:
install: false
- run: mise run test:py:plugin-hermes ${{ matrix.python-version }}

python-lint:
name: Python Lint
runs-on: ubuntu-latest
Expand Down
7 changes: 5 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,14 @@ Three steps run per release, and none of them creates a tag on the release PR:

## Plugins layout

`plugins/` ships two flavors of launcher. They are not uniform; don't assume they are.
`plugins/` contains launchers and in-process plugins. They are not uniform; don't assume they are.

| Plugin dir | What it actually is |
|------------|---------------------|
| `plugins/agento11y/` | The shared Go binary, installed as `agento11y` (`brew install grafana/grafana/agento11y`; the old `sigil` name still works but will be removed). Has subcommands `claude`, `codex`, `copilot`, `cursor`, `opencode`, `pi`, `vibe`, `login`, `doctor`, `local`, `history`, `skills`, `help`. This is also what consumers use. |
| `plugins/claude-code/`, `plugins/codex/`, `plugins/copilot/`, `plugins/cursor/` | Thin glue: hook scripts and READMEs that wire the host agent to the shared `agento11y` binary. No independent code paths. |
| `plugins/opencode/` | Independent npm package `@grafana/agento11y-opencode`. Runs in-process inside opencode through its TypeScript plugin API; `agento11y opencode` installs and launches it. |
| `plugins/hermes/` | Independent Python package `grafana-agento11y-hermes`. Runs in-process through Hermes's `agento11y` plugin entry point. No shared launcher/login/config/local mode or release-table registration. See `plugins/hermes/README.md` for setup and release limitations. |
| `plugins/pi/` | Independent npm package `@grafana/agento11y-pi`. Runs in-process inside pi; `agento11y pi` installs and launches it. |
| `plugins/vibe/` | README only. `agento11y vibe` upserts three `[[hooks]]` entries into `hooks.toml` under `$VIBE_HOME` (default `~/.vibe`) and sets `VIBE_ENABLE_EXPERIMENTAL_HOOKS=true` on the child, which only a vibe below 2.21.0 needs. Vibe 2.21.0 renamed all three hook types, so the install path picks the spelling from `vibe --version` and the hook dispatcher answers to both. See `internal/agents/vibe/version.go`. |

Expand Down Expand Up @@ -120,4 +121,6 @@ test_home=$(mktemp -d)
GOPATH="$go_path" GOCACHE="$go_cache" TMPDIR=/tmp GOWORK=off "$go_root/bin/go" test ./...)
```

`mise run check` is the full local CI gate: lint + typecheck + proto-drift + redaction-drift + every SDK suite. For a focused change, run the matching narrow task (e.g. `mise run test:py:sdk-langgraph`); the full gate is slow.
Hermes uses `format:py:plugin-hermes`, `lint:py:plugin-hermes`, `typecheck:py:plugin-hermes`, `test:py:plugin-hermes`, and `build:py:plugin-hermes`. The build validates wheel and source-distribution artifacts. CI tests Python 3.11–3.14 with branch coverage and a 99% minimum. Optional real-Hermes tests require an explicit loopback provider and both telemetry channels routed locally or disabled; inspect the plugin's e2e skill and scripts first.

`mise run check` is the full local CI gate: lint + typecheck + proto-drift + redaction-drift + every SDK suite + Hermes artifact validation. For a focused change, run the matching narrow task (e.g. `mise run test:py:sdk-langgraph`); the full gate is slow.
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@

Capture sessions from the coding agents you already use — Cursor, Claude Code, Codex, Copilot CLI, OpenCode, Pi, Vibe, and others — so you can observe usage, cost, tokens, and tools across all of them in one place.

Install `agento11y`:
For Hermes, use its separate [Python plugin guide](plugins/hermes/README.md). The shared launcher does not support Hermes.

For the other coding agents, install `agento11y`:

```sh
# Linux or macOS
Expand Down
14 changes: 9 additions & 5 deletions docs/concepts/content-capture-modes.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,12 @@ The default differs between SDK clients and coding-agent plugins.
| Surface | Default mode |
| --- | --- |
| Core SDK client (Go, Python, JS/TS, Java, .NET) | `no_tool_content`. Generation content is captured; tool-execution arguments and results stay out of spans. |
| Coding-agent plugins (shared `agento11y` binary, `@grafana/agento11y-pi`, `@grafana/agento11y-opencode`) | `metadata_only`. Coding-agent sessions usually run on shared machines, so the plugins ship metadata-only by default. |
| Coding-agent plugins (shared `agento11y` binary, `@grafana/agento11y-pi`, `@grafana/agento11y-opencode`, `grafana-agento11y-hermes`) | `metadata_only`. Coding-agent sessions usually run on shared machines, so the plugins ship metadata-only by default. |

`default` at the client level resolves to `no_tool_content`. To get full content on a core SDK client, set `contentCapture: 'full'` (or the language equivalent) explicitly.

These Hermes defaults apply to source installations. Published PyPI `grafana-agento11y-hermes` `0.10.0` defaults to full content without shared secret redaction. See the [Hermes installation guide](../../plugins/hermes/README.md#install).

## Resolution precedence

The SDK resolves capture mode differently by recording type and language.
Expand Down Expand Up @@ -76,9 +78,9 @@ Per-language READMEs include code examples:
- Java: [`java/README.md`](../../java/README.md)
- .NET: [`dotnet/README.md`](../../dotnet/README.md)

For coding-agent plugins, the relevant env var is `AGENTO11Y_CONTENT_CAPTURE_MODE`. All plugins (the shared `agento11y` binary used by Claude Code, Codex, Copilot, Cursor, and Vibe; Pi via `@grafana/agento11y-pi`; OpenCode via `@grafana/agento11y-opencode`) accept `full`, `no_tool_content`, `metadata_only`, and `full_with_metadata_spans`. `default` is accepted as an alias for `metadata_only` so plugins match the Go envconfig resolver rather than the JS SDK's client-level default of `no_tool_content`.
For coding-agent plugins, the relevant env var is `AGENTO11Y_CONTENT_CAPTURE_MODE`. All plugins (the shared `agento11y` binary used by Claude Code, Codex, Copilot, Cursor, and Vibe; Pi via `@grafana/agento11y-pi`; OpenCode via `@grafana/agento11y-opencode`; Hermes via `grafana-agento11y-hermes`) accept `full`, `no_tool_content`, `metadata_only`, and `full_with_metadata_spans`. `default` is accepted as an alias for `metadata_only` so plugins match the Go envconfig resolver rather than the JS SDK's client-level default of `no_tool_content`.

Unknown values fall back to `metadata_only` with a warning in the plugin log. A plugin can still export less than the SDK allows. For example, an adapter may drop a field if the host agent does not pass it through.
Unknown values fall back to `metadata_only`. The launchers, Pi, and OpenCode log a warning; Hermes falls back silently. A plugin can still export less than the SDK allows. For example, an adapter may drop a field if the host agent does not pass it through.

## Secret redaction in the plugins

Expand All @@ -88,11 +90,13 @@ A plugin redacts known secret formats out of every content field it exports: use

Set `AGENTO11Y_REDACT_INPUT_MESSAGES=false` in `~/.config/agento11y/config.env` or the environment to export prompt text without redaction. The flag covers the prompt only: every other field stays redacted, and message structure, roles, token counts, tags, and IDs do not change. An unrecognised value keeps redaction on, so a typo cannot disable it.

For Hermes, set variables in Hermes's environment or its `.env`, which overrides shell exports. The plugin does not read the shared launcher's config file. Prompt redaction defaults to on; the same opt-out applies. Hermes also sanitizes tool-execution spans, which do not pass through the generation sanitizer. Payload limits and upstream request clipping can reduce content further. Hermes also redacts secret-pattern matches in hook-derived IDs and metadata. Custom `AGENTO11Y_TAGS` values are not sanitized.

### Strength per field

There are two pattern tiers. Tier 1 is high-confidence secret formats (`glc_…`, `AKIA…`, a PEM block, a connection string). Tier 2 is the key/value heuristics (`PASSWORD=…`, `"token": "…"`), which catch a secret with no recognisable format but also fire on ordinary text.

Every plugin applies the same tier per field, and it is the tier the SDKs' generation sanitizer applies:
Every plugin applies the same tier per content field as the SDKs' generation sanitizer:

| Field | Tier | Why |
| --- | --- | --- |
Expand All @@ -105,7 +109,7 @@ Every plugin applies the same tier per field, and it is the tier the SDKs' gener

Tier 2 on a prompt has a real cost: `sort key: name` is exported as `sort key: [REDACTED:env-secret-value]`, because the heuristic cannot tell that `key:` is part of a sentence. Turn prompt redaction off with `AGENTO11Y_REDACT_INPUT_MESSAGES=false` if the prompt text matters more than the coverage. Tier 2 is kept off prose for that reason, and a secret a model repeats in prose is still caught by tier 1 as long as it has a known format.

On a tool payload that decodes as JSON, the shared `agento11y` binary also redacts a value under a secret-looking key (`authorization`, `cookie`, `client_secret`), which the tier 2 key list does not cover. The OpenCode and Pi plugins do not: they redact the encoded JSON as text, so they catch only the key names in the tier 2 patterns.
On a tool payload that decodes as JSON, the shared `agento11y` binary also redacts a value under a secret-looking key (`authorization`, `cookie`, `client_secret`), which the tier 2 key list does not cover. The SDK-based OpenCode, Pi, and Hermes sanitizers do not: they redact the encoded JSON as text, so they catch only the key names in the tier 2 patterns.

## Related

Expand Down
10 changes: 10 additions & 0 deletions docs/concepts/tags-and-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,12 @@ The coding-agent plugins (claude-code, codex, copilot, cursor, opencode, pi, vib

Launchers also set a few keys specific to one host, so this table is not the full list of what arrives on a generation.

### Hermes tags

The [Hermes plugin](../../plugins/hermes/README.md) does not use the shared launcher. It emits no automatic `cwd` and no unconditional `git.branch`. The launcher built-ins above do not apply. Automatic `user`, `repo`, and `git.branch` client tags require the switches below. Explicit `AGENTO11Y_TAGS` values win over automatically resolved values.

Hermes resolves the user from `AGENTO11Y_USER_ID`, then the operating-system account; it has no signed-in host-account lookup. Its process-wide client freezes automatic values at initialization, unlike a launcher invocation or a per-session client. Set the switches in Hermes's environment, not the shared launcher's config. `agento11y login` and `doctor` do not configure or diagnose Hermes.

## Opt-in automatic tags (`AGENTO11Y_AUTO_CODING_AGENT_TAGS`)

The built-in tags above are per-generation tags, so they reach the Agent Observability UI but never become metric labels. `AGENTO11Y_AUTO_CODING_AGENT_TAGS` resolves the same kind of session facts and attaches them as **client tags** instead, which is the one mechanism that does reach OTel metrics. That is what lets the Usage and Cost view filter and break down by user, repository, or branch. It is a coding-agent-plugin feature; the SDKs have nothing like it.
Expand Down Expand Up @@ -192,6 +198,8 @@ Enabling these names is a deliberate trade. Read this first:
- `repo` and `user` are usually bounded per organization. `branch` is not. Set `AGENTO11Y_AUTO_CODING_AGENT_TAGS_NAMES=user,repo` first and add `branch` only if you need per-branch cost.
- In the pi and opencode plugins the client is built once per session, so their metric labels freeze at session start. A checkout that changes mid-session keeps the label it started with. The hook-based agents (claude-code, codex, copilot, cursor, vibe) build a client per invocation and follow the checkout.

For Hermes, automatic tags use the same opt-in and allowlist contract, but unsupported names and an inactive allowlist are currently ignored without logging. No per-generation branch override is added. Removing content with `metadata_only` does not remove tags; enabling user or repository labels still exposes those values.

## Built-in metadata from the agent launchers

Metadata is exported but never turned into a metric label, so launchers use it for numbers and for keys with too many distinct values to be a tag.
Expand All @@ -209,6 +217,8 @@ Codex and copilot also add their own `codex.*` and `copilot.*` keys, so this tab
| `opencode.parent_session_id` | Session id of the run that spawned this subagent session. On every subagent generation, including one whose parent turn could not be named. | opencode |
| `opencode.child_session_id` | Subagent's own session id. Present when its turns were reparented onto the spawning conversation, where `conversation_id` names the root session of the subagent chain instead. | opencode |

Hermes exports host facts under `hermes.*` metadata, including `hermes.request_facts_reused` when it reuses cached request fields. These facts are generation metadata, not client tags or metric labels. Tool names and sampling parameters may remain visible in metadata-only mode; request text and tool schemas do not. See the [Hermes README](../../plugins/hermes/README.md) for clipping and cache limitations.

## See also

- [Content Capture Modes](content-capture-modes.md) — which content fields ship. Content capture does not strip `tags` or `metadata`; both are always exported.
Expand Down
20 changes: 19 additions & 1 deletion docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,24 @@

Notes for contributors working in this repo.

## Hermes plugin

[`plugins/hermes`](../plugins/hermes/README.md) is an independent Python package. There is no release-table registration or publishing workflow. Install in Hermes's Python environment, not through the shared launcher.

Run these tasks from the repository root:

```sh
mise run format:py:plugin-hermes
mise run lint:py:plugin-hermes
mise run typecheck:py:plugin-hermes
mise run test:py:plugin-hermes
mise run build:py:plugin-hermes
```

The build task validates the wheel and source distribution, including package identity, version, dependencies, and entry point. Root `mise run check` includes artifact validation. CI tests Python 3.11, 3.12, 3.13, and 3.14 with branch coverage enabled and a 99% minimum. Keep the plugin's `uv.lock` synchronized with dependencies.

Tests must run without inherited Cloud/provider credentials or personal configuration. Real-Hermes tests are optional and separate from the normal checks. Read the [e2e skill](../plugins/hermes/.agents/skills/e2e-test/SKILL.md) and inspect scripts before running them. Use an explicit loopback model provider. Route both telemetry channels to loopback receivers, or disable unused channels. A local OTLP sink alone does not prevent paid model calls.

## Regenerating protobuf stubs

The proto lives at [`proto/agento11y/v1/generation_ingest.proto`](../proto/agento11y/v1/generation_ingest.proto). After editing it, regenerate every language's stubs from the repo root:
Expand Down Expand Up @@ -78,7 +96,7 @@ That writes five files:
| Output | Consumer |
| --- | --- |
| `go/agento11y/redaction_patterns_gen.go` | Go SDK |
| `python/agento11y/_redaction_patterns.py` | Python SDK |
| `python/agento11y/_redaction_patterns.py` | Python SDK; reused by the Hermes plugin's SDK redaction |
| `js/src/redaction-patterns.generated.ts` | JS SDK and, through `@grafana/agento11y-core`, the opencode plugin |
| `dotnet/src/Grafana.Agento11y/RedactionPatterns.g.cs` | .NET SDK |
| `plugins/agento11y/internal/redact/patterns_gen.go` | shared `agento11y` binary |
Expand Down
Loading
Loading