Skip to content

feat(agents): emit decision.answered and decision.defaulted telemetry (spec 136) - #929

Merged
arielshad merged 7 commits into
mainfrom
feat/135-decision-telemetry
Oct 11, 2026
Merged

arielshad merged 7 commits into
mainfrom
feat/135-decision-telemetry

Conversation

@arielshad

@arielshad arielshad commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

What

This is the telemetry follow-up noted in #922 and #924. Every recorded answer to an agent decision now emits decision.answered, and a background agent's question that reaches its deadline emits the new decision.defaulted event. Both go through the ITelemetry port from #923.

Event Properties Recorded by
decision.answered kind (DecisionKind), surface (new DecisionSurface: web / cli / chat / supervisor / other), latency (bucket from asked to answered), pickedRecommended AnswerAgentQuestionUseCase, once per answer it actually records
decision.defaulted kind, timeout (bucket of the deadline the agent set) AskAgentDecisionUseCase, when its deadline settle wins

Why

Spec 133 defined decision.answered but did not emit it, and its kind and surface were raw strings. The T3 Code review asks three things that need data: whether people answer agent questions, where they answer them, and whether they take the recommendation. It also asks how often agents proceed on their own at the deadline.

  • Emit points: each event is emitted right after the use case's atomic settlePending succeeds. Each decision is therefore counted exactly once without an onceKey. A lost race (CLI vs web), a refused answer, or agent questions being off records nothing. A person answering at the deadline records an answer, not a default.
  • Surface: each caller passes its own surface: the web action (web, which overrides anything the client sends), the CLI (cli), the chat bridge (chat) and the supervisor router (supervisor). Nothing parses the free-form answeredBy actor.
  • Typing: in the property map, kind and surface are now enums. That makes the compiler reject a content property at every emit site.
  • Refactor while touching the file: the AskAgentDecisionOutcome string union is now the DecisionOutcome enum, with a new Disabled member. The MCP tool's wire values are unchanged. Stored questions use RecordedDecisionOutcome, which excludes Disabled, so the activity log needs no UI for a state it can never show.
  • Docs: docs/telemetry.md lists both events. Spec: specs/136-decision-telemetry/. It was renumbered from 135, which belongs to feat(web): review cleanup — per-area feature flags, ASPM off by default #925; the branch name still says 135.

Screenshots / Recording

No UI changes.

Testing

  • answer-agent-question.use-case.test.ts:
    • Properties for a decision answered on the web, including the latency bucket with a faked clock and pickedRecommended: true.
    • Typed text counts as not the recommendation.
    • A legacy question is reported as Legacy, with Other as the default surface.
    • Answering twice records one event.
    • Agent questions off, or a refused answer, records nothing.
  • ask-agent-decision.use-case.test.ts:
    • The deadline default records decision.defaulted with the timeout bucket.
    • A person answering at the deadline records no default.
    • Cancelled and disabled outcomes record no default.
  • Caller tests assert the surface they pass: web action, CLI answer (with --answer and interactive), chat bridge, supervisor router.

Ran locally: pnpm lint, format:check, check:stories, typecheck, test:unit (14864 passed), test:int (2132 passed), build, build:web, build:storybook, and generate with a clean diff. After the renumbering: lint, format:check, typecheck and the affected test files.

Checklist

  • pnpm lint passes
  • pnpm format:check passes
  • pnpm typecheck passes
  • pnpm test:unit and pnpm test:int pass
  • pnpm build succeeds
  • (Domain changes) pnpm tsp:compile ran and packages/core/src/domain/generated/output.ts is committed
  • Tests landed RED-first per the TDD guide
  • No domain/ or application/ file imports anything from infrastructure/
  • Commit messages follow Conventional Commits

🤖 Generated with Claude Code

https://claude.ai/code/session_014Zwb23kb4xTBxJtqc2hKbU

claude and others added 6 commits October 11, 2026 10:53
Claude-Session: https://claude.ai/code/session_014Zwb23kb4xTBxJtqc2hKbU

Co-Authored-By: Shep Bot <shep-agent@users.noreply.github.com>
…ion.defaulted

decision.answered carries DecisionKind and the new DecisionSurface enum instead of raw strings.
DecisionOutcome gains Disabled for an ask made while agent questions are off; stored questions
keep the narrower RecordedDecisionOutcome.

Claude-Session: https://claude.ai/code/session_014Zwb23kb4xTBxJtqc2hKbU

Co-Authored-By: Shep Bot <shep-agent@users.noreply.github.com>
…endation pick

AnswerAgentQuestionUseCase records one decision.answered after its settle write wins, with the
decision kind, the surface its caller names (web, cli, chat, supervisor), a latency bucket and
whether the recommended option was picked. A lost race or a refused answer records nothing.

Claude-Session: https://claude.ai/code/session_014Zwb23kb4xTBxJtqc2hKbU

Co-Authored-By: Shep Bot <shep-agent@users.noreply.github.com>
…eadline

AskAgentDecisionUseCase records decision.defaulted with the decision kind and the deadline bucket
when its expired settle wins; a person answering at the deadline records nothing. Its outcome
is now the DecisionOutcome enum instead of a string union, with identical wire values.

Claude-Session: https://claude.ai/code/session_014Zwb23kb4xTBxJtqc2hKbU

Co-Authored-By: Shep Bot <shep-agent@users.noreply.github.com>
135 belongs to the review-cleanup spec in #925.

Claude-Session: https://claude.ai/code/session_014Zwb23kb4xTBxJtqc2hKbU

Co-Authored-By: Shep Bot <shep-agent@users.noreply.github.com>
@arielshad arielshad changed the title feat(agents): emit decision.answered and decision.defaulted telemetry (spec 135) feat(agents): emit decision.answered and decision.defaulted telemetry (spec 136) Oct 11, 2026
…emetry

Co-Authored-By: Shep Bot <shep-agent@users.noreply.github.com>
@arielshad
arielshad merged commit 1e1f859 into main Oct 11, 2026
23 checks passed
@arielshad
arielshad deleted the feat/135-decision-telemetry branch October 11, 2026 12:11
arielshad pushed a commit that referenced this pull request Oct 11, 2026
<p align="center">
  <a href="https://github.com/shep-ai/shep">
    <img src="https://raw.githubusercontent.com/shep-ai/shep/main/docs/screenshots/shep-card.jpg" alt="Shep — run multiple AI agents in parallel" width="720" />
  </a>
</p>

# 🚀 Shep [v1.241.0](/compare/v1.240.0...v1.241.0) · _2026-10-11_

> Your organization does not have access to Claude. Please login again or contact your administrator.

### ✨ Features

* **agents:** emit decision.answered and decision.defaulted telemetry (spec 136) ([#929](#929)) ([1e1f859](1e1f859)), closes [#922](#922) [#925](#925)

  ![default](https://raw.githubusercontent.com/shep-ai/shep/v1.241.0/specs/134-unified-decisions/evidence/decision-panel-default.png)
  ![dark](https://raw.githubusercontent.com/shep-ai/shep/v1.241.0/specs/134-unified-decisions/evidence/decision-panel-dark.png)
  ![gate](https://raw.githubusercontent.com/shep-ai/shep/v1.241.0/specs/134-unified-decisions/evidence/decision-panel-approval-gate.png)
  ![not resumable](https://raw.githubusercontent.com/shep-ai/shep/v1.241.0/specs/134-unified-decisions/evidence/decision-panel-not-resumable.png)
* **agents:** git checkpoints after each feature-agent step (spec 138) ([#931](#931)) ([26265ed](26265ed)), closes [#925](#925)

  ![checkpoints tab light](https://raw.githubusercontent.com/shep-ai/shep/v1.241.0/specs/138-git-checkpoints/evidence/checkpoints-tab-light.png)
  ![checkpoints tab dark](https://raw.githubusercontent.com/shep-ai/shep/v1.241.0/specs/138-git-checkpoints/evidence/checkpoints-tab-dark.png)
  ![checkpoints tab compare light](https://raw.githubusercontent.com/shep-ai/shep/v1.241.0/specs/138-git-checkpoints/evidence/checkpoints-tab-compare-light.png)
  ![checkpoints tab compare dark](https://raw.githubusercontent.com/shep-ai/shep/v1.241.0/specs/138-git-checkpoints/evidence/checkpoints-tab-compare-dark.png)
* **web:** put the SDLC board behind a feature flag, off by default ([#932](#932)) ([e71f138](e71f138)), closes [#925](#925)

### 🐛 Bug Fixes

* **cli:** wait for the conpty host to exit when a terminal closes ([#928](#928)) ([94a846d](94a846d)), closes [#925](#925) [#925](#925)

## 📦 Install or update

```bash
# upgrade an existing install
npm i -g @shepai/cli@1.241.0

# or run instantly without installing
npx @shepai/cli@latest
```

## 💬 Join the community

[💬 **Discord**](https://discord.gg/ES6tdVFfur) · [📖 **Docs**](https://github.com/shep-ai/shep#readme) · [⭐ **Star on GitHub**](https://github.com/shep-ai/shep) · [🐛 **Report an issue**](https://github.com/shep-ai/shep/issues)

---

<sub>🤖 Released autonomously by Shep — built by parallel AI agents working in isolated git worktrees. Try it: `npx @shepai/cli`</sub>

Co-Authored-By: Shep Bot <shep-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants