Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .storybook/mocks/app/actions/telemetry.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
/** Storybook mock for app/actions/telemetry (spec 133). */

export interface TelemetryStatusView {
enabled: boolean;
reason: string | null;
includeIdentity: boolean;
contactConsent: boolean;
queuedEvents: number;
configured: boolean;
destination: string;
}

export interface SetTelemetryPreferencesResult {
ok: boolean;
status?: TelemetryStatusView;
error?: string;
}

let status: TelemetryStatusView = {
enabled: true,
reason: null,
includeIdentity: false,
contactConsent: false,
queuedEvents: 12,
configured: true,
destination: 'https://eu.i.posthog.com/batch/',
};

export async function getTelemetryStatus(): Promise<TelemetryStatusView | null> {
return status;
}

export async function setTelemetryPreferences(input: {
enabled?: boolean;
includeIdentity?: boolean;
contactConsent?: boolean;
}): Promise<SetTelemetryPreferencesResult> {
status = {
...status,
...input,
reason: input.enabled === false ? 'user-opt-out' : status.reason,
queuedEvents: input.enabled === false ? 0 : status.queuedEvents,
};
return { ok: true, status };
}

export function recordWebAreaView(): Promise<void> {
return Promise.resolve();
}

export function recordOnboardingStep(): Promise<void> {
return Promise.resolve();
}
10 changes: 9 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,13 @@ See [docs/architecture/repository-pattern.md](./docs/architecture/repository-pat

## Data Storage

Everything Shep knows lives locally. There is no Shep server.
Everything Shep knows lives locally. There is no Shep server; the only outbound data Shep
itself produces is opt-out **usage metrics** (spec 133): every process writes content-free
events to the `telemetry_outbox` table and only the daemon / `shep ui` sends them to PostHog
(EU, `/batch/`). Record events through the `ITelemetry` port with a typed property map
(`TelemetryEventPropertyMap`) — never prompts, code, paths, repo/branch names, titles, ids or
error messages. Identity fields (agent account hash, GitHub username, GitHub owners) are
**opt-in** — `includeIdentity` defaults to `false`. See [docs/telemetry.md](./docs/telemetry.md).

| Path | What |
| ---- | ---- |
Expand All @@ -133,6 +139,8 @@ turns on verbose CLI/deployment logging. The web daemon's port comes from `shep
`shep ui --port`, not an env var — `SHEP_WEB_PORT` is *written* by the server and read only by
the middleware's Host-header check. `SHEP_BIND_HOST`, `SHEP_ALLOW_PUBLIC_BIND`,
`SHEP_ALLOWED_HOSTS` and `SHEP_WEB_REQUIRE_TOKEN` gate non-localhost access.
`SHEP_TELEMETRY_DISABLED=1`, `DO_NOT_TRACK=1` and `CI` force usage metrics off;
`SHEP_POSTHOG_KEY` / `SHEP_POSTHOG_HOST` override the PostHog project key and host.

- Engine: `better-sqlite3`, opened as a process-wide singleton in
`packages/core/src/infrastructure/persistence/sqlite/connection.ts`.
Expand Down
25 changes: 25 additions & 0 deletions LESSONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3017,6 +3017,23 @@ that a squash merge would invalidate.
`'packages/core/src/...'` passed on Linux and failed on `windows-latest`. Always
`.replace(/\\/g, '/')` a computed path before comparing or printing it in a test.

## A new hook in a shared shell is a change to every test that renders the shell

Adding `useRouteViewTelemetry()` to `AppShell` (it calls `useParams`) failed four AppShell suites
whose `vi.mock('next/navigation')` only returned `usePathname`/`useRouter`, and a new
`container.resolve('ITelemetry')` in `startBackgroundSync` failed the `ui` and `_serve` command
tests whose stubs throw on unknown tokens. Before mounting a hook in a layout, grep the tests that
mock its imports; before adding a resolve to a shared starter, grep the stubs of every command that
calls it. The web test setup replaces `localStorage` with `vi.fn()` mocks (`getItem` returns
`undefined`), so drive it with `vi.mocked(localStorage.getItem)` and treat "missing" as falsy, not
`!== null`.

## PostHog `/batch/` answers 200 for any project key

A live probe returned `200 {"status":"Ok"}` for a fake key and `400` for a malformed body. A
successful send proves the wire format, not that events reach a project — check the project's
event stream after configuring a real key.

## A scripted "insert before the first import" can land above `import 'reflect-metadata'`

Adding an import to `container-bootstrap.test.ts` with a script that inserted it before the
Expand All @@ -3038,3 +3055,11 @@ best-effort (the approval use cases already wrap theirs); a use case that writes
needs at least one test against the migrated SQLite schema; and a new process entry point
(an MCP server, a worker) gets one real spawn-and-call smoke before it is called done — that run
also caught the entry skipping `initializeSettings()`.

## Fields that identify a person are opt-in, even when usage metrics are opt-out

Spec 133 first shipped "Include my identity" on by default (account hash, GitHub username and
owners). The owner reversed it before merge: anonymous usage events may be opt-out, but anything
that identifies a person defaults to off and the notice offers to turn it on. Keep one source for
the default (`DEFAULT_TELEMETRY_PREFERENCES`) and make the migration's column default, the mapper's
fallback and every `?? default` read it — the first version had `?? true` in three places.
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ Full reference: [docs/cli/commands.md](./docs/cli/commands.md) — every command
<details>
<summary><strong>Trust and safety</strong> — what runs where, and what protects you</summary>

Shep runs entirely on your machine. All data lives in `~/.shep/` as SQLite; your code is sent only to whichever agent you configure, under that agent's own terms. Nothing is sent to Shep servers — there are none.
Shep runs entirely on your machine. All data lives in `~/.shep/` as SQLite; your code is sent only to whichever agent you configure, under that agent's own terms. The one thing Shep itself sends is **usage metrics** (below), and you can turn them off.

| Concern | How Shep handles it |
|---------|---------------------|
Expand All @@ -173,6 +173,7 @@ Shep runs entirely on your machine. All data lives in `~/.shep/` as SQLite; your
| Credentials | With session auth (the default) Shep never touches your agent credentials — the agent uses its own login. If you choose token auth, the key you hand `shep settings agent --token` is stored in your local settings database (`~/.shep/data`, as plain text) and passed only to the provider you selected. |
| Audit trail | Every action and state transition is logged — `shep feat logs <id>`. |
| Emergency stop | `shep agent stop <id>` or the dashboard stop button. The worktree is preserved. |
| Usage metrics | On by default, announced on first run. About ten kinds of usage event go to the Shep team's PostHog project (EU region), with a random install id. Identity is opt-in: only if you turn it on (`shep telemetry identity on`) do events also carry a SHA-256 hash of your Claude/Codex account id, your GitHub username and the GitHub owners of repositories you use. Never prompts, code, file paths, repository or branch names, feature titles or error messages. `shep telemetry off` (or `DO_NOT_TRACK=1`) stops it; it is always off under `CI`. See [docs/telemetry.md](./docs/telemetry.md). |

**Agent permissions:** Shep runs your agent non-interactively, so by default it passes permission-bypass flags (e.g. `--dangerously-skip-permissions` for Claude Code — each agent has an equivalent). Your safety net is three layers deep: worktree isolation, draft PRs, and your CI pipeline. The bypass flag is a default, not a requirement — configure your agent's permission model independently for tighter control. Note that some agents sandbox network access by default; if `npm install` fails inside a feature, allow the hosts in your agent's settings.

Expand Down Expand Up @@ -201,6 +202,9 @@ Shep runs locally per developer. Features are just branches and PRs — your exi
**Is my code sent anywhere?**
Not by Shep. It goes only to the agent you configure, under that agent's privacy terms. Shep stores everything locally.

**Does Shep collect anything?**
Usage metrics, on by default: which commands, pages and features are used, how runs end, and whether PRs merge — never content. Identity fields (an account-id hash, GitHub username and owners) are off by default and sent only if you turn them on with `shep telemetry identity on`. `shep telemetry show` prints exactly what will be sent; `shep telemetry off` turns it all off. Details in [docs/telemetry.md](./docs/telemetry.md).

**Not in a git repo yet?**
Shep initializes one for you — `git init`, a branch, and off it goes.

Expand Down
10 changes: 10 additions & 0 deletions apis/json-schema/OnboardingStep.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
$schema: https://json-schema.org/draft/2020-12/schema
$id: OnboardingStep.yaml
type: string
enum:
- agent-setup
- add-project
- project-added
- start-from-prompt
- telemetry-notice
description: Web onboarding step
3 changes: 3 additions & 0 deletions apis/json-schema/Settings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,9 @@ properties:
harness:
$ref: HarnessConfig.yaml
description: Query-aware agent harness configuration (optional, defaults applied at runtime)
telemetry:
$ref: TelemetryConfig.yaml
description: Usage metrics preferences (optional, defaults applied at runtime)
required:
- models
- user
Expand Down
32 changes: 32 additions & 0 deletions apis/json-schema/TelemetryConfig.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
$schema: https://json-schema.org/draft/2020-12/schema
$id: TelemetryConfig.yaml
type: object
properties:
enabled:
type: boolean
default: true
description: Whether usage metrics are recorded and sent
includeIdentity:
type: boolean
default: false
description: Whether identity (agent account hash, GitHub username and owners) is attached; opt-in
contactConsent:
type: boolean
default: false
description: Whether the Shep team may contact the user on GitHub
installId:
type: string
description: Random install identifier (UUID), assigned on first use
noticeShownAt:
type: string
format: date-time
description: When the first-run notice was shown
lastHeartbeatAt:
type: string
format: date-time
description: When the last install.heartbeat event was recorded
required:
- enabled
- includeIdentity
- contactConsent
description: Opt-out usage metrics preferences (spec 133)
10 changes: 10 additions & 0 deletions apis/json-schema/TelemetryDisabledReason.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
$schema: https://json-schema.org/draft/2020-12/schema
$id: TelemetryDisabledReason.yaml
type: string
enum:
- ci
- do-not-track
- env-disabled
- test
- user-opt-out
description: Reason telemetry is disabled
15 changes: 15 additions & 0 deletions apis/json-schema/TelemetryEvent.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
$schema: https://json-schema.org/draft/2020-12/schema
$id: TelemetryEvent.yaml
type: string
enum:
- install.heartbeat
- cli.command
- web.area.viewed
- feature.created
- feature.run.finished
- pr.opened
- pr.merged
- decision.answered
- onboarding.step
- error.unhandled
description: Usage event recorded by Shep's opt-out telemetry
8 changes: 8 additions & 0 deletions apis/json-schema/TelemetryProcessKind.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
$schema: https://json-schema.org/draft/2020-12/schema
$id: TelemetryProcessKind.yaml
type: string
enum:
- cli
- daemon
- worker
description: Process that recorded a telemetry event
115 changes: 115 additions & 0 deletions docs/telemetry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Usage Metrics (Telemetry)

Shep sends a small set of **usage events** to the Shep team so we can tell which parts of Shep
are used and whether runs end in merged pull requests. It is on by default, announced the first
time you run Shep, and easy to turn off. Fields that identify you are **opt-in**: they are sent
only after you turn on *Include my identity*. This page lists everything that is sent.

Spec: [`specs/133-telemetry`](../specs/133-telemetry/spec.md).

## Turning it off

| How | Effect |
| --- | ------ |
| `shep telemetry off` | Off; events not yet sent are deleted |
| Settings → Usage metrics → *Send usage metrics* | Same as above |
| Web onboarding → *Turn off* | Same as above |
| `DO_NOT_TRACK=1` or `SHEP_TELEMETRY_DISABLED=1` | Off for every process that sees the variable |
| `CI` set (any value except `0`/`false`) | Always off |
| Running under a test runner (`VITEST`, `NODE_ENV=test`) | Always off |

Identity is off by default. `shep telemetry identity on` (or *Include my identity* in Settings or
the onboarding notice) adds the identity fields below; `shep telemetry identity off` removes them
again, including from events already queued.
`shep telemetry contact on|off` sets whether the Shep team may contact you on GitHub.
`shep telemetry status` shows the current state; `shep telemetry show` prints the exact request
body the next send would post (with the project key replaced by a placeholder).

## What is sent

Every event carries:

| Field | Example | Notes |
| ----- | ------- | ----- |
| `distinct_id` | random UUID | The install id, created on first run. Never derived from an account. |
| `uuid` | random UUID | Per event; retries reuse it so the backend can drop copies. |
| `process` | `cli` / `daemon` / `worker` | Which Shep process recorded it. |
| `shepVersion`, `os`, `arch`, `nodeVersion` | `1.236.0`, `darwin`, `arm64`, `22.11.0` | |
| `contactConsent` | `false` | Whether the Shep team may contact you on GitHub (off unless you allow it). |
| `$geoip_disable` | `true` | PostHog's GeoIP enrichment is always off. |

Only after you turn **Include my identity** on (it is off by default), events also carry:

| Field | Source |
| ----- | ------ |
| `agentAccountHash`, `agentAccountSource` | SHA-256 of `<agentType>:<accountId>`, read from the agent's own login file through its catalog entry (Claude: `~/.claude.json` `userID`, honouring `CLAUDE_CONFIG_DIR`; Codex: `~/.codex/auth.json` `tokens.account_id`, honouring `CODEX_HOME`). The raw id never leaves the reader. |
| `githubUsername` | `gh api user` |
| `githubOwners` | The GitHub owners (users or organisations) parsed from the remotes of repositories added to Shep. Repository names are dropped. |

With identity on, PostHog creates a person profile for the install with `githubUsername`,
`githubOwners` and `contactConsent`; with identity off (the default), person profiles are not
processed.

### Events

| Event | Properties | Recorded by |
| ----- | ---------- | ----------- |
| `install.heartbeat` | `agentType`, `enabledFeatureFlags` (flag names), `repositoryCount`, `activeFeatureCount` | Daemon, at most once a day |
| `cli.command` | `command` — the command path such as `feat new`, never arguments | CLI |
| `web.area.viewed` | `area` (`/aspm`), `route` — a route template such as `/feature/[featureId]` | Web UI |
| `feature.created` | `buildMode`, `agentType` | `CreateFeatureUseCase` |
| `feature.run.finished` | `status`, `agentType`, `duration` (bucket: `under-1m` … `over-4h`) | Feature-agent worker |
| `pr.opened` | `buildMode` | Merge node (once per feature) |
| `pr.merged` | `buildMode`, `viaPullRequest` | Merge node, PR sync, GitHub webhook (once per feature) |
| `decision.answered` | `kind`, `surface`, `latency` (bucket), `pickedRecommended` | Defined for the unified-decisions work |
| `onboarding.step` | `step`, `completed` | Web onboarding |
| `error.unhandled` | `errorClass`, `sourceHash` (truncated SHA-256 of the top stack frame's `function@file:line`) | CLI, daemon, worker crash handlers |

**Never sent:** prompts, code, file paths, repository or branch names, feature titles, local ids,
or error messages. Property types are declared per event (`TelemetryEventPropertyMap` in
`packages/core/src/application/ports/output/services/telemetry-events.ts`), so a content property
does not compile.

The north-star metric is **merged PRs per weekly-active install**.

## How it is delivered

```
any process ── ITelemetry.record() ──► SQLite telemetry_outbox ──► daemon / shep ui flush (60s)
(drops when off) (uuid, attempts, cap 5000) │
├─ off? → delete everything queued
├─ no key? → wait
└─ POST https://eu.i.posthog.com/batch/
```

- Every process writes to the local `telemetry_outbox` table; only the long-lived server
process (the `shep start` daemon or `shep ui`) sends.
- Batches of 20 are claimed with a two-minute lease, so two server processes never send the same
events.
- A failed batch backs off with jitter (2 s doubling to 5 min) and is dropped after 5 attempts.
- The outbox keeps at most 5,000 events.
- Identity is attached when a batch is sent, so turning identity off also covers events queued
before.
- Plain `fetch`, no SDK.

## Configuration (maintainers)

| Setting | Where |
| ------- | ----- |
| Project key | `SHEP_POSTHOG_KEY`, else `BUILT_IN_POSTHOG_KEY` in `packages/core/src/infrastructure/services/telemetry/telemetry-config.ts` (empty in source: no key, nothing is sent) |
| Host | `SHEP_POSTHOG_HOST`, default `https://eu.i.posthog.com` |

In the PostHog project, also turn on **Discard client IP data**, since the capture endpoint
otherwise records the sender's IP address.

## Code map

| Layer | Files |
| ----- | ----- |
| TypeSpec | `tsp/common/enums/telemetry.tsp`, `TelemetryConfig` in `tsp/domain/entities/settings.tsp` |
| Domain | `packages/core/src/domain/shared/telemetry/` |
| Ports | `ITelemetry`, `ITelemetryTransport`, `ITelemetryIdentityProvider`, `ITelemetryRuntime`, `ITelemetryOutboxRepository` |
| Use cases | `packages/core/src/application/use-cases/telemetry/` |
| Adapters | `packages/core/src/infrastructure/services/telemetry/`, migration `168-create-telemetry.ts` |
| CLI | `src/presentation/cli/commands/telemetry/`, `src/presentation/cli/telemetry-hooks.ts` |
| Web | `src/presentation/web/components/features/telemetry/`, `app/actions/telemetry.ts`, `hooks/use-route-view-telemetry.ts` |
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,8 @@ export type { IWorkflowExecutionRepository } from './workflow-execution-reposito
export type { IPluginRepository } from './plugin-repository.interface.js';
export type { IDevServerRunPlanRepository } from './dev-server-run-plan-repository.interface.js';
export type { IFleetRepository, FleetTriageFilters } from './fleet-repository.interface.js';
export type {
ITelemetryOutboxRepository,
TelemetryOutboxEntry,
EnqueueTelemetryOptions,
} from './telemetry-outbox.repository.interface.js';
Loading
Loading