-
Notifications
You must be signed in to change notification settings - Fork 0
fix: CSP Clerk-origin drift + Production Smoke failure paging (linejam-912, linejam-913) #300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
d430a4e
9e17782
d506d53
c321937
b08bb6f
7ee876d
5e4a5bf
33e6205
65de13f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -99,3 +99,23 @@ follow-up artifact is intentionally required for an incident or release | |||||
| decision. | ||||||
|
|
||||||
| Canary itself treats generic signed webhooks as the stable product contract, so the responder should stay thin: verify, fetch context, store evidence, trigger the smoke harness, then hand off to follow-on agents. | ||||||
|
|
||||||
| ## Production Smoke Failure Wire (linejam-913) | ||||||
|
|
||||||
| The `Production Smoke` workflow (`.github/workflows/prod-smoke.yml`) runs hourly and, before 2026-07-04, a red run just sat in the Actions tab: it failed hourly for ~15 hours before the operator found the outage by hand. The gate was working; nothing wired the red signal to a human or to BB triage. | ||||||
|
|
||||||
| Two scripts close that wire, running as the last steps of the job regardless of outcome (`if: always()`): | ||||||
|
|
||||||
| - `scripts/ops/count-consecutive-prod-smoke-failures.mjs` walks the workflow's recent completed-run history (via the GitHub REST API, `GITHUB_TOKEN`) to compute the consecutive-failure streak ending at the current run. A single blip does not escalate; only a genuine repeat does. On an API error it fails OPEN toward escalation rather than silence. | ||||||
| - `scripts/ops/report-prod-smoke-status.mjs` reports the outcome to the `linejam-production-smoke` Canary TTL monitor (`expected_every_ms=3600000`, `grace_ms=1800000`, created via `POST /api/v1/monitors`) via `POST /api/v1/check-ins`: | ||||||
| - Success → `status: "ok"` (Canary's Up health state; resolves any open incident). | ||||||
| - Failure, streak < 2 → `status: "alive"` (still Up; recorded in the monitor's check-in history and the run's step summary, but does not open an incident). | ||||||
| - Failure, streak >= 2 → `status: "error"` (Canary maps `error` check-ins directly to its Down health state, which opens/holds a `health_transition` incident that BB triage and the bridge feed already consume — no linejam-side webhook or bridge-specific code needed). | ||||||
|
|
||||||
| Reuses the existing `NEXT_PUBLIC_CANARY_API_KEY`/`NEXT_PUBLIC_CANARY_ENDPOINT` repository secrets (already ingest-scoped for client-side error reporting); no new secret was provisioned. | ||||||
|
|
||||||
| Live-verified end to end against the deployed monitor: a first simulated failure stayed `up`/`alive`; a second consecutive simulated failure flipped the monitor to `down` and opened `INC-bhg3g284flhp` (`signal_type: health_transition`); a simulated recovery run immediately resolved it. Query current state any time with: | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Minor: hyphenate "end-to-end." LanguageTool flags this as a compound-modifier hyphenation opportunity. ✏️ Proposed fix-Live-verified end to end against the deployed monitor: a first simulated failure stayed `up`/`alive`; a second consecutive simulated failure flipped the monitor to `down` and opened `INC-bhg3g284flhp` (`signal_type: health_transition`); a simulated recovery run immediately resolved it. Query current state any time with:
+Live-verified end-to-end against the deployed monitor: a first simulated failure stayed `up`/`alive`; a second consecutive simulated failure flipped the monitor to `down` and opened `INC-bhg3g284flhp` (`signal_type: health_transition`); a simulated recovery run immediately resolved it. Query current state any time with:📝 Committable suggestion
Suggested change
🧰 Tools🪛 LanguageTool[grammar] ~117-~117: Use a hyphen to join words. (QB_NEW_EN_HYPHEN) 🤖 Prompt for AI AgentsSource: Linters/SAST tools |
||||||
|
|
||||||
| ```bash | ||||||
| curl -fsS "$CANARY_ENDPOINT/api/v1/report?window=24h" -H "Authorization: Bearer $CANARY_API_KEY" | jq '.monitors[] | select(.name=="linejam-production-smoke")' | ||||||
| ``` | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| /** | ||
| * Derive the Clerk Frontend API origin encoded in a publishable key. | ||
| * | ||
| * Clerk publishable keys are `pk_(test|live)_<base64url(frontendApiHost + "$")>`. | ||
| * Decoding the trailing segment recovers the exact host Clerk serves clerk-js | ||
| * and its Frontend API from. For a custom domain (e.g. `clerk.linejam.app`) | ||
| * this is the only source of truth for that host — nothing else in env | ||
| * encodes it. | ||
| * | ||
| * This is the canonical implementation. `next.config.ts` (CSP allowlist) and | ||
| * `scripts/ci/bootstrap-convex-env.mjs` (Convex JWT issuer domain) both | ||
| * derive from it, so a Clerk domain change can never silently diverge | ||
| * between the two again (2026-07-04 outage: CSP hand-listed domains and | ||
| * missed the production custom domain, blocking auth site-wide for ~16h | ||
| * while preview smoke stayed green on the dev Clerk domain). | ||
| * | ||
| * @param {string | undefined | null} publishableKey | ||
| * @returns {string} the origin (e.g. "https://clerk.linejam.app"), or '' if | ||
| * it cannot be derived (missing/malformed key). | ||
| */ | ||
| export function deriveClerkFrontendOrigin(publishableKey) { | ||
| const key = publishableKey?.trim() ?? ''; | ||
| if (!key) return ''; | ||
|
|
||
| const encodedDomain = key.split('_').at(-1); | ||
| if (!encodedDomain) return ''; | ||
|
Comment on lines
+33
to
+34
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
node -e '
const hosts = ["clerk.linejam.app","auth.example-tenant.com","great-moose-1.clerk.accounts.dev"];
// brute-force a few synthetic hosts to prove _ is reachable in base64url output
for (let i=0;i<5000;i++){ hosts.push("clerk-"+i+".linejam.app"); }
let hit=null;
for (const h of hosts){
const enc = Buffer.from(h+"$").toString("base64url");
if (enc.includes("_")){ hit={h,enc}; break; }
}
console.log("example host whose key segment contains _ :", hit);
if (hit){
const key = "pk_live_"+hit.enc;
const tail = key.split("_").at(-1);
console.log("split-based decode:", Buffer.from(tail,"base64url").toString("utf8"));
console.log("prefix-strip decode:", Buffer.from(hit.enc,"base64url").toString("utf8"));
}
'Repository: misty-step/linejam Length of output: 205 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '\n== scripts/lib/clerk-domain.mjs ==\n'
sed -n '1,220p' scripts/lib/clerk-domain.mjs
printf '\n== related references ==\n'
rg -n "deriveClerkFrontendOrigin|clerk-domain|frontendApiHost|Convex JWT issuer|bootstrap-convex-env|next.config" scripts . -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/build/**' || true
printf '\n== tests around clerk domain ==\n'
rg -n "clerk.*domain|frontendApiHost|base64url|pk_(test|live)_" scripts test tests . -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/build/**' || trueRepository: misty-step/linejam Length of output: 14127 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '\n== file inventory ==\n'
git ls-files | rg '^scripts/lib/clerk-domain\.mjs$|^scripts/.*(bootstrap-convex-env|next\.config|clerk).*|(^|/)(test|tests)/.*clerk|.*clerk.*test.*'
printf '\n== source around the target ==\n'
nl -ba scripts/lib/clerk-domain.mjs | sed -n '1,120p'Repository: misty-step/linejam Length of output: 473 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Inspect the target file and nearby callers/tests without executing repository code.
printf '\n== scripts/lib/clerk-domain.mjs ==\n'
nl -ba scripts/lib/clerk-domain.mjs | sed -n '1,200p'
printf '\n== references to the helper and related derivation ==\n'
rg -n "deriveClerkFrontendOrigin|clerk-domain|frontendApiHost|base64url|pk_(test|live)_" scripts . \
-g '!**/node_modules/**' -g '!**/dist/**' -g '!**/build/**' || true
printf '\n== test files referencing clerk domain ==\n'
rg -n "clerk.*domain|frontendApiHost|base64url|pk_(test|live)_" tests test scripts \
-g '!**/node_modules/**' -g '!**/dist/**' -g '!**/build/**' || trueRepository: misty-step/linejam Length of output: 232 🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
import base64
import itertools
import string
alphabet = string.ascii_lowercase + string.digits + "-."
# Search short, syntactically plausible host labels first.
def enc(s: str) -> str:
return base64.urlsafe_b64encode(s.encode()).decode().rstrip("=")
for n in range(1, 7):
for s in itertools.product(alphabet, repeat=n):
host = "".join(s)
e = enc(host)
if "_" in e:
print("found_host=", host)
print("encoded=", e)
key = "pk_live_" + e
tail = key.split("_")[-1]
print("split_tail=", tail)
print("split_decoded=", base64.urlsafe_b64decode(tail + "==").decode("utf-8", "replace"))
print("full_decoded=", base64.urlsafe_b64decode(e + "==").decode("utf-8", "replace"))
raise SystemExit(0)
print("no host found in search space")
PYRepository: misty-step/linejam Length of output: 148
🤖 Prompt for AI Agents |
||
|
|
||
| try { | ||
| const decoded = Buffer.from(encodedDomain, 'base64url') | ||
| .toString('utf8') | ||
| .replace(/\$+$/, ''); | ||
|
|
||
| if (!decoded) return ''; | ||
|
|
||
| return decoded.startsWith('https://') ? decoded : `https://${decoded}`; | ||
| } catch { | ||
| return ''; | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| #!/usr/bin/env node | ||
| import { pathToFileURL } from 'node:url'; | ||
|
|
||
| const WORKFLOW_FILE = 'prod-smoke.yml'; | ||
|
|
||
| /** | ||
| * Count the consecutive failure streak ending at the current run. | ||
| * | ||
| * `priorConclusions` is the conclusions of earlier completed runs of the | ||
| * same workflow, ordered most-recent-first, excluding the current run. | ||
| * Runs whose conclusion is neither `success` nor `failure` (cancelled, | ||
| * skipped, stale, action_required, ...) are ignored — they carry no signal | ||
| * about whether the underlying check passed — rather than breaking or | ||
| * extending the streak. | ||
| * | ||
| * @param {'success' | 'failure'} currentOutcome | ||
| * @param {Array<string | null>} priorConclusions | ||
| * @returns {number} | ||
| */ | ||
| export function countConsecutiveFailures(currentOutcome, priorConclusions) { | ||
| if (currentOutcome !== 'failure') return 0; | ||
|
|
||
| let streak = 1; | ||
| for (const conclusion of priorConclusions) { | ||
| if (conclusion === 'failure') { | ||
| streak += 1; | ||
| continue; | ||
| } | ||
| if (conclusion === 'success') break; | ||
| // else: ignore and keep looking further back | ||
| } | ||
| return streak; | ||
| } | ||
|
|
||
| /** | ||
| * @param {{ | ||
| * owner: string, | ||
| * repo: string, | ||
| * excludeRunId: string | number, | ||
| * perPage?: number, | ||
| * token: string, | ||
| * fetchImpl?: typeof fetch, | ||
| * }} params | ||
| * @returns {Promise<Array<string | null>>} | ||
| */ | ||
| export async function fetchPriorRunConclusions({ | ||
| owner, | ||
| repo, | ||
| excludeRunId, | ||
| perPage = 10, | ||
| token, | ||
| fetchImpl = globalThis.fetch, | ||
| }) { | ||
| const url = `https://api.github.com/repos/${owner}/${repo}/actions/workflows/${WORKFLOW_FILE}/runs?status=completed&per_page=${perPage}`; | ||
| const response = await fetchImpl(url, { | ||
| headers: { | ||
| Authorization: `Bearer ${token}`, | ||
| Accept: 'application/vnd.github+json', | ||
| 'X-GitHub-Api-Version': '2022-11-28', | ||
| }, | ||
| }); | ||
|
|
||
| if (!response.ok) { | ||
| throw new Error( | ||
| `Failed to list ${WORKFLOW_FILE} runs: HTTP ${response.status}` | ||
| ); | ||
| } | ||
|
|
||
| const body = await response.json(); | ||
| return (body.workflow_runs ?? []) | ||
| .filter((run) => String(run.id) !== String(excludeRunId)) | ||
| .sort((a, b) => new Date(b.created_at) - new Date(a.created_at)) | ||
| .map((run) => run.conclusion); | ||
| } | ||
|
|
||
| if (import.meta.url === pathToFileURL(process.argv[1]).href) { | ||
| const currentOutcome = process.argv[2]; | ||
| const [owner, repo] = (process.env.GITHUB_REPOSITORY || '').split('/'); | ||
|
|
||
| fetchPriorRunConclusions({ | ||
| owner, | ||
| repo, | ||
| excludeRunId: process.env.GITHUB_RUN_ID, | ||
| token: process.env.GITHUB_TOKEN, | ||
| }) | ||
| .then((conclusions) => { | ||
| console.log(countConsecutiveFailures(currentOutcome, conclusions)); | ||
| }) | ||
| .catch((error) => { | ||
| console.error(error instanceof Error ? error.message : String(error)); | ||
| // Fail OPEN toward escalation, never toward silence: the 2026-07-04 | ||
| // outage was caused by a red signal nobody saw for ~15 hours, not by | ||
| // an over-eager page. If we cannot read history on a failing run, | ||
| // assume the threshold is already met rather than assuming zero. | ||
| console.log(currentOutcome === 'failure' ? 2 : 0); | ||
| }); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: misty-step/linejam
Length of output: 5521
🏁 Script executed:
Repository: misty-step/linejam
Length of output: 837
Use a unique
GITHUB_OUTPUTdelimiter here. A fixedSMOKE_DETAIL_EOFmarker can appear in the captured stderr and truncate the multiline value, letting extra text be parsed as new outputs. Generate a per-run delimiter instead.🤖 Prompt for AI Agents