feat(stacks): manage the spec mirrors and the monorepo's own credentials - #445
Open
Mkassabov wants to merge 16 commits into
Open
feat(stacks): manage the spec mirrors and the monorepo's own credentials#445Mkassabov wants to merge 16 commits into
Mkassabov wants to merge 16 commits into
Conversation
Mkassabov
changed the title
Adds stacks/github, an Alchemy stack that manages one alchemy-run/distilled-<name> repository per packages/* SDK (23 today: 22 SDKs + core), discovered from the filesystem at deploy time. A new deploy-github-stack workflow deploys it on every push to main (usually a no-op) using the ALCHEMY_GITHUB_TOKEN PAT secret. The stack is standalone (own bun.lock, not a workspace member) because alchemy@2.0.0-beta.70 requires effect 4.0.0-beta.103 while the monorepo catalog hoists beta.105.
Swap localState() for Cloudflare.state() — the same remote state store the alchemy monorepo uses — so local and CI deploys converge one shared stack. The workflow now also passes CLOUDFLARE_API_TOKEN / CLOUDFLARE_ACCOUNT_ID and deploys with --yes so state-store upgrade prompts can't hang CI. The target org is now the DISTILLED_REPOS_OWNER repository variable (falling back to alchemy-run) ahead of the move to a dedicated snapshot org.
Latest main release. Kills the effect version skew: beta.70 was built against the pre-rename Schema API (effect beta.103) while its @distilled.cloud deps floated to beta.105.
A stale bun manifest cache resolved next to the broken beta.70 (built against pre-rename effect Schema API); force-refreshed so the lockfile pins beta.72 + effect beta.105.
Follows the alchemy CI/CD tutorial: deployed once with an admin profile, it mints a Cloudflare API token scoped to the state-store worker and seeds all four CI values (CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, ALCHEMY_GITHUB_TOKEN, DISTILLED_REPOS_OWNER) into this repo's Actions configuration as code.
beta.72 pinned effect at the beta.105 line; beta.73 moved the peer range to `>=4.0.0-rc.110`, so the lockfile moves with it to effect 4.0.0-rc.112 (the current rc) and the matching @effect/platform-* packages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Retargets the stack at the spec mirrors it actually deploys — `distilled-mirror/spec-mirror-<pkg>`, one per spec-consuming `packages/*` directory — and makes it own what is inside them, not just that they exist. Why: several spec submodules are enormous relative to what we read from them. github/rest-api-description is ~6.7 GB checked out for one 13 MB document, kubernetes/kubernetes ~1.5 GB for one 4.3 MB document, and Azure/azure-rest-api-specs is 338,547 paths of which the converter reads about 780. `specs:sync` at the repo root materialises every one of them in full. A mirror holds the subset, so submoduling it costs what the specs cost. - Scaffold.ts defines `SyncScaffold`, an Alchemy Action that commits a repository's file set through the git data API. It is authoritative in both directions: it writes the managed files and deletes anything else under a managed prefix, plus any case-variant of a managed path — which is what removes the `README.md` GitHub writes for an `autoInit` repository next to the scaffold's `readme.md`. `specs/` is deliberately unmanaged (it is the payload each mirror's own workflow produces) and `.meta/bun.lock` is preserved. Unchanged files are detected by comparing git blob ids, so a converged repository costs no write calls at all. - SpecRepos.ts is the manifest, and alchemy.run.ts fails the deploy if a package has a `specs/` directory but no mirror, so a new SDK cannot quietly stay on a multi-gigabyte upstream submodule. - spec-repos/<pkg>/ holds each mirror's own fetch script. There is no generic one: the upstreams are a plain URL fetch, a GraphQL introspection, a discovery-directory crawl, a raw file download and a sparse partial clone. The nine that already worked are vendored unchanged. - aws and azure need directories rather than files, so both clone with `--filter=blob:none --no-checkout`, decide what to keep from the tree listing alone, then narrow the sparse checkout to exactly those paths — content is fetched last, so the multi-gigabyte working tree never exists. azure additionally picks the latest stable api-version per resource provider and closes the selection over external `$ref`s, skipping `x-ms-examples` (~16k files no step of the pipeline reads): 338,547 paths in, 777 out. - cloudflare is managed but unscaffolded. developers.cloudflare.com migrated to Astro/Starlight and now serves the full HTML page at every `<page>/index.md` URL, including the ones its own /api/llms.txt advertises as markdown, so there is nothing correct to mirror yet. Reported upstream; packages/cloudflare/scripts/download-api-docs.ts has the same problem. Verified: all 21 fetch scripts run, and every mirror's `update-specs` workflow is green in CI with specs committed. A forced redeploy reports every scaffold already up to date, so the Action is idempotent and self-healing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sion The spec-submodule entries described a problem that stacks/github now largely solves, so they say what is left (rewiring .gitmodules and the generator spec paths onto the mirrors) rather than restating the options. Adds the cloudflare finding: developers.cloudflare.com no longer serves markdown at <page>/index.md, so download-api-docs.ts silently downloads HTML and packages/cloudflare/specs is a stale snapshot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`stacks/github` is its own install root — its own bun.lock, its own alchemy/effect versions — so it is not a workspace member and the existing `typecheck` job, which runs `tsc -b` over the root project references, never sees it. Nothing enforced it; `format:check` was the only gate it passed through. Add a `typecheck-stacks` job that installs and checks it in place, and split the mirror fetch scripts into `tsconfig.spec-repos.json` so they are checked too. Those ship verbatim into the mirror repositories and run there under Bun, so a break in one would otherwise stay invisible until that mirror's nightly workflow ran a day later. Checking them here needs `graphql` and `yaml` resolvable, hence the two new devDependencies — keep them in step with the mirror package.json files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`stacks/github` was two unrelated things in one directory: the stack that manages the spec mirrors, and a one-shot stack that provisions the credentials CI needs to deploy it. Split them, and name them after what they own rather than which provider they happen to use. - `stacks/github` -> `stacks/distilled-submodules`, and the `Alchemy.Stack` name with it. The stack name keys its state, so this needs a destroy of the old `distilled-github` stack before the first deploy under the new name, or the old state is orphaned in the store. Safe to destroy: `GitHub.Repository` defaults to `retain` (destroying the stack does not delete a repository, and nothing here opts into `destroy()`), and removing an Action drops its state without invoking the body, so the mirrors and their specs are untouched. - `credentials.run.ts` -> `stacks/distilled-monorepo`, following https://alchemy.run/environments/ci/#the-github-stack. It now also manages `alchemy-run/distilled` itself, so the repository settings are reviewable and diffable instead of clicked into the UI. Every value mirrors the live repository, so the first deploy is a no-op. Two fixes fall out of writing it down: - The state-store token was missing `Secrets Store Write`. `Cloudflare.state()` reads the worker's bearer token back by BINDING it to an edge-preview worker, and binding is a write - `Read` alone would have failed every CI deploy. - The secrets are written as `STACKS_CLOUDFLARE_*`, not `CLOUDFLARE_API_TOKEN` / `CLOUDFLARE_ACCOUNT_ID`. Those names already exist on this repository as long-lived, broadly-scoped credentials belonging to other tooling; writing them here would have silently replaced them with a token scoped to the state store alone. The prefix follows the existing `WEBSITE_CLOUDFLARE_*` convention. `deploy-github-stack.yml` is renamed to `deploy-submodules-stack.yml`, and the CI typecheck job becomes a matrix so every `stacks/*` directory is checked rather than a hardcoded one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
main migrated the monorepo from bun to pnpm; the stacks were still carrying `bun.lock` files and `bun alchemy` commands. They were kept out of the workspace because a shared bun install hoists one copy of `effect`, and the alchemy CLI is built against a different prerelease than the SDK packages track. pnpm resolves each project's dependencies in isolation, so that reason is gone: `stacks/*` join `pnpm-workspace.yaml` and one `pnpm install` covers everything. The alternative — standalone install roots — needs `--ignore-workspace` on every invocation (the `.npmrc` key is not honoured; pnpm walks up, finds the workspace root and installs all 27 projects instead) plus per-stack build-approval config. Membership is the simpler shape. - exact `alchemy`/`effect` pins stay exact rather than moving to `catalog:effect`; `@types/bun` and `typescript` now take the root catalogs, since nothing about them is stack-specific. - `workerd` added to `allowBuilds` — alchemy pulls in miniflare for the Cloudflare state store. - the deploy workflow installs with `--filter "./stacks/distilled-submodules..."` so a deploy does not pull 20 SDK packages and their generators. - `typecheck-stacks` becomes a recursive filter instead of a matrix, so a new stack is covered the day it is added, and picks up main's runner and action pins. The mirror repositories keep their own bun tooling: they are separate repositories with their own package.json, and nothing about this monorepo's package manager applies to them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rcel main added three spec-consuming packages while this branch was behind. The coverage guard caught them on the first deploy after the rebase and refused to run, which is what it is for — silently skipping a package would leave it on its upstream fetch forever. All three are the simple case: one published OpenAPI document at a fixed URL, no git repo and no versioned tags, so the mirror snapshots it rather than sparse-cloning anything. hetzner https://docs.hetzner.cloud/cloud.spec.json -> cloud.spec.json huggingface https://huggingface.co/.well-known/openapi.json -> openapi.json vercel https://openapi.vercel.sh/ -> openapi.json Each fetch script re-serializes with a 2-space indent and a trailing newline, matching the package's own `scripts/download-spec.ts`, so a whitespace-only change upstream produces no diff. Each also asserts `openapi`/`paths` before writing: a login page or a gutted response is still valid JSON, and failing at fetch time is far cheaper than failing three steps later inside the generator. Verified by running all three against upstream. hetzner's output is BYTE-IDENTICAL to the spec committed in packages/hetzner/specs, which confirms the serialization matches. huggingface and vercel are larger than their committed snapshots because upstream has moved since — the reason the mirrors refresh daily. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ository Every scaffold failed on the first from-scratch deploy: the Actions ran concurrently with the repositories they were trying to commit into, so `getRef` 404'd on a repository that did not exist yet. An Action's upstream is derived from the Outputs its input references — `Plan.ts:987` puts it plainly: "Actions have no bindings — their upstream is input + init captures". The input here was built from plain strings (`owner`, the computed mirror name, a hardcoded "main"), so the Action referenced no Output, had no upstream, and the scheduler was free to run all 24 scaffolds alongside all 25 creates. Feed it the repository resource's Outputs instead: `fullName` replaces the separate owner/repository fields, and `branch` becomes the resolved `defaultBranch` rather than a hardcoded "main" — both real edges, and `defaultBranch` is the more correct value anyway. This stayed hidden through every earlier deploy because the repositories already existed by then; only a genuinely empty org exposes the race. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`destroy-legacy.run.ts` existed only to hold the stack's previous name (`distilled-github`) so `alchemy destroy` could find state the renamed `alchemy.run.ts` no longer described. That teardown has been run, the stack has been redeployed under `distilled-submodules`, and all 25 mirrors are live with their specs — so the shim has nothing left to point at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mkassabov
changed the title
Mkassabov
force-pushed
the
claude/alchemy-github-repos-stack-dab787
branch
from
August 26, 2026 16:08
28321dc to
b93f2fc
Compare
Contributor
|
Install the packages built from this commit: Distilled@distilled.cloud/core bun add https://pkg.ing/@distilled.cloud/core/b93f2fc@distilled.cloud/cloudflare bun add https://pkg.ing/@distilled.cloud/cloudflare/b93f2fc@distilled.cloud/railway bun add https://pkg.ing/@distilled.cloud/railway/b93f2fc |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds two Alchemy stacks under
stacks/:distilled-submodules— owns the spec mirrors: onedistilled-mirror/spec-mirror-<package>repository per SDK, each holding exactly the upstream API-spec files its generator reads, refreshed daily, and nothing else.distilled-monorepo— owns this repository and the credentials its CI runs on, following the GitHub stack pattern.The problem is submodule size.
packages/github/specsis a ~6.7 GB checkout ofgithub/rest-api-descriptionfor one file;packages/azure/specsis tens of gigabytes ofAzure/azure-rest-api-specsfor 777 documents.specs:syncmaterialises every one of them in full.Rewiring
.gitmodulesand each package'sspecs:fetchonto the mirrors is a follow-up — nothing here changes how the monorepo builds.Deployed and verified live
All 25 mirrors exist in
distilled-mirror, scaffolded and populated. 24update-specsruns dispatched, 24/24 green:googleapisaws-sdk-js-v3modelsThe rest are single documents.
cloudflaregets a repository but no fetch machinery — see below.How
GitHub.Repositoryper entry in theSPEC_REPOSmanifest. AcheckCoverageguard fails the deploy if apackages/<x>/specsdirectory exists with no manifest entry, so a new SDK cannot silently skip its mirror. It earned its keep immediately: rebasing onto main surfacedhetzner,huggingfaceandvercel, and the deploy refused to run until they had mirrors.SyncScaffold, an Alchemy Action, writes each mirror's six managed files as one commit through the git data API:.github/and.meta/, plus case-variants of managed files (autoInitleaves aREADME.mdbeside ourreadme.md).specs/and.meta/bun.lockare written by the mirror's own workflow and never touched.Idempotency, since Actions are the new part: a plain redeploy reports
Done: 0 succeededwith zero API writes, and--force— which re-runs every body against repos that now containspecs/— reports "already up to date" for all of them with HEAD SHAs byte-identical before and after.CI
stacks/*are workspace members, but are absent from the root tsconfig's project references, sotsc -bnever sees them. Atypecheck-stacksjob covers them with a recursive filter (pnpm --filter "./stacks/*" run typecheck) rather than a matrix, so a new stack is checked the day it is added — including the mirror fetch scripts, where a break would otherwise stay invisible until that mirror's nightly run.deploy-submodules-stack.ymldeploys the submodules stack on every push tomain.Credentials
distilled-monorepomints a Cloudflare API token as code and writes it, with the mirror-org PAT, straight into this repo's Actions secrets — nothing pasted by hand, and the raw token never reaches a terminal.Written as
STACKS_CLOUDFLARE_API_TOKEN/STACKS_CLOUDFLARE_ACCOUNT_ID, notCLOUDFLARE_API_TOKEN/CLOUDFLARE_ACCOUNT_ID: those names already exist here as long-lived, broadly-scoped credentials belonging to other tooling, and writing them would have replaced them with a token scoped to the state store alone. The prefix follows the existingWEBSITE_CLOUDFLARE_*convention.The token needs
Workers Scripts Write,Account Settings Write, andSecrets Store Write— the last becauseCloudflare.state()reads the state-store worker's bearer token back by binding it to an edge-preview worker, and binding is a write.Readalone fails every CI deploy.Remaining before merge
distilled-monorepois deployed by hand, never from CI. Four of its six resources are live;state-store-tokenandcf-api-tokenstill need a Cloudflare Super Administrator — the Administrator role excludes API token management, and Cloudflare will not let a credential grant a permission it does not itself hold.Until that lands,
deploy-submodules-stack.ymlwill fail on merge tomainfor want ofSTACKS_CLOUDFLARE_API_TOKEN.Known gap: cloudflare
developers.cloudflare.commigrated to Astro/Starlight and now returns the full HTML page at every<page>/index.mdURL, including the ones its own/api/llms.txtadvertises as markdown. A naive fetch yields 7.2 GB of HTML.packages/cloudflare/scripts/download-api-docs.tshas the same problem today. The repository is still managed; its manifest entry is markedblockedso the scaffold is skipped, and unblocking is a one-field change once upstream is fixed or the generator moves tocloudflare/api-schemas.