Sync branch ci rollout - #527
Open
mcocdawc wants to merge 13 commits into
Open
Conversation
Migrates eccodes onto the manifest-driven CI in ecmwf/ci-infrastructure. The
legacy ecmwf/downstream-ci pipeline keeps running unchanged, moved to
old_CI.yml (name: old CI) so the new one can take the plain `CI` name that the
generated downstream orchestrators match their `workflow_run` trigger against.
The eckit edge is the interesting part. eccodes needs eckit only for FEATURE
ECKIT_GEO, which is DEFAULT OFF, so declaring it as an ordinary dependency
would charge every leg for it: eckit would enter each leg's deps-hash8, and
every eckit commit would invalidate all eccodes artifacts and force full
rebuilds of builds that never link it. Instead the [[deps]] entry carries
`when = { options = ["eckit-geo"] }`, so eckit applies only to the leg that
enables the feature. Verified offline: the plain legs resolve to exactly the
artifact names they would have if eckit were not declared at all.
`options` is part of artifact identity, so the geo leg publishes under an
opts.eckit-geo segment and cannot collide with the plain g++-13 leg. The build
action maps that value to the CMake flags and rejects any value it does not
know, rather than quietly building the default configuration under a name that
claims otherwise.
stack-deps supplies libaec, without which FEATURE AEC silently switches off.
No pre-commit job: this repo has no .pre-commit-config.yaml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the generated cross-repo-trigger.yml, and trigger-downstream.yml for the three repos that have consumers. Regenerate with the ci-infrastructure-generate CLI; never edit by hand. [[trigger-downstream]].ref stays at develop. That ref is what the orchestrator pins its `uses:` to — GHA forbids an expression there — and it selects only which workflow DEFINITION runs, never the code. Branch coordination is entirely a runtime affair and needs no help from it: the orchestrator passes branch: workflow_run.head_branch, pick-ref checks out the same-named branch in each consumer when one exists, and resolve_deps sees a current-branch matching ^sync-branch- and resolves every upstream's manifest and artifacts from it. One consequence worth knowing: `on: workflow_run` only fires for a workflow file present on the default branch, and runs that copy. So the fan-out cannot be exercised from sync-branch-ci-rollout at all — it goes live only once these files reach develop. Nothing to flip later; this is the steady-state form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci-infrastructure removed its reusable .github/workflows/pre-commit.yml (ecmwf/ci-infrastructure#14), so the comment here pointed at a workflow that no longer exists. Each repo now carries the job itself; stack-dependencies' ci.yml has the copyable version. Comment only — no job runs either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The expensive work — the actual compile — is already on arc-runner-very-large via the matrix legs. resolve was not: it sat on arc-runner-normal inside the base container, for a job that is one batched GraphQL walk of the manifest graph with no compiler and nothing to build. ubuntu-slim is the cheap 1-CPU ARC runner and needs no container, which is also what ci-infrastructure's generator picks for the identical resolve job it emits into cross-repo-trigger.yml. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These are the first expensive compiles in the new system, and artifact reuse does not help them: it avoids a rebuild only when nothing changed, so a PR touching one source file still recompiles the whole tree in a fresh container. That is exactly what sccache is for. ecbuild and stack-dependencies did without it for good reason — one compiles nothing, the other builds once and is then reused from its artifact. The steps live in the build action rather than the workflow so both entry points get them: the generated cross-repo-trigger.yml calls this action directly and never sees the workflow's own steps. It already exports SCCACHE_BUCKET; ci.yml now does too, alongside the artifact bucket. They share the object store's endpoint and credentials and differ only in bucket. Opt-in by presence of that bucket. sccache has no default bucket and a daemon that cannot reach one refuses to start, which would fail the build outright, so with SCCACHE_BUCKET unset both steps no-op, the launcher flags come out empty and the compile just runs uncached. A repo without the secret still builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts the ubuntu-slim change: it saved runner size and lost far more to a per-job venv build. ci-infrastructure is baked into the base image and advertised via CI_INFRASTRUCTURE_PYTHON, which makes ensure-infrastructure-present a true no-op — no build, no PyPI/GitHub egress. ubuntu-slim is a runner label, not an image, so with no container there is no bake and every run installs the package from scratch. It cannot even be cached between runs: the venv lives under RUNNER_TOOL_CACHE inside _work, and ARC's Kubernetes mode overlays _work with an ephemeral per-job volume, so the cache is discarded each time. resolve is short, so the container dominates its cost, not the runner size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All three build legs failed at configure with "No CMAKE_Fortran_COMPILER could be found" (CMakeLists.txt:192, ecbuild_enable_fortran). eccodes builds its Fortran interface by default — FEATURE FORTRAN, DEFAULT ON — and the CI images ship only versioned binaries, so there is no unversioned `gfortran` for CMake to discover. The build action set CMAKE_CXX_COMPILER and nothing else. Adds fortran-compiler to the legs and threads it through to -DCMAKE_Fortran_COMPILER. It also joins [package].compiler-inputs: the artifact carries Fortran objects and .mod files, so the Fortran compiler is genuinely part of its identity, and a future consumer linking that interface would need to match it. Nothing consumes eccodes yet, so the artifact-name change is free to make now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #527 +/- ##
========================================
Coverage 88.22% 88.22%
========================================
Files 854 854
Lines 63549 63549
Branches 11290 11290
========================================
Hits 56067 56067
Misses 7482 7482 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
eccodes built and tested inside a SLURM job against the atos-hpc-gnu ecbuild and
stack-deps artifacts, in two legs mirroring the runner lane: the default build,
and one with the eckit-backed geography backend that is the only configuration
linking eckit at all.
The `when = { options = ["eckit-geo"] }` predicate on the eckit [[deps]] entry
is per-leg and lane-agnostic, so it scopes this lane exactly as it scopes the
runner one. Verified by resolving the legs offline: the plain HPC leg neither
fetches eckit nor carries it in its deps-hash, and its artifact name is
byte-identical to a world where eckit were not declared.
`options` is part of artifact identity, so the geo leg publishes under an
opts.eckit-geo segment and cannot collide with the plain leg; `job-script` is
per-leg and deliberately is not part of identity, which is what lets the two
legs share a kind while running different recipes.
Both recipes pass -DCMAKE_Fortran_COMPILER for the same reason the runner leg
does: FEATURE FORTRAN is DEFAULT ON and there is no unversioned gfortran for
CMake to discover.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two legs of each kind differ ONLY by `options`: the plain g++-13 leg and the
eckit-geo one on the runner lane, and the two atos-hpc-gnu legs on HPC. `options`
is part of artifact identity, so those pairs publish different artifacts — but
compiler and platform, the fields the generated name was built from, are
identical, and both rendered as "eccodes/build (g++-13, ubuntu-24.04)" and
"eccodes/build-hpc (Release, atos-hpc-gnu)".
That is not only an Actions-tab annoyance: the same string is the name of the
check run posted back to the dispatcher's commit, so an upstream PR showed two
identically-named checks and there was no way to tell which artifact a red one
belonged to.
The hand-written ci.yml was already correct here — it renders
`${{ matrix.options || 'default' }}` — so only the generated cross-repo
workflows were affected. The generator now appends the same slot; regenerated
with the change in ecmwf/ci-infrastructure.
NOTE ON ORDER: these files are only consistent with a generator that has that
change. validate-generated-workflows re-renders with ci-infrastructure@main and
diffs, so until it lands there this repo's validate job will report these two
files as stale.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1jc9nyMBC8eQaxkF4vhoq
ci.yml has run ctest here all along, but the generated cross-repo-trigger.yml job only built and published. So when an upstream fanned out to us, a green downstream/runner proved only that eccodes still LINKED against the new ecbuild, stack-deps and eckit -- never that it still worked. Set `ctest = true` on [matrix.build] and regenerate. The step lands after the publish step, matching ci.yml: whoever consumes our artifact picks it up as soon as publish runs and never waits on these tests. Plain `ctest`, exactly what ci.yml runs. Also add .github/actionlint.yaml so the ARC scale-set labels stop reading as unknown runners. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZXmNdJJYqZkpBNTycoPdP
The PR had gone mergeable=CONFLICTING, and GitHub does not deliver pull_request events for a PR whose merge ref it cannot compute -- so eccodes had run no CI at all since 2026-08-21 and was contributing nothing to the rollout signal. This unblocks it. develop landed the cross-repo orchestration independently (#530), so .ci/manifest.toml and both generated cross-repo-trigger*.yml files existed on both sides as add/add conflicts. Resolved to this branch's copies, which are strictly ahead of develop's on all three -- they carry the `options` job-name slot, the HPC leg container, and `ctest = true` -- then regenerated and re-checked to confirm the resolution matches what the generator produces. The only content the merge actually brings in is the MTG2Encoder key (#529). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZXmNdJJYqZkpBNTycoPdP
sawom666
reviewed
Aug 22, 2026
| -DCMAKE_BUILD_TYPE=Release \ | ||
| -DCMAKE_Fortran_COMPILER=gfortran \ | ||
| -DENABLE_EXTRA_TESTS=1 \ | ||
| -DENABLE_ECCODES_OMP_THREADS=1 \ |
Contributor
There was a problem hiding this comment.
Why OpenMP (as opposed to POSIX) ?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contributor Declaration
By opening this pull request, I affirm the following: