-
Notifications
You must be signed in to change notification settings - Fork 107
Sync branch ci rollout #527
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mcocdawc
wants to merge
13
commits into
develop
Choose a base branch
from
sync-branch-ci-rollout
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
c475c0c
CI: add the ecmwf/ci-infrastructure pipeline alongside the existing one
mcocdawc 0ec3d86
CI: generate the cross-repo workflows
mcocdawc c807de5
CI: point the pre-commit note at the inlined job
mcocdawc f4a4549
CI: run resolve on ubuntu-slim, keep the builds on very-large
mcocdawc ce10997
CI: compile through sccache
mcocdawc 016d7cd
CI: run resolve in the base image again
mcocdawc fc44038
CI: pass the Fortran compiler
mcocdawc d93d192
CI: add HPC build legs, plain and eckit-geo
mcocdawc 6e6d19a
trigger pipeline
mcocdawc c3e73fa
CI: regenerate — job names now carry the build options
mcocdawc 6ac576c
trigger pipeline
mcocdawc d88ec9e
CI: run ctest in the fan-out, not just on push/PR
mcocdawc de9f475
Merge branch 'develop' into sync-branch-ci-rollout
mcocdawc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| #!/bin/bash | ||
| # eccodes' HPC build recipe WITH the eckit-backed geography backend enabled. | ||
| # | ||
| # Identical to .ci/hpc/build.sh except for the two ENABLE_ flags below, which | ||
| # pull in the atos-hpc-gnu eckit artifact. This is the HPC counterpart of the | ||
| # runner `options = "eckit-geo"` leg, and the only configuration in which eckit | ||
| # is a dependency of eccodes at all — see the `when` predicate on the eckit | ||
| # [[deps]] entry in .ci/manifest.toml. | ||
| # | ||
| # ci-infrastructure wraps this file (it waits for the source transfer, unpacks | ||
| # into node-local $TMPDIR and cds there, exports $CMAKE_PREFIX_PATH / | ||
| # $CI_INSTALL_PREFIX, appends the sentinel), so this script owns only its #SBATCH | ||
| # resources, module loads and the build/test/install — and must NOT print | ||
| # "Finished: ..." itself. | ||
|
|
||
| # atos (hpc2020) selects on QoS rather than partition; ssdtmp sizes the | ||
| # node-local SSD behind $TMPDIR, which holds the unpacked source and the build. | ||
| #SBATCH --qos=nf | ||
| #SBATCH --gres=ssdtmp:20G | ||
| #SBATCH --time=00:40:00 | ||
| #SBATCH --nodes=1 | ||
| #SBATCH --ntasks=8 | ||
|
|
||
| module load prgenv/gnu | ||
| module load cmake | ||
| module load ninja | ||
|
|
||
| # FEATURE FORTRAN is DEFAULT ON, so the Fortran compiler must be named | ||
| # explicitly — same reason as the runner leg, where the image ships only a | ||
| # versioned gfortran. ENABLE_AEC finds libaec in the stack-deps prefix. | ||
| cmake -S "$CI_SOURCE_DIR" -B "${TMPDIR:-/tmp}/build" \ | ||
| -GNinja \ | ||
| -DCMAKE_BUILD_TYPE=Release \ | ||
| -DCMAKE_Fortran_COMPILER=gfortran \ | ||
| -DENABLE_EXTRA_TESTS=1 \ | ||
| -DENABLE_ECCODES_OMP_THREADS=1 \ | ||
| -DENABLE_GEOGRAPHY=1 \ | ||
| -DENABLE_ECKIT_GEO=1 \ | ||
| -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON \ | ||
| -DCMAKE_PREFIX_PATH="$CMAKE_PREFIX_PATH" \ | ||
| -DCMAKE_INSTALL_PREFIX="$CI_INSTALL_PREFIX" | ||
| cmake --build "${TMPDIR:-/tmp}/build" --parallel "${SLURM_NTASKS:-8}" | ||
| ctest --test-dir "${TMPDIR:-/tmp}/build" --output-on-failure | ||
| cmake --install "${TMPDIR:-/tmp}/build" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| #!/bin/bash | ||
| # eccodes' HPC build recipe, submitted as a SLURM job by build-on-hpc. | ||
| # | ||
| # Configures against the resolved $CMAKE_PREFIX_PATH (the cluster-built ecbuild | ||
| # and stack-deps), builds the library and its tests, and runs ctest — so a green | ||
| # job proves those artifacts link and run on a compute node. | ||
| # | ||
| # ci-infrastructure wraps this file (it waits for the source transfer, unpacks | ||
| # into node-local $TMPDIR and cds there, exports $CMAKE_PREFIX_PATH / | ||
| # $CI_INSTALL_PREFIX, appends the sentinel), so this script owns only its #SBATCH | ||
| # resources, module loads and the build/test/install — and must NOT print | ||
| # "Finished: ..." itself. | ||
|
|
||
| # atos (hpc2020) selects on QoS rather than partition; ssdtmp sizes the | ||
| # node-local SSD behind $TMPDIR, which holds the unpacked source and the build. | ||
| #SBATCH --qos=nf | ||
| #SBATCH --gres=ssdtmp:20G | ||
| #SBATCH --time=00:40:00 | ||
| #SBATCH --nodes=1 | ||
| #SBATCH --ntasks=8 | ||
|
|
||
| module load prgenv/gnu | ||
| module load cmake | ||
| module load ninja | ||
|
|
||
| # FEATURE FORTRAN is DEFAULT ON, so the Fortran compiler must be named | ||
| # explicitly — same reason as the runner leg, where the image ships only a | ||
| # versioned gfortran. ENABLE_AEC finds libaec in the stack-deps prefix. | ||
| cmake -S "$CI_SOURCE_DIR" -B "${TMPDIR:-/tmp}/build" \ | ||
| -GNinja \ | ||
| -DCMAKE_BUILD_TYPE=Release \ | ||
| -DCMAKE_Fortran_COMPILER=gfortran \ | ||
| -DENABLE_EXTRA_TESTS=1 \ | ||
| -DENABLE_ECCODES_OMP_THREADS=1 \ | ||
| -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON \ | ||
| -DCMAKE_PREFIX_PATH="$CMAKE_PREFIX_PATH" \ | ||
| -DCMAKE_INSTALL_PREFIX="$CI_INSTALL_PREFIX" | ||
| cmake --build "${TMPDIR:-/tmp}/build" --parallel "${SLURM_NTASKS:-8}" | ||
| ctest --test-dir "${TMPDIR:-/tmp}/build" --output-on-failure | ||
| cmake --install "${TMPDIR:-/tmp}/build" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # SPDX-FileCopyrightText: 2026 European Centre for Medium-Range Weather Forecasts (ECMWF) | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # actionlint validates runs-on against GitHub's hosted labels and flags anything | ||
| # else as unknown. These are this org's ARC scale sets and the HPC login-node | ||
| # runner; add new ones here when a scale set is added. Kept identical to | ||
| # ecmwf/ci-infrastructure/.github/actionlint.yaml so the files stay diffable. | ||
| self-hosted-runner: | ||
| labels: | ||
| - arc-runner-normal | ||
| - arc-runner-large | ||
| - arc-runner-very-large | ||
| - arc-hpc-pet-vsphere-prod | ||
| - hpc | ||
| # Legacy: the macOS builders nightly-testing.yml selects via its `labels` | ||
| # matrix. They go once old_CI.yml and the nightly-* workflows retire. | ||
| - platform-builder-macosx-13.4.1-arm64 | ||
| - platform-builder-macosx-13.4.1-x86_64 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| name: Build eccodes from source | ||
| description: > | ||
| Configures, builds and installs the already-checked-out eccodes against the | ||
| resolved dependency prefixes, and leaves the build tree in place so the caller | ||
| can run ctest against it without reconfiguring. Caller must have already run | ||
| actions/checkout@v6 in the same job — the action reads the source from | ||
| $GITHUB_WORKSPACE. | ||
|
|
||
| inputs: | ||
| cmake-prefix-path: | ||
| description: 'Semicolon-separated prefixes of the resolved deps' | ||
| required: true | ||
| cxx-compiler: | ||
| description: 'C++ compiler binary including version (e.g. clang++-18, g++-13)' | ||
| required: true | ||
| fortran-compiler: | ||
| description: > | ||
| Fortran compiler binary including version (e.g. gfortran-13). eccodes | ||
| builds its Fortran interface by default (FEATURE FORTRAN, DEFAULT ON) and | ||
| the CI images ship only versioned binaries — there is no unversioned | ||
| `gfortran` for CMake to discover — so this must be passed explicitly. | ||
| required: true | ||
| build-type: | ||
| description: 'CMake build type (e.g. Release, Debug, RelWithDebInfo)' | ||
| required: true | ||
| options: | ||
| description: > | ||
| Named feature configuration for this leg. Empty is the default build. | ||
| "eckit-geo" turns on the eckit-backed geography backend, and is the only | ||
| configuration for which eckit is a dependency at all (see the `when` | ||
| predicate on the eckit [[deps]] entry in .ci/manifest.toml). | ||
| required: false | ||
| default: '' | ||
|
|
||
| outputs: | ||
| install-path: | ||
| description: 'Absolute path where eccodes is installed' | ||
| value: ${{ steps.set-paths.outputs.install-path }} | ||
| build-dir: | ||
| description: 'Absolute path of the build tree, for ctest' | ||
| value: ${{ steps.set-paths.outputs.build-dir }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Get install prefix | ||
| id: prefix | ||
| uses: ecmwf/ci-infrastructure/actions/install-prefix@main | ||
|
|
||
| - name: Set paths | ||
| id: set-paths | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| { | ||
| echo "install-path=${{ steps.prefix.outputs.base }}/eccodes" | ||
| echo "build-dir=$RUNNER_TEMP/_eccodes-build" | ||
| } >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Map options to CMake flags | ||
| id: opts | ||
| shell: bash | ||
| env: | ||
| OPTIONS: ${{ inputs.options }} | ||
| # Fail loud on an unknown value: silently building the default | ||
| # configuration would publish it under an opts.<name> artifact name and | ||
| # quietly claim to be something it is not. | ||
| run: | | ||
| set -euo pipefail | ||
| case "$OPTIONS" in | ||
| '') flags='' ;; | ||
| 'eckit-geo') flags='-DENABLE_GEOGRAPHY=1 -DENABLE_ECKIT_GEO=1' ;; | ||
| *) | ||
| echo "::error::unknown options value '$OPTIONS' for eccodes; add it here and to .ci/manifest.toml" >&2 | ||
| exit 1 | ||
| ;; | ||
| esac | ||
| echo "cmake-flags=$flags" >> "$GITHUB_OUTPUT" | ||
|
|
||
| # Compile caching. Lives in the build action, not the workflow, so both the | ||
| # push/PR path and the generated cross-repo-trigger.yml path get it — the | ||
| # latter calls this action directly and never sees the workflow's steps. | ||
| # | ||
| # Opt-in by presence of the bucket: sccache has no default bucket and a | ||
| # daemon that cannot reach one refuses to start, which would fail the build. | ||
| # With SCCACHE_BUCKET unset both steps below no-op and the compile runs | ||
| # uncached, so a repo without the secret still builds. | ||
| - name: Set up sccache | ||
| if: env.SCCACHE_BUCKET != '' | ||
| uses: ecmwf/ci-infrastructure/actions/setup-sccache@main | ||
| with: | ||
| cache-key-suffix: ${{ inputs.cxx-compiler }}-${{ inputs.build-type }}-${{ inputs.options }} | ||
|
|
||
| - name: Select compiler launcher | ||
| id: launcher | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| if [ -n "${{ env.SCCACHE_BUCKET }}" ]; then | ||
| echo "flags=-DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "::notice::SCCACHE_BUCKET unset — compiling without sccache" | ||
| echo "flags=" >> "$GITHUB_OUTPUT" | ||
| fi | ||
|
|
||
| - name: Configure eccodes | ||
| shell: bash | ||
| env: | ||
| CMAKE_PREFIX_PATH_IN: ${{ inputs.cmake-prefix-path }} | ||
| CXX_COMPILER: ${{ inputs.cxx-compiler }} | ||
| FC_COMPILER: ${{ inputs.fortran-compiler }} | ||
| BUILD_TYPE: ${{ inputs.build-type }} | ||
| INSTALL_PATH: ${{ steps.set-paths.outputs.install-path }} | ||
| BUILD_DIR: ${{ steps.set-paths.outputs.build-dir }} | ||
| EXTRA_FLAGS: ${{ steps.opts.outputs.cmake-flags }} | ||
| LAUNCHER: ${{ steps.launcher.outputs.flags }} | ||
| run: | | ||
| set -euo pipefail | ||
| # Word-splitting EXTRA_FLAGS is intended — it is a fixed, known set of | ||
| # flags produced by the step above, never user input. | ||
| # shellcheck disable=SC2086 | ||
| cmake -S "$GITHUB_WORKSPACE" \ | ||
| -B "$BUILD_DIR" \ | ||
| -DCMAKE_BUILD_TYPE="$BUILD_TYPE" \ | ||
| -DCMAKE_CXX_COMPILER="$CXX_COMPILER" \ | ||
| -DCMAKE_Fortran_COMPILER="$FC_COMPILER" \ | ||
| -DCMAKE_INSTALL_PREFIX="$INSTALL_PATH" \ | ||
| -DCMAKE_PREFIX_PATH="$CMAKE_PREFIX_PATH_IN" \ | ||
| $EXTRA_FLAGS $LAUNCHER | ||
|
|
||
| - name: Build and install eccodes | ||
| shell: bash | ||
| env: | ||
| BUILD_DIR: ${{ steps.set-paths.outputs.build-dir }} | ||
| run: | | ||
| set -euo pipefail | ||
| cmake --build "$BUILD_DIR" --parallel "$(nproc)" | ||
| cmake --install "$BUILD_DIR" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why OpenMP (as opposed to POSIX) ?