Feature/new grib1 to grib2 distributed tool - #292
Merged
Conversation
MircoValentiniECMWF
requested review from
dsarmany and
tweska
and
a lite review from Copilot
and removed request for
dsarmany and
tweska
August 21, 2026 10:49
Contributor
There was a problem hiding this comment.
Pull request overview
This PR replaces the legacy GRIB1→GRIB2 distributed conversion implementation with a new grib2grib-based distributed tool that relies on metkit helpers (grib2mars, mars2mars, mars2grib) instead of MultIO-native conversion paths, and also includes fixes for IFS→MARS and statistics handling (notably SES6-related behavior).
Changes:
- Remove the old
dist-grib1-to-grib2tool and its supporting utils/docs, and introduce the newsrc/multio/tools/grib2gribpipeline +distributed-grib-to-gribexecutable. - Update IFS2MARS output-manager metadata population (satellite keys) and adjust flush-step propagation.
- Update statistics/encoding/printing helpers (time handling in hours, SES6-related statistics filtering, and a MARS-focused print stream mode).
Reviewed changes
Copilot reviewed 93 out of 93 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/multiom/ifs2mars/output-manager/model_data2multio_mod.F90 | Adds satellite-key mapping to MARS metadata; adjusts flush-step handling. |
| src/multio/tools/utils/Summary.md | Removes legacy distributed-tool summary documentation. |
| src/multio/tools/utils/scalarGrib1ToGrib2DebugOutputs.h | Removes legacy GRIB1→GRIB2 scalar debug output utilities. |
| src/multio/tools/utils/scalarGrib1ToGrib2DebugOutputs.cc | Removes legacy GRIB1→GRIB2 scalar debug output utilities. |
| src/multio/tools/utils/grib2MarsMisc.h | Removes legacy GRIB-to-MARS misc helper interface used by the old tool. |
| src/multio/tools/utils/distGrib1ToGrib2ProcessFiles.h | Removes legacy per-rank file processing helpers for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2ProcessFiles.cc | Removes legacy per-rank file processing helpers for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2OutcomesReport.h | Removes legacy reporting helpers for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2Options.h | Removes legacy YAML options loader/broadcast helpers for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2Options.cc | Removes legacy YAML options loader/broadcast helpers for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2Mpi.h | Removes legacy MPI helpers for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2Mpi.cc | Removes legacy MPI helpers for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2Logging.h | Removes legacy logging/outcome types for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2Logging.cc | Removes legacy logging/outcome types for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2LoadBalancer.h | Removes legacy load balancer for old distributed tool. |
| src/multio/tools/utils/distGrib1ToGrib2LoadBalancer.cc | Removes legacy load balancer for old distributed tool. |
| src/multio/tools/utils/dist-grib1-to-grib2.example.yaml | Removes legacy example YAML for old distributed tool. |
| src/multio/tools/utils/CodesHandleToEckitMessage.cc | Removes legacy CodesHandle→eckit::Message bridge from old utils. |
| src/multio/tools/grib2grib/WorkUnitLoadBalancer.h | Introduces new work-bucketing/load-balancing interface for grib2grib. |
| src/multio/tools/grib2grib/Utils.h | Adds shared small utility interface for new pipeline (policy parsing, disclaimers). |
| src/multio/tools/grib2grib/Utils.cc | Implements shared small utilities for new pipeline. |
| src/multio/tools/grib2grib/UnitOfWork.h | Adds work-unit abstraction for chunked GRIB iteration in the new pipeline. |
| src/multio/tools/grib2grib/Summary.h | Adds per-file summary aggregation helpers for the new tool. |
| src/multio/tools/grib2grib/Summary.cc | Implements per-file summary aggregation helpers for the new tool. |
| src/multio/tools/grib2grib/stages/PostEncodeValidation.h | Adds standalone post-encode validation stage interface. |
| src/multio/tools/grib2grib/stages/PostEncodeValidation.cc | Implements post-encode validation stage. |
| src/multio/tools/grib2grib/stages/Overrides.h | Adds override-stage interface for MARS/misc dictionaries. |
| src/multio/tools/grib2grib/stages/MarsToMars.h | Adds standalone mars2mars stage interface. |
| src/multio/tools/grib2grib/stages/MarsToMars.cc | Implements standalone mars2mars stage using metkit. |
| src/multio/tools/grib2grib/stages/MarsToGrib.h | Adds standalone mars2grib stage interface + testcase support. |
| src/multio/tools/grib2grib/stages/MarsToGrib.cc | Implements mars2grib stage + optional testcase generation. |
| src/multio/tools/grib2grib/stages/MarsBasedFilter.h | Adds mars-based filtering stage interface (stat-window + selector-based rejection). |
| src/multio/tools/grib2grib/stages/GribToMars.h | Adds standalone grib2mars stage interface. |
| src/multio/tools/grib2grib/stages/GribToMars.cc | Implements grib2mars stage using metkit and value extraction. |
| src/multio/tools/grib2grib/stages/GribBasedFilter.h | Adds GRIB-based coarse filter stage interface (edition/discipline/validity policies). |
| src/multio/tools/grib2grib/stages/Grib2Fdb5.h | Adds sink-write stage interface for archiving encoded GRIB2 to a sink. |
| src/multio/tools/grib2grib/stages/Grib2Fdb5.cc | Implements sink-write stage with error trapping. |
| src/multio/tools/grib2grib/Sink.h | Adds rank-local sink orchestration and debug-sink plumbing for new tool. |
| src/multio/tools/grib2grib/Sink.cc | Implements sink construction + debug sinks + testcase sink. |
| src/multio/tools/grib2grib/ReaderContext.h | Adds configuration for work-unit reader mode (eccodes-stream vs candidate-boundary). |
| src/multio/tools/grib2grib/ProcessRankOwnedUnitsOfWork.h | Adds API to process all rank-owned work units. |
| src/multio/tools/grib2grib/ProcessRankOwnedUnitsOfWork.cc | Implements rank-owned work-unit processing. |
| src/multio/tools/grib2grib/ProcessOneUnitOfWork.h | Adds API to run one work unit through the message pipeline. |
| src/multio/tools/grib2grib/ProcessOneUnitOfWork.cc | Implements per-work-unit processing with error trapping + sink flush. |
| src/multio/tools/grib2grib/ProcessOneMessage.h | Adds flat message pipeline orchestration API. |
| src/multio/tools/grib2grib/ProcessOneMessage.cc | Implements stage-by-stage message pipeline with debug-sink writes on early exits. |
| src/multio/tools/grib2grib/OptionsUtilsMpi.cc | Adds MPI-enabled options loading/broadcast implementation for new tool. |
| src/multio/tools/grib2grib/OptionsUtils.h | Adds options parsing/loading interfaces for new tool. |
| src/multio/tools/grib2grib/OptionsUtils.cc | Implements options parsing/loading (YAML payload checks). |
| src/multio/tools/grib2grib/MultioToolUtils.h | Adds tool-level orchestration helpers for the distributed grib2grib tool. |
| src/multio/tools/grib2grib/MultioToolUtils.cc | Implements orchestration: read file list, distribute work, gather outcomes, write summary. |
| src/multio/tools/grib2grib/MpiUtils.h | Adds MPI wrapper APIs for bucket distribution and outcome gathering. |
| src/multio/tools/grib2grib/MpiUtils.cc | Implements MPI distribution/gather for buckets/outcomes using eckit::mpi::Comm. |
| src/multio/tools/grib2grib/handleGribBoundaries.h | Adds candidate GRIB-boundary scanning helper interface. |
| src/multio/tools/grib2grib/handleGribBoundaries.cc | Implements GRIB boundary scanning/validation (GRIB1+GRIB2) prior to ecCodes decode. |
| src/multio/tools/grib2grib/grib2grib-sink-test.cc | Adds a local harness to exercise sink + testcase writing. |
| src/multio/tools/grib2grib/grib2grib_options-parser.cc | Adds an options parser/validator tool for grib2grib configs. |
| src/multio/tools/grib2grib/GlobalContext.h | Adds aggregated stage-context model for new pipeline. |
| src/multio/tools/grib2grib/GlobalContext.cc | Implements strict parsing/validation for reader+stages sections. |
| src/multio/tools/grib2grib/distributed-grib-to-grib.cc | Introduces the new MPI distributed tool entrypoint using grib2grib pipeline. |
| src/multio/tools/grib2grib/CodesHandleToEckitMessage.h | Adds new CodesHandle→eckit::Message bridge under the new tool namespace. |
| src/multio/tools/grib2grib/CodesHandleToEckitMessage.cc | Implements the new CodesHandle→eckit::Message bridge. |
| src/multio/tools/grib2grib/CMakeLists.txt | Adds build targets for new grib2grib libraries and executables. |
| src/multio/tools/dist-grib1-to-grib2.cc | Removes the legacy distributed tool executable. |
| src/multio/tools/CMakeLists.txt | Stops building legacy GRIB1→GRIB2 tools; adds grib2grib subdirectory. |
| src/multio/action/statistics-mtg2/TimeUtils.cc | Forces statistics current-time computation to treat step as hours. |
| src/multio/action/statistics-mtg2/TemporalStatistics.h | Exposes period name from the period updater. |
| src/multio/action/statistics-mtg2/Statistics.cc | Adjusts timespan computation and adds SES6-driven “skip partial first period” behavior. |
| src/multio/action/statistics-mtg2/cfg/StatisticsConfiguration.cc | Forces current datetime computation to treat step as hours. |
| src/multio/action/print/Print.h | Adds support for a MARS-focused print mode and internal helpers/counter. |
| src/multio/action/print/Print.cc | Implements MARS-focused printing, flush formatting, and prefix helpers. |
| src/multio/action/encode-mtg2/fakeDoubleLoop.cc | Extends stat-type mappings and adds seasonal fcmonth handling. |
| src/multio/action/encode-mtg2/EncodeMtg2.h | Switches encoder ownership to unique_ptr to allow optional configuration. |
| src/multio/action/encode-mtg2/EncodeMtg2.cc | Adds validation/whitelisting of mars2grib-options and constructs encoder accordingly. |
| share/multio/grib1-to-grib2/mtg2-convert.sh.in | Updates job script to call new CLI and sets FDB environment. |
| share/multio/grib1-to-grib2/grib2grib.example.yaml | Adds example YAML for the new pipeline tool. |
| share/multio/grib1-to-grib2/grib2grib.example-03.yaml | Adds additional example YAML for the new pipeline tool. |
| share/multio/grib1-to-grib2/grib2grib.example-02.yaml | Adds additional example YAML for the new pipeline tool. |
| share/multio/grib1-to-grib2/grib2grib-options-example.yaml | Adds minimal example YAML for the options-parser tool. |
| share/multio/grib1-to-grib2/CMakeLists.txt | Installs new example options file(s) with the share data. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+418
to
+419
| std::int64_t ldm = util::lastDayOfTheMonth(ts.cwin().creationPoint().date().year(), | ||
| ts.cwin().creationPoint().date().month()); |
Comment on lines
+22
to
+27
| #include "multio/datamod/MarsMiscGeo.h" | ||
| #include "multio/datamod/core/EntryDumper.h" | ||
|
|
||
| #include "multio/datamod/ContainerInterop.h" | ||
| #include "multio/datamod/MarsMiscGeo.h" | ||
| #include "multio/datamod/core/EntryDumper.h" |
| mars-based-filter: | ||
| verbosity: 0 | ||
| api-options: | ||
| allowFromStartStatisticsForAnalisys: false |
| mars-based-filter: | ||
| verbosity: 0 | ||
| api-options: | ||
| allowFromStartStatisticsForAnalisys: false |
| echo "OPTIONFILE=\${OPTIONFILE}" | ||
| echo "ARCHITECTURE_FILE=\${ARCHITECTURE_FILE}" | ||
|
|
||
| export FDB_HOME="/home/fdbsc" |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #292 +/- ##
===========================================
- Coverage 33.51% 33.00% -0.51%
===========================================
Files 379 409 +30
Lines 42648 43458 +810
Branches 1839 1978 +139
===========================================
+ Hits 14292 14344 +52
- Misses 28356 29114 +758 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… encode-mtg2 action
…asonal statistics
dsarmany
force-pushed
the
feature/new-grib1-to-grib2-distributed-tool
branch
from
August 21, 2026 17:35
e44b119 to
5bd2fbb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contributor Declaration
By opening this pull request, I affirm the following:
🌈🌦️📖🚧 Documentation 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/dev-section/multio/pull-requests/PR-292