NWP Consumer

Download and convert weather data for use in ML pipelines

Some renewables, such as solar and wind, generate power according to the weather conditions. Any forecasting therefore requires predictions of how these conditions will change. Many meteorological organisations provide Numerical Weather Prediction (NWP) data, which can then used for model training and inference.

This data is often very large and can come in various formats. Furthermore, these formats are not necessarily suitable for training, so may require preprocessing and conversion.

This package aims to streamline the collection and processing of this NWP data.

Note

This is not built to replace tools such as Herbie. It is built to produce data specific to the needs of Open Climate Fix's models, so things like the output format and the variable selection are hard-coded. If you need a more configurable cli-driven tool, consider using herbie instead.

Installation

Install from PyPi using pip:

$ pip install nwp-consumer

Or use the container image:

$ docker pull ghcr.io/openclimatefix/nwp-consumer

Example usage

To download the latest available day of GFS data:*

$ nwp-consumer consume

To create an archive of a month of GFS data:

Note

This will download several gigabytes of data to your home partition. Make sure you have plenty of free space (and time!)

$ nwp-consumer archive --year 2024 --month 1

Documentation

Documentation is generated via pdoc. To build the documentation, run the following command in the repository root:

$ python -m pdoc -o docs --docformat=google src/nwp_consumer

FAQ

How do I authenticate with model repositories that require accounts?

Authentication, and model repository selection, is handled via environment variables. Choose a repository via the MODEL_REPOSITORY environment variable. Required environment variables can be found in the repository's metadata function. Missing variables will be warned about at runtime.

How do I use an S3 bucket for created stores?

The ZARRDIR environment variable can be set to an S3 url (ex: s3://some-bucket-name/some-prefix). Valid credentials for accessing the bucket must be discoverable in the environment as per Botocore's documentation

How do I write to Icechunk instead of Zarr?

Set the STORE_BACKEND environment variable to icechunk. Data is then written to one Icechunk store per month in ZARRDIR (ex: 2026-02.icechunk), which holds every init time of the model in that month. Each run adds its init times to the store, committing each one only once it has been written and validated in full, so readers never see an incomplete init time. Runs for different init times can safely write to the same store concurrently.

The store's boolean init_time_complete coordinate records which init times were written in full. Init times committed with a small number of failed writes (within the consumer's error tolerance) are left unmarked, and are retried by later runs that request them. Readers can use the coordinate to select only complete init times.

Failed and conflicting runs leave behind data that is never committed. Remove it by periodically running the maintain command for each month's store:

$ nwp-consumer maintain --year 2026 --month 2

This only deletes uncommitted data more than a day old, so it is safe to run alongside consuming runs. Add --expire-snapshots-older-than-days N to also discard the store's history older than N days, along with any data that only that history references.

How do I change what variables are pulled?

With difficulty! This package pulls data specifically tailored to Open Climate Fix's needs, and as such, the data it pulls (and the schema that data is surfaced with) is a fixed part of the package. A large part of the value proposition of this consumer is that the data it produces is consistent and comparable between different sources, so pull requests to the effect of adding or changing this for a specific model are unlikely to be approved.

However, desired changes can be made via cloning the repo and making the relevant parameter modifications to the model's expected coordinates in it's metadata for the desired model repository.

Development

Linting and static type checking

This project uses ty for static type checking and Ruff for linting. Installing the development dependencies makes them available in your virtual environment.

Use them via the provided Makefile targets:

$ make typecheck
$ make lint

Be sure to do this periodically while developing to catch any errors early and prevent headaches with the CI pipeline. It may seem like a hassle at first, but it prevents accidental creation of a whole suite of bugs.

Running the test suite

Run the unittests with:

$ make test

Further reading

On packaging a python project using setuptools and pyproject.toml:

• The official PyPA packaging guide.
• A step-by-step practical guide on the godatadriven blog.
• The pyproject.toml metadata specification.

On hexagonal architecture:

• A concrete example using Python.
• An overview of the fundamentals incorporating Typescript
• Another example using Go.

On the directory structure:

• The official PyPA discussion on src and flat layouts.

---

Contributing and community

• PR's are welcome! See the Organisation Profile for details on contributing
• Find out about our other projects in the OCF Meta Repo
• Check out the OCF blog for updates
• Follow OCF on LinkedIn

devsjc 📆

Peter Dudfield 💻

Jacob Prince-Bieker 💻

Aditya Sawant 💻

Mark van der Broek 📖

---

Part of the Open Climate Fix community.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!