One (PVNet) app to rule them all (⛰️💍)

We currently have

uk-pvnet-app
india-forecast-app
site-forecast-app

Note that uk-pvnet-app currently save to national-gsp database, where as india-forecast-app and site-forecast-app saves to site database. We hope to mvoe these all to the data-platform soon

It would be nice to just have one app, that is configured in different ways.
This means data checks on NWP and Satellite, can be used throughout.

Other upgrades, india-forecast-app currently runs 1 site at a time, but really we could batch these up and run them together.

Slightly negative of having everything together, is perhaps development speed is reduced.

Im going to close this. I think we are happy with uk-pvnet-app and site-forecast-app

I thought it would be nice to write down what the current forecasters have. This helps plan the features we want in the one forecast app to rule them all

Updated 2026-05-20

Feature	uk-pvnet-app	site-forecast-app
pydantic config for models	yes	yes
runs pvnet	yes	yes
runs summatiomal model	yes	yes
loop over	models	sites and models
load generation values	no	yes
load capacities	yes	yes
data validation	X%	Y%
batch validation	yes	yes
save to data-platform	yes	yes
read to data-platform	yes	yes
removed datamodel	yes	no
can run other models than PVNet	no	yes
code quality (not sure how much should trust ChatGPT on this)	7/10	6/10
Dynamic for groups of sites	no	yes
NL Curtailment	no	yes
Other

Would like to create an implementation plan for this in Q2 26, alongside the work to add data platform to both of these tables which will make it much easier to do this

We did some feature planning here, next steps are to

https://miro.com/app/board/uXjVHGr-b9k=/
Next steps

Move to github (Peter)
Plan config (Peter v0)
Plan the main app structure
Plan module structure
Build

General notes for this app

We want to be able to select groups of models to run, run N models with N locations
R&D models (or older data-sampler) should live on branches. This allows the flexibility of high quality code, but also allows us to use it for R&D
Lets use data contracts (some in ocf-data-sampler) as needed
Only data-platform, lets not include the old datamodels
Just PVNet for the moment
Load from icechunk for satellite data
Clear logging if some models failed
Deploy with new trained models, lets try to minimize the edge cases around old models, by just retraining new models
Run on one energy source at once i.e only wind or pv
Aim for one end-to-end tests
Run models even if other models can’t be run
Module code
We want to be able to run the app locally

General / Infrastructure

Logging, clear logging and discuss about what is the right level
Tests, shold aim for X% coverage
Linting shold be part of it
CI, see standard OCF CI templates)
Docker, we need to build a docker file
use uv when possible, put we might need conda due to regridding library
Regions: NL, GB, India + …

Data Layer

Get generation data
Get NWP data
Get NL DA prices (might also be some other countries data we need from enstoe)
Location metadata, like capacity and lat, long
Satellite (and backup satellite)
Validation for example no nans
Validate enough timestamps that the data sources have enough timestamps, and remove models if needed
Only download data once from S3
Processing inputs
- Regridding for MO and ...
- options for Unit scale
- Support icechunk (trim out stuff) for satellite

Forecasting

Run on all GSPs (for UK)
Run on a site group
Run for NL
Download models from HF
Run PVNet (and only PVNet)
Run summation model (only if neded)
We want to use ML batches, not one sample at a time
Lets do some batch validation

Post-processing

Run curtailment (NL)
Night time zeros
run adjuster, note we need options for different observes in the adjuster
validation, like no zig zag forecasts, or no forecast above X% of capacity

Saving

Data platform, and lets save Input data metadata (with satellite)
Option to save to csv
Save uncurtailed models too

Configuration

HF hashes for model
HF hashes for summation model
Grouping (e.g., UK, NL, India) or site groups (perhaps need to think about all the different use cases here)
PV / wind (run one at a time)
Different models, different processing
Saving
Post-processing (adjuster, curtailment, etc.)

Misc

Lets include NL Blend, and later decide if we want to bring UK blend into this

WIP: Here's a draft of what the configuration could look like

I'm trying to make one big configuration that has everything in it, like a list of all models.
Then have a second configuration (much smaller) that is more used for running the app

App configuration

data:
- [InputData]
models:
- [Model]
models_locations:
- [ModelsAndLocations].  
processing_steps:
- [ProcessingStep]

Model:
- name
- hf 
    - repo
    - hash
- summation_model: Model
- processing_steps: list[ProcessingStep.name]
- data
    - [InputData.name] 

ModelsAndLocations (this is like uk, nl or india or ....)
- name (e.g. uk_gsp_and_national, or nl_regions_and_national)
- model_names: [Model.name]
- location_type 
- location_type_summational
- organization_filter (optional, this is a way to filter on different ones, for example use NL National not UK nationa. We might want a more defined solution in the data-platform)
- energy_source
- processing_steps: [ProcessingStep.name] 

InputData:
- name
- type (nwp or satellite or ...)
- location (env_var)
- backup_location (env_var)
- processing_config: [ProcessingStep.name]
- observer_name: Optional

ProcessingStep
- name (e.g. nl_curtailed, adjuster)
- class: 
- type: e.g after model run, before save, inputdata
- add other inputs? (not quite sure how to do this)

The models are basically what we have here

App run configuration

This will be more like env_vars set

models_and_locations:
save_method: 
input_data_names: e.g satellite, satellite_backup, nwp_ecwmwf, nwp_mo, e.t.c

Here's a possible example of a yaml file

models:
  - name: pvnet_v2 
    hf:
        repo: openclimatefix-models/pvnet_uk_region
        commit: cfcb233576c3c8110daed0179452e3d9913c744c
    summation: pvnet_v2_summation
    data: [uk_sat, uk_nwp_ecwmwf, uk_nwp_mo]
  - name: pvnet_v2_summation
    hf:
        repo: openclimatefix-models/pvnet_uk_region
        commit: cfcb233576c3c8110daed0179452e3d9913c744c
  - name: pvnet_ecmwf 
    hf:
        repo: openclimatefix-models/pvnet_uk_region
        commit: 6b7cc5fec03b83837672a759d7e00f249685de6b
    summation: pvnet_ecmwf_summation
    data: [uk_nwp_ecmwf]
  - name: pvnet_ecmwf_summation
    hf:
        repo: openclimatefix-models/pvnet_uk_region
        commit: 0c26b19ab90001ee51f9886227be8286e84a393c

models_locations:
- name: uk_gsp_and_national
  models_names:
  - pvnet_v2
  location_type: gsp
  location_type_summation: nation
  energy_source: solar
  processing_steps:
  - validate_zig_zag
- name: nl
  models_names:
  - TODO
  location_type: state
  location_type_summation: nation
  energy_source: solar
  processing_steps:
  - validate_zig_zag

data:
- name: uk_sat
  type: satellite
  location: ENV_VAR
  back_location: ENV_VAR
  processing_config: [Check_satellite_nans, check_timestamps]
- name: uk_nwp_ecwmwf
  type: nwp
  location: ENV_VAR
  back_location: ENV_VAR
  processing_config: [Check_nwp_nans]
- name: uk_nwp_mo
  type: nwp
  location: ENV_VAR
  back_location: ENV_VAR
  processing_config: [Check_nwp_nans]


processing_steps:
- name: Check_satellite_nans
  class: CheckSatelliteNans
  type: post_data_check
- name: Check_nwp_nans
  class: CheckNwpNans
  type: post_data_check
- name: check_timestamps
  class: CheckTimestamps
  type: post_data_check
- name: validate_zig_zag
  class: ValidateZigZag
  type: post_forecast_check

Feedback

make country while registry
Dont want to load env vars in code, so make sure they are all in config. There are some ways when loading yaml to get env vars in the settings
Processing step could just be a Class and not config
Need thing about how we pass in HuggingFace token, aws region, zigzag thresholds, save batch dir, scratch_dir, dataplatform address + port,
tags on models would be useful, like code, extra ....

v0.2

Idea is to have a registry file for each country (or group) and then a main settings file

Registry configuration

- name (e.g. uk_gsp_and_national, or nl_regions_and_national)
- location_type 
- location_type_summational
- organization_filter (optional, this is a way to filter on different ones, for example use NL National not UK nationa. We might want a more defined solution in the data-platform)
- energy_source
- processing_steps: [ProcessingStep.name] 
- model_names: [Model.name]
- tags: [str]

Model:
- name
- hf 
    - repo
    - hash
- summation_model: Optional[Model]
- processing_steps: list[ProcessingStep.name]
 - tags: Optional[list[str]]

The models are basically what we have here

App settings

This can be set with env_vars and then when it loads the YAML we can load the envvars in

- registry_config_name:
- save_method: 
- hugging_face_token:
- aws_region:
- scratch_dir: 
- data_platform_address: (host and address)
- (other one off variables) ...
- input_data: [InputData]
- processing_step: [ProcessingStep]

InputData:
- name
- type (nwp or satellite or ...)
- location (env_var)
- processing_config: [ProcessingStep.name]
- observer_name: Optional
- kwargs # can put in a list of things like `observer_name` and `backup_location`

ProcessingStep
- name (e.g. nl_curtailed, adjuster)
- class: 
- type: e.g after model run, before save, inputdata
- kwargs # can put in things like `adjuster_limit_mw` or `zigzag_threshold`

Example - uk_gsp_and_national.yaml

name: uk_gsp_and_national
location_type: gsp
location_type_summation: nation
energy_source: solar
processing_steps:
- validate_zig_zag
- adjuster
models:
- name: pvnet_v2 
   hf:
     repo: openclimatefix-models/pvnet_uk_region
     commit: cfcb233576c3c8110daed0179452e3d9913c744c
   summation: pvnet_v2_summation
  tags: core
- name: pvnet_v2_summation
  hf:
      repo: openclimatefix-models/pvnet_uk_region_summational
      commit: cfcb233576c3c8110daed0179452e3d9913c744c
   tags: summation
- name: pvnet_mo
   hf:
      repo: openclimatefix-models/pvnet_uk_region
      commit: 6b7cc5fec03b83837672a759d7e00f249685de6b
  summation: pvnet_mo_summation
  tags: extra
- name: pvnet_mo_summation
  hf:
      repo: openclimatefix-models/pvnet_uk_region_summational
      commit: 0c26b19ab90001ee51f9886227be8286e84a393c
  tags: summation

Example 2 - nl

name: nl_state_and_national
location_type: state
location_type_summation: nation
energy_source: solar
processing_steps:
- adjuster
models:
- name: nl_all
   hf:
     repo: openclimatefix-models/pvnet_uk_region
     commit: cfcb233576c3c8110daed0179452e3d9913c744c
  summation: nl_all_summation
  processing_steps: 
  - nl_curtail_and_uncurtailed # save both the curtailed and uncurtailed, by DA prices
  tags: core
- name: nl_all_summation
  hf:
      repo: openclimatefix-models/pvnet_uk_region
      commit: cfcb233576c3c8110daed0179452e3d9913c744c
   tags: summation
- name: nl_ecmwf 
   hf:
      repo: openclimatefix-models/pvnet_uk_region
      commit: 6b7cc5fec03b83837672a759d7e00f249685de6b
  summation: nl_ecmwf_summation
  processing_steps: 
  - nl_curtail # curtail the forecast by DA prices
  tags: extra
- name: nl_ecmwf_summation
  hf:
      repo: openclimatefix-models/pvnet_uk_region
      commit: 0c26b19ab90001ee51f9886227be8286e84a393c
  tags: summation

Settings config

registry_config_name: REGISTRY_CONFIG_NAME
save_method: SAVE_METHOD
hugging_face_token: HUGGING_FACE_TOKEN
aws_region: AWS_REGION
scratch_dir: SCRATCH_DIR
data_platform_address: DP_ADDRESS:DP_HOST

input_data: 
- name: uk_sat
  type: satellite
  location: SATELLITE_ICECHUNK
  processing_config: [Check_satellite_nans, check_timestamps]
  kwargs:
  - satellite_icechunk_backup:SATELLITE_ICECHUNK_BACKUP
- name: uk_nwp_ecmwf
  type: nwp
  location: NWP_ECMWF_ZARR
  processing_config: [Check_nwp_nans]
- name: uk_nwp_mo
  type: nwp
  location: NWP_MO_ZARR
  processing_config: [Check_nwp_nans]

processing_steps:
- name: Check_satellite_nans
  class: CheckSatelliteNans
  type: post_data_check
- name: Check_nwp_nans
  class: CheckNwpNans
  type: post_data_check
- name: check_timestamps
  class: CheckTimestamps
  type: post_data_check
- name: validate_zig_zag
  class: ValidateZigZag
  kwargs:
  - forecast_validate_zig_zag_error: FORECAST_VALIDATE_ZIG_ZAG_ERROR
  - forecast_validate_zig_zag_warning: FORECAST_VALIDATE_ZIG_ZAG_WARNING
  type: post_forecast_check
- name: nl_curtail
  class: NLCurtailed
- name: nl_curtail_and_uncurtailed
  class: NLCurtailedAndUncurtailed
- adjuster:
  class: Adjuster
  kwargs:
  - observer_name: OBSERVER_NAME
  - uncurtailed_observer_name: UNCURTAILED_OBSERVER_NAME
  - adjuster_limit_mw:ADJUSTER_LIMIT_MW
  - adjuster_limit_per:ADJUSTER_LIMIT_PER

Just to flag this small change we need for DE in site-forecast-app https://github.com/openclimatefix/client-private/issues/615

Navigation Menu

Repository navigation

One (PVNet) app to rule them all (⛰️💍) #409

General notes for this app

General / Infrastructure

Data Layer

Forecasting

Post-processing

Saving

Configuration

Misc

App configuration

App run configuration

Registry configuration

App settings

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

One (PVNet) app to rule them all (⛰️💍) #409

Description

Activity

peterdudfield commented on Jul 10, 2025

peterdudfield commented on Mar 19, 2026

Sukh-P commented on Mar 19, 2026

peterdudfield commented on Jun 18, 2026

peterdudfield commented on Jun 18, 2026

General notes for this app

General / Infrastructure

Data Layer

Forecasting

Post-processing

Saving

Configuration

Misc

peterdudfield commented on Jun 18, 2026

App configuration

App run configuration

peterdudfield commented on Jun 30, 2026

peterdudfield commented on Jul 16, 2026

peterdudfield commented on Jul 16, 2026

Registry configuration

App settings

peterdudfield commented on Sep 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions