Skip to content

Suggested refactor for to include parameterised tests with and witho… #250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lcdurkan wants to merge 17 commits into
base: staging
Choose a base branch
from
Open

Suggested refactor for to include parameterised tests with and witho… #250

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
61587bd
Suggested refactor for to include parameterised tests with and witho…
lcdurkan Mar 27, 2026
2f9b349
Remove original test_aggregation_methods.py and update name of copy
lcdurkan Mar 27, 2026
b226eb2
Merge branch 'main' into consolidate-aggregation-unit-tests
lcdurkan Apr 15, 2026
786b94b
Add StandardDeviation method to aggregation methods. Add unit test. T…
lcdurkan Apr 15, 2026
83c36d1
Add unit test for aggregation threshold of albedo_sd. Unit tests pass…
lcdurkan Apr 21, 2026
40095fa
Try to force processed NMDB data to be loaded as raw
lcdurkan Apr 21, 2026
d99d5a3
Revert changes to time_series_processor for processed v raw
lcdurkan Apr 24, 2026
88a91b6
Fetch site metadata for dependents from different sites/networks
richjam May 1, 2026
40bc81a
Add EddyPro ancillary biomet/dynamic metadata support
hkhan38 Apr 17, 2026
a3b1747
Add StandardDeviation method to aggregation methods. Add unit test. T…
lcdurkan Apr 15, 2026
aaf48fb
Add unit test for aggregation threshold of albedo_sd. Unit tests pass…
lcdurkan Apr 21, 2026
1616bbc
Try to force processed NMDB data to be loaded as raw
lcdurkan Apr 21, 2026
02a4341
Revert changes to time_series_processor for processed v raw
lcdurkan Apr 24, 2026
cee02b4
Clean up EddyPro ancillary builders
hkhan38 Apr 30, 2026
7bf5e7e
Update time-stream version and fix flag-api calls
richjam May 5, 2026
c6b60ee
remove main from workflow
nkshaw23 May 6, 2026
968a677
Remove original test_aggregation_methods.py and update name of copy
lcdurkan Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 0 additions & 21 deletions .github/workflows/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ on:
pull_request:
push:
branches:
- main
- staging
- production

Expand All @@ -22,18 +21,6 @@ jobs:
coverage_threshold: 90
localstack_profile_name: localstack

build-test-deploy-docker-current:
if: ${{ github.ref_name == 'main' || github.base_ref == 'main' }}
needs: [test-python]
uses: NERC-CEH/dri-cicd/.github/workflows/build-test-deploy-docker.yml@main
with:
use_uv: true
package_name: dritimeseriesprocessor
event_type: ${{ github.event_name }}
secrets:
AWS_REGION: ${{ secrets.AWS_REGION }}
AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }}

build-test-deploy-docker-staging:
if: ${{ github.ref_name == 'staging' || github.base_ref == 'staging' }}
needs: [test-python]
Expand Down Expand Up @@ -72,14 +59,6 @@ jobs:
AWS_REGION: ${{ secrets.AWS_REGION }}
AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }}

create-main-to-staging-pr:
needs: [build-test-deploy-docker-current, build-test-deploy-docker-eddypro-current]
if: ${{ needs.build-test-deploy-docker-current.result == 'success' && github.event_name == 'push' }}
uses: NERC-CEH/dri-cicd/.github/workflows/auto-create-pr.yml@main
with:
base: staging
head: main

create-staging-to-production-pr:
needs: [build-test-deploy-docker-staging]
if: ${{ needs.build-test-deploy-docker-staging.result == 'success' && github.event_name == 'push' }}
Expand Down
Binary file added flux-data/fdri/dataset=FLUX_BIOMET_15MIN/site=flux-plynl/date=2026-01-20/data.parquet
View file
Open in desktop
Binary file not shown.
Binary file added flux-data/fdri/dataset=FLUX_BIOMET_1MIN/site=flux-plynl/date=2026-01-20/data.parquet
View file
Open in desktop
Binary file not shown.
Binary file added flux-data/fdri/dataset=FLUX_METADATA/site=flux-plynl/date=2026-01-20/data.parquet
View file
Open in desktop
Binary file not shown.
2 changes: 1 addition & 1 deletion src/dritimeseriesprocessor/__assets__/eddypro_templates/processing_template.eddypro
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,7 +384,7 @@ biom_tstamp_ref=end
biom_ta=2
biom_pa=4
biom_rh=3
biom_rg=0
biom_rg=5
biom_lwin=0
biom_ppfd=0

Expand Down
259 changes: 257 additions & 2 deletions src/dritimeseriesprocessor/__metadata__/eddypro/datasets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,268 @@
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "Flux",
"source_site": "flux-plynl",
"source_site_identifier": "PLYNL",
"source_site_identifier": "flux-plynl",
"source_column": null,
"time_column_name": "TIMESTAMP",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-ta",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-met",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_BIOMET_1MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "AirTemp_C",
"time_column_name": "time",
"resolution": "PT1M",
"periodicity": "PT1M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-rh",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_BIOMET_1MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "RH",
"time_column_name": "time",
"resolution": "PT1M",
"periodicity": "PT1M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-pa",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_BIOMET_1MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "Pressure_Avg",
"time_column_name": "time",
"resolution": "PT1M",
"periodicity": "PT1M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-shf1",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_BIOMET_15MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "G_PLATE_1_1_1",
"time_column_name": "time",
"resolution": "PT15M",
"periodicity": "PT15M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-shf2",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_BIOMET_15MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "G_PLATE_1_1_2",
"time_column_name": "time",
"resolution": "PT15M",
"periodicity": "PT15M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-rn",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_BIOMET_1MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "T_nr_Avg",
"time_column_name": "time",
"resolution": "PT1M",
"periodicity": "PT1M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-swin",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_BIOMET_1MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "R_SW_in_Avg",
"time_column_name": "time",
"resolution": "PT1M",
"periodicity": "PT1M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-dynamic-sonic-azimuth",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_METADATA",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "sonic_azimuth",
"time_column_name": "timestamp",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-dynamic-height-measurement",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_METADATA",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "height_measurement",
"time_column_name": "timestamp",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-dynamic-height-canopy",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-level-0",
"source_dataset": "FLUX_METADATA",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "height_canopy",
"time_column_name": "timestamp",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "raw"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-ta-30min",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-met",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-processed",
"source_dataset": "FLUX_BIOMET_30MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "AirTemp_C",
"time_column_name": "time",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "processed"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-rh-30min",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-processed",
"source_dataset": "FLUX_BIOMET_30MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "RH",
"time_column_name": "time",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "processed"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-pa-30min",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-processed",
"source_dataset": "FLUX_BIOMET_30MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "Pressure_Avg",
"time_column_name": "time",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "processed"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-shf1-30min",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-processed",
"source_dataset": "FLUX_BIOMET_30MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "G_PLATE_1_1_1",
"time_column_name": "time",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "processed"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-shf2-30min",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-processed",
"source_dataset": "FLUX_BIOMET_30MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "G_PLATE_1_1_2",
"time_column_name": "time",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "processed"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-rn-30min",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-processed",
"source_dataset": "FLUX_BIOMET_30MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "T_nr_Avg",
"time_column_name": "time",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "processed"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-biomet-swin-30min",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
"platform_id": "http://fdri.ceh.ac.uk/id/platform/flux-plynl-ec",
"network": "fdri",
"source_bucket": "ukceh-fdri-staging-timeseries-processed",
"source_dataset": "FLUX_BIOMET_30MIN",
"source_site": "flux-plynl",
"source_site_identifier": "flux-plynl",
"source_column": "R_SW_in_Avg",
"time_column_name": "time",
"resolution": "PT30M",
"periodicity": "PT30M",
"processing_level": "processed"
},
{
"dataset_id": "http://fdri.ceh.ac.uk/id/dataset/flux-plynl-processed",
"site_id": "http://fdri.ceh.ac.uk/id/site/flux-plynl",
Expand All @@ -23,7 +278,7 @@
"source_bucket": "ukceh-fdri-staging-timeseries-processed",
"source_dataset": "Flux_processed",
"source_site": "flux-plynl",
"source_site_identifier": "PLYNL",
"source_site_identifier": "flux-plynl",
"source_column": null,
"time_column_name": "date",
"resolution": "PT30M",
Expand Down
Loading