Skip to content

adopt: workflow.source_path, retries, drop HTTPRemoteProvider #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
veprbl merged 2 commits into from
Nov 6, 2025
Merged

adopt: workflow.source_path, retries, drop HTTPRemoteProvider #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
057271b
adopt: workflow.source_path, retries, drop HTTPRemoteProvider
veprbl Nov 6, 2025
843339c
remove ENV_MODE
veprbl Nov 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 20 additions & 39 deletions _episodes/01-analysis_scripts_and_snakemake.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,27 +77,16 @@ In order to demonstrate the advantages of using snakefiles, let's start using th
First let's use snakemake to grab some simulation campaign files from the online storage space. In your `tutorial_directory/starting_script/` directory make a new file called `Snakefile`.
Open the file and add these lines:
```python
import os

# Set environment mode (local or eicweb)
ENV_MODE = os.getenv("ENV_MODE", "local") # Defaults to "local" if not set
# Output directory based on environment
OUTPUT_DIR = "../../sim_output/" if ENV_MODE == "eicweb" else "sim_output/"
# Benchmark directory based on environment
BENCH_DIR = "benchmarks/your_benchmark/" if ENV_MODE == "eicweb" else "./"

rule your_benchmark_campaign_reco_get:
output:
f"{OUTPUT_DIR}rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
f"sim_output/rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
shell: """
xrdcp root://dtn-eic.jlab.org//volatile/eic/EPIC/RECO/24.07.0/epic_craterlake/EXCLUSIVE/UCHANNEL_RHO/10x100/rho_10x100_uChannel_Q2of0to10_hiDiv.{wildcards.INDEX}.eicrecon.tree.edm4eic.root {output}
"""
```

If you're having trouble copying and pasting, you can also copy from [here](https://github.com/eic/tutorial-developing-benchmarks/blob/gh-pages/files/Snakefile).

Thinking ahead to when we want to put our benchmark on eicweb, we add this `ENV_MODE` variable which allows us to specify paths differently based on whether we're running locally or in GitLab's pipelines.

We also defined a new rule: `your_benchmark_campaign_reco_get`. This rule defines how to download a single file from the JLab servers to the location `sim_output`.

After saving the Snakefile, let's try running it.
Expand All @@ -120,18 +109,18 @@ But the benefits from using Snakemake become more apparent as the number of task
```python
rule your_benchmark_analysis:
input:
script=f"{BENCH_DIR}analysis/uchannelrho.cxx",
data=f"{OUTPUT_DIR}rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
script=workflow.source_path("analysis/uchannelrho.cxx"),
data=f"sim_output/rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
output:
plots=f"{OUTPUT_DIR}campaign_24.07.0_{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
plots=f"sim_output/campaign_24.07.0_{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
shell:
"""
mkdir -p $(dirname "{output.plots}")
root -l -b -q '{input.script}+("{input.data}","{output.plots}")'
"""
```

This rule runs an analysis script to create ROOT files containing plots. The rule uses the simulation campaign file downloaded from JLab as input data, and it runs the analysis script `uchannelrho.cxx`.
This rule runs an analysis script to create ROOT files containing plots. The rule uses the simulation campaign file downloaded from JLab as input data, and it runs the analysis script `uchannelrho.cxx`. Note that we use `workflow.source_path()` to reference the script - this function returns the correct path to the script relative to the Snakefile's location in the benchmark directory.

Now let's request the output file `"sim_output/campaign_24.07.0_0005.eicrecon.tree.edm4eic/plots.root"`. When we request this, Snakemake will identify that it needs to run the new `your_benchmark_analysis` rule. But in order to do this, it now needs a file we don't have: `sim_output/rho_10x100_uChannel_Q2of0to10_hiDiv.0005.eicrecon.tree.edm4eic.root` because we only downloaded the file with index `0000` already. What Snakemake will do automatically is recognize that in order to get that file, it first needs to run the `your_benchmark_campaign_reco_get` rule. It will do this first, and then circle back to the `your_benchmark_analysis` rule.

Expand All @@ -154,13 +143,13 @@ That's still not very impressive. Snakemake gets more useful when we want to run
rule your_benchmark_combine:
input:
lambda wildcards: expand(
f"{OUTPUT_DIR}campaign_24.07.0_{% raw %}{{INDEX:04d}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
f"sim_output/campaign_24.07.0_{% raw %}{{INDEX:04d}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
INDEX=range(int(wildcards.N)),
),
wildcard_constraints:
N="\d+",
output:
f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
shell:
"""
hadd {output} {input}
Expand All @@ -185,10 +174,10 @@ Now let's add one more rule to create benchmark plots:
```python
rule your_benchmark_plots:
input:
script=f"{BENCH_DIR}macros/plot_rho_physics_benchmark.C",
plots=f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
script=workflow.source_path("macros/plot_rho_physics_benchmark.C"),
plots=f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
output:
f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots_figures/benchmark_rho_mass.pdf",
f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots_figures/benchmark_rho_mass.pdf",
shell:
"""
if [ ! -d "{input.plots}_figures" ]; then
Expand Down Expand Up @@ -227,28 +216,20 @@ If we want to scale up the plots to include 15 simulation campaign files instead

The final Snakefile should look like this:
```python
import os

# Set environment mode (local or eicweb)
ENV_MODE = os.getenv("ENV_MODE", "local") # Defaults to "local" if not set
# Output directory based on environment
OUTPUT_DIR = "../../sim_output/" if ENV_MODE == "eicweb" else "sim_output/"
# Benchmark directory based on environment
BENCH_DIR = "benchmarks/your_benchmark/" if ENV_MODE == "eicweb" else "./"

rule your_benchmark_campaign_reco_get:
output:
f"{OUTPUT_DIR}rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
f"sim_output/rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
retries: 3
shell: """
xrdcp root://dtn-eic.jlab.org//volatile/eic/EPIC/RECO/24.07.0/epic_craterlake/EXCLUSIVE/UCHANNEL_RHO/10x100/rho_10x100_uChannel_Q2of0to10_hiDiv.{wildcards.INDEX}.eicrecon.tree.edm4eic.root {output}
"""

rule your_benchmark_analysis:
input:
script=f"{BENCH_DIR}analysis/uchannelrho.cxx",
data=f"{OUTPUT_DIR}rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
script=workflow.source_path("analysis/uchannelrho.cxx"),
data=f"sim_output/rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
output:
plots=f"{OUTPUT_DIR}campaign_24.07.0_{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
plots=f"sim_output/campaign_24.07.0_{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
shell:
"""
mkdir -p $(dirname "{output.plots}")
Expand All @@ -258,24 +239,24 @@ root -l -b -q '{input.script}+("{input.data}","{output.plots}")'
rule your_benchmark_combine:
input:
lambda wildcards: expand(
f"{OUTPUT_DIR}campaign_24.07.0_{% raw %}{{INDEX:04d}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
f"sim_output/campaign_24.07.0_{% raw %}{{INDEX:04d}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
INDEX=range(int(wildcards.N)),
),
wildcard_constraints:
N="\d+",
output:
f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
shell:
"""
hadd {output} {input}
"""

rule your_benchmark_plots:
input:
script=f"{BENCH_DIR}macros/plot_rho_physics_benchmark.C",
plots=f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
script=workflow.source_path("macros/plot_rho_physics_benchmark.C"),
plots=f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
output:
f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots_figures/benchmark_rho_mass.pdf",
f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots_figures/benchmark_rho_mass.pdf",
shell:
"""
if [ ! -d "{input.plots}_figures" ]; then
Expand Down
4 changes: 3 additions & 1 deletion _episodes/02-first_benchmark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ mkdir benchmarks/your_benchmark

The Continuous Integration system needs to know what steps it has to execute. This is specified using YAML files. Create a file `benchmarks/your_benchmark/config.yml`.

For a physics benchmark and to follow along with this tutorial, create a `config.yml` with the following contents:
For a physics benchmark, create a `config.yml` with the following contents:
~~~
your_benchmark:compile:
extends: .phy_benchmark
Expand All @@ -69,6 +69,8 @@ your_benchmark:simulate:
your_benchmark:results:
extends: .phy_benchmark
stage: collect
needs:
- ["your_benchmark:simulate"]
script:
- echo "I will collect results here!"

Expand Down
2 changes: 0 additions & 2 deletions _episodes/03-filling_out_your_benchmark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,6 @@ Create a new file: [`benchmarks/your_benchmark/setup.config`](https://github.com
#!/bin/bash
source strict-mode.sh

export ENV_MODE=eicweb

USE_SIMULATION_CAMPAIGN=true

N_EVENTS=100
Expand Down
27 changes: 10 additions & 17 deletions files/Snakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,19 @@
import os
from snakemake.remote.HTTP import RemoteProvider as HTTPRemoteProvider

# Set environment mode (local or eicweb)
ENV_MODE = os.getenv("ENV_MODE", "local") # Defaults to "local" if not set
# Output directory based on environment
OUTPUT_DIR = "../../sim_output/" if ENV_MODE == "eicweb" else "sim_output/"
# Benchmark directory based on environment
BENCH_DIR = "benchmarks/your_benchmark/" if ENV_MODE == "eicweb" else "./"

rule your_benchmark_campaign_reco_get:
output:
f"{OUTPUT_DIR}rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
f"sim_output/rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
retries: 3
shell: """
xrdcp root://dtn-eic.jlab.org//volatile/eic/EPIC/RECO/24.07.0/epic_craterlake/EXCLUSIVE/UCHANNEL_RHO/10x100/rho_10x100_uChannel_Q2of0to10_hiDiv.{wildcards.INDEX}.eicrecon.tree.edm4eic.root {output}
"""

rule your_benchmark_analysis:
input:
script=f"{BENCH_DIR}analysis/uchannelrho.cxx",
data=f"{OUTPUT_DIR}rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
script=workflow.source_path("analysis/uchannelrho.cxx"),
data=f"sim_output/rho_10x100_uChannel_Q2of0to10_hiDiv.{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic.root",
output:
plots=f"{OUTPUT_DIR}campaign_24.07.0_{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
plots=f"sim_output/campaign_24.07.0_{% raw %}{{INDEX}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
shell:
"""
mkdir -p $(dirname "{output.plots}")
Expand All @@ -30,24 +23,24 @@ root -l -b -q '{input.script}+("{input.data}","{output.plots}")'
rule your_benchmark_combine:
input:
lambda wildcards: expand(
f"{OUTPUT_DIR}campaign_24.07.0_{% raw %}{{INDEX:04d}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
f"sim_output/campaign_24.07.0_{% raw %}{{INDEX:04d}}{% endraw %}.eicrecon.tree.edm4eic/plots.root",
INDEX=range(int(wildcards.N)),
),
wildcard_constraints:
N="\d+",
output:
f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
shell:
"""
hadd {output} {input}
"""

rule your_benchmark_plots:
input:
script=f"{BENCH_DIR}macros/plot_rho_physics_benchmark.C",
plots=f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
script=workflow.source_path("macros/plot_rho_physics_benchmark.C"),
plots=f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots.root",
output:
f"{OUTPUT_DIR}campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots_figures/benchmark_rho_mass.pdf",
f"sim_output/campaign_24.07.0_combined_{% raw %}{{N}}{% endraw %}files.eicrecon.tree.edm4eic.plots_figures/benchmark_rho_mass.pdf",
shell:
"""
if [ ! -d "{input.plots}_figures" ]; then
Expand Down
2 changes: 0 additions & 2 deletions files/setup.config
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
#!/bin/bash
source strict-mode.sh

export ENV_MODE=eicweb

USE_SIMULATION_CAMPAIGN=true

N_EVENTS=100
Expand Down