diff --git a/.claude/skills/airtable-build-collection/SKILL.md b/.claude/skills/airtable-build-collection/SKILL.md index 260d1c9da..ce7911e5f 100644 --- a/.claude/skills/airtable-build-collection/SKILL.md +++ b/.claude/skills/airtable-build-collection/SKILL.md @@ -32,6 +32,7 @@ Key fields in the Datasets table: | `channel_0_marker` .. `channel_N_marker` | Protein marker for each channel | | `t_shape`, `c_shape`, `z_shape`, `y_shape`, `x_shape` | Array dimensions | | `pixel_size_xy_um`, `pixel_size_z_um` | Physical pixel sizes | +| `microscope` | Acquisition platform (e.g. `mantis_v1`, `mantis_v2`); derive from pixel size if absent | ## Usage @@ -155,6 +156,7 @@ experiments: moi: pixel_size_xy_um: pixel_size_z_um: + microscope: ``` Key notes: @@ -162,6 +164,10 @@ Key notes: - `perturbation_wells` uses `uninfected` / `` keys inferred from the `perturbation` field - `channels` lists only the channels needed for training (not all channels in the zarr) - `marker` at the experiment level is the primary marker for this experiment entry +- `microscope` labels the acquisition platform — always populate it. Use the Airtable microscope + field when present; otherwise derive from acquisition grid: `pixel_size_xy_um == 0.1494` → + `mantis_v1`, `0.1133` → `mantis_v2`. Downstream QC (embedding-consistency-qc) partitions + datasets by `obs["microscope"]`, so a missing/empty value silently collapses the v1/v2 split. ### Step 7: Save and Validate diff --git a/.gitignore b/.gitignore index c3f8ab9c9..49120a527 100644 --- a/.gitignore +++ b/.gitignore @@ -73,6 +73,3 @@ lightning_logs/ checkpoints/ plot_related/ - -# Local-only planning docs (not for upstream) -applications/dynaclr/docs/DAGs/evaluation_matrix.md diff --git a/applications/dynaclr/README.md b/applications/dynaclr/README.md index ae362a5bb..d4edf419d 100644 --- a/applications/dynaclr/README.md +++ b/applications/dynaclr/README.md @@ -48,6 +48,18 @@ uv run --package dynaclr viscy predict -c examples/configs/predict.yml sbatch examples/configs/fit_slurm.sh ``` +### End-to-end pipeline (new dataset → embeddings → eval) + +For the full "dataset → per-marker embeddings → downstream evals" workflow — including the +one-call `dynaclr predict-triplet` (writes the dataset-centric tree +`/2-phenotyping/predictions/{model}/{run}/{ckpt}/{marker}.zarr`), the decoupled +Nextflow `eval_from_embeddings` entry, and the **model matrix** for running many models through +train → predict → eval in parallel — see: + +- [`docs/DAGs/end_to_end.md`](docs/DAGs/end_to_end.md) — the pipeline overview + Quickstart runbook +- [`nextflow/README.md`](nextflow/README.md) — the two Nextflow eval entries +- [`tools/README.md`](tools/README.md) — the model matrix (`dynaclr run-matrix`) + launchers + The YAML config determines which model and data module to use via `class_path`: ```yaml @@ -78,12 +90,21 @@ DynaCLR also provides evaluation-specific commands via `dynaclr `: | Command | Description | |---------|-------------| +| `predict-triplet` | One-call per-marker embedding inference from a collection + checkpoint → dataset-centric tree | +| `predict-batch` | Run `predict-triplet` for one model with AI-readiness preflight | +| `eval` | Evaluate frozen embeddings selected by model/run/checkpoint | +| `run-matrix` | Chain train → predict → eval for many models through SLURM dependencies | +| `split-embeddings` | Split a combined embeddings zarr per experiment/marker (`--route-by-dataset` → the dataset tree) | +| `embedding-consistency-qc` | Build per-marker cross-dataset MMD² and correlation matrices | +| `run-linear-classifiers` | Train linear classifiers on embeddings (batch, CSV metrics) | +| `append-annotations` / `append-predictions` | Join annotation / predicted-label columns onto per-experiment zarrs | | `train-linear-classifier` | Train a linear classifier on cell embeddings | | `apply-linear-classifier` | Apply a trained linear classifier to new embeddings | | `append-obs` | Append columns from a CSV to an AnnData zarr obs (with optional prefix, e.g. `annotated_`, `feature_`) | | `reduce-dimensionality` | Compute PCA, UMAP, and/or PHATE on saved embeddings | | `evaluate-smoothness` | Evaluate temporal smoothness of embedding models | | `compare-models` | Compare previously saved smoothness results | +| `compute-mmd` | MMD between perturbation / experiment groups (cross-experiment consistency) | | `info` | Print summary of an AnnData zarr store | ```bash diff --git a/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2024_11_07_A549_SEC61_DENV.yml b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2024_11_07_A549_SEC61_DENV.yml new file mode 100644 index 000000000..d6ccbdee9 --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2024_11_07_A549_SEC61_DENV.yml @@ -0,0 +1,41 @@ +name: organelle-box-2024_11_07_A549_SEC61_DENV +description: "Profiling/handoff collection: 2024_11_07 A549 SEC61 DENV (single-marker + organelle box). Channels embedded as bag-of-channels samples: SEC61B (GFP), + viral_sensor (mCherry), Phase3D. Prepared zarr on VAST with timepoint_statistics + + focus_slice. Sourced from Airtable base app8vqaoWyOwa0sB5 (dataset + '2024_11_07_A549_SEC61_DENV'). FOV C/1/000000 exists in the zarr but has no tracking + CSV and is excluded at build time." +datasets_root: /hpc/projects/organelle_phenotyping/datasets + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + airtable_query: "{dataset}=\"2024_11_07_A549_SEC61_DENV\"" + created_by: "eduardo.hirata" + +experiments: + - name: 2024_11_07_A549_SEC61_DENV + data_path: ${datasets_root}/2024_11_07_A549_SEC61_DENV/2024_11_07_A549_SEC61_DENV.zarr + tracks_path: ${datasets_root}/2024_11_07_A549_SEC61_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + - name: raw mCherry EX561 EM600-37 + marker: viral_sensor + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: + - B/1 + - B/3 + DENV: + - B/2 + - C/2 + exclude_fovs: + - C/1/000000 + interval_minutes: 10.0 + start_hpi: 4.0 + marker: SEC61B + organelle: endoplasmic_reticulum + moi: 5.0 + pixel_size_xy_um: 0.1494 + pixel_size_z_um: 0.174 diff --git a/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_08_A549_G3BP1_ZIKV.yml b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_08_A549_G3BP1_ZIKV.yml new file mode 100644 index 000000000..ac2b060fd --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_08_A549_G3BP1_ZIKV.yml @@ -0,0 +1,38 @@ +name: organelle-box-2026_04_08_A549_G3BP1_ZIKV +description: "Embed-only collection: 2026_04_08 A549 G3BP1 ZIKV (single-marker organelle box). + Channels embedded as bag-of-channels samples: G3BP1 (GFP), viral_sensor (mCherry), Phase3D. + Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations." +datasets_root: /hpc/projects/organelle_phenotyping/datasets + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + airtable_query: "SEARCH(\"2026_04_08_A549_G3BP1_ZIKV\", {dataset})" + created_by: "eduardo.hirata" + +experiments: + - name: 2026_04_08_A549_G3BP1_ZIKV + data_path: ${datasets_root}/2026_04_08_A549_G3BP1_ZIKV/2026_04_08_A549_G3BP1_ZIKV.zarr + tracks_path: ${datasets_root}/2026_04_08_A549_G3BP1_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + - name: raw mCherry EX561 EM600-37 + marker: viral_sensor + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: + - D/6 + ZIKV: + - B/5 + - B/6 + - C/5 + - C/6 + - D/5 + interval_minutes: 30.0 + start_hpi: 3.0 + marker: G3BP1 + organelle: stress_granules + moi: 5.0 + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 diff --git a/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_10_A549_TOMM20_ZIKV.yml b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_10_A549_TOMM20_ZIKV.yml new file mode 100644 index 000000000..db0f5ffe4 --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_10_A549_TOMM20_ZIKV.yml @@ -0,0 +1,38 @@ +name: organelle-box-2026_04_10_A549_TOMM20_ZIKV +description: "Embed-only collection: 2026_04_10 A549 TOMM20 ZIKV (single-marker organelle box). + Channels embedded as bag-of-channels samples: TOMM20 (GFP), viral_sensor (mCherry), Phase3D. + Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations." +datasets_root: /hpc/projects/organelle_phenotyping/datasets + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + airtable_query: "SEARCH(\"2026_04_10_A549_TOMM20_ZIKV\", {dataset})" + created_by: "eduardo.hirata" + +experiments: + - name: 2026_04_10_A549_TOMM20_ZIKV + data_path: ${datasets_root}/2026_04_10_A549_TOMM20_ZIKV/2026_04_10_A549_TOMM20_ZIKV.zarr + tracks_path: ${datasets_root}/2026_04_10_A549_TOMM20_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: TOMM20 + - name: raw mCherry EX561 EM600-37 + marker: viral_sensor + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: + - A/2 + ZIKV: + - A/3 + - A/4 + - B/2 + - B/3 + - B/4 + interval_minutes: 30.0 + start_hpi: 5.0 + marker: TOMM20 + organelle: mitochondria + moi: 5.0 + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 diff --git a/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_14_A549_SEC61B_DENV.yml b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_14_A549_SEC61B_DENV.yml new file mode 100644 index 000000000..e653fd307 --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_14_A549_SEC61B_DENV.yml @@ -0,0 +1,38 @@ +name: organelle-box-2026_04_14_A549_SEC61B_DENV +description: "Embed-only collection: 2026_04_14 A549 SEC61B DENV (single-marker organelle box). + Channels embedded as bag-of-channels samples: SEC61B (GFP), viral_sensor (mCherry), Phase3D. + Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations." +datasets_root: /hpc/projects/organelle_phenotyping/datasets + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + airtable_query: "SEARCH(\"2026_04_14_A549_SEC61B_DENV\", {dataset})" + created_by: "eduardo.hirata" + +experiments: + - name: 2026_04_14_A549_SEC61B_DENV + data_path: ${datasets_root}/2026_04_14_A549_SEC61B_DENV/2026_04_14_A549_SEC61B_DENV.zarr + tracks_path: ${datasets_root}/2026_04_14_A549_SEC61B_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + - name: raw mCherry EX561 EM600-37 + marker: viral_sensor + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: + - C/2 + DENV: + - C/3 + - C/4 + - D/2 + - D/3 + - D/4 + interval_minutes: 30.0 + start_hpi: 3.0 + marker: SEC61B + organelle: endoplasmic_reticulum + moi: 5.0 + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 diff --git a/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_21_A549_G3BP1_DENV.yml b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_21_A549_G3BP1_DENV.yml new file mode 100644 index 000000000..f5b0329d0 --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_21_A549_G3BP1_DENV.yml @@ -0,0 +1,41 @@ +name: organelle-box-2026_04_21_A549_G3BP1_DENV +description: "Embed-only collection: 2026_04_21 A549 G3BP1 DENV (single-marker organelle box). + Channels embedded as bag-of-channels samples: G3BP1 (GFP), viral_sensor (mCherry), Phase3D. + Source zarrs are the intracellular_dashboard preprocess outputs (5-assemble data + + 4-track ultrack labels); no prepared VAST copy exists. Predict uses fov_statistics + normalization, so timepoint_statistics is not required. Annotations exist (infection, + organelle, cell_division, cell_death)." +datasets_root: /hpc/projects/intracellular_dashboard/organelle_dynamics + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + airtable_query: "SEARCH(\"2026_04_21_A549_G3BP1_DENV\", {dataset})" + created_by: "eduardo.hirata" + +experiments: + - name: 2026_04_21_A549_G3BP1_DENV + data_path: ${datasets_root}/2026_04_21_A549_G3BP1_DENV/1-preprocess/5-assemble/2026_04_21_G3BP1_DENV.zarr + tracks_path: ${datasets_root}/2026_04_21_A549_G3BP1_DENV/1-preprocess/4-track/2026_04_21_G3BP1_DENV.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + - name: raw mCherry EX561 EM600-37 + marker: viral_sensor + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: + - A/1 + DENV: + - A/2 + - A/3 + - B/1 + - B/2 + - B/3 + interval_minutes: 30.0 + start_hpi: 3.0 + marker: G3BP1 + organelle: stress_granules + moi: 5.0 + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 diff --git a/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_28_A549_SEC61B_DENV.yml b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_28_A549_SEC61B_DENV.yml new file mode 100644 index 000000000..b9bbf7e96 --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_28_A549_SEC61B_DENV.yml @@ -0,0 +1,38 @@ +name: organelle-box-2026_04_28_A549_SEC61B_DENV +description: "Embed-only collection: 2026_04_28 A549 SEC61B DENV (single-marker organelle box). + Channels embedded as bag-of-channels samples: SEC61B (GFP), viral_sensor (mCherry), Phase3D. + Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations." +datasets_root: /hpc/projects/organelle_phenotyping/datasets + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + airtable_query: "SEARCH(\"2026_04_28_A549_SEC61B_DENV\", {dataset})" + created_by: "eduardo.hirata" + +experiments: + - name: 2026_04_28_A549_SEC61B_DENV + data_path: ${datasets_root}/2026_04_28_A549_SEC61B_DENV/2026_04_28_A549_SEC61B_DENV.zarr + tracks_path: ${datasets_root}/2026_04_28_A549_SEC61B_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + - name: raw mCherry EX561 EM600-37 + marker: viral_sensor + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: + - A/2 + DENV: + - A/3 + - A/4 + - B/2 + - B/3 + - B/4 + interval_minutes: 30.0 + start_hpi: 3.0 + marker: SEC61B + organelle: endoplasmic_reticulum + moi: 5.0 + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 diff --git a/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_30_A549_TOMM20_DENV.yml b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_30_A549_TOMM20_DENV.yml new file mode 100644 index 000000000..e2e8bf159 --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_30_A549_TOMM20_DENV.yml @@ -0,0 +1,38 @@ +name: organelle-box-2026_04_30_A549_TOMM20_DENV +description: "Embed-only collection: 2026_04_30 A549 TOMM20 DENV (single-marker organelle box). + Channels embedded as bag-of-channels samples: TOMM20 (GFP), viral_sensor (mCherry), Phase3D. + Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations." +datasets_root: /hpc/projects/organelle_phenotyping/datasets + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + airtable_query: "SEARCH(\"2026_04_30_A549_TOMM20_DENV\", {dataset})" + created_by: "eduardo.hirata" + +experiments: + - name: 2026_04_30_A549_TOMM20_DENV + data_path: ${datasets_root}/2026_04_30_A549_TOMM20_DENV/2026_04_30_A549_TOMM20_DENV.zarr + tracks_path: ${datasets_root}/2026_04_30_A549_TOMM20_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: TOMM20 + - name: raw mCherry EX561 EM600-37 + marker: viral_sensor + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: + - A/2 + DENV: + - A/3 + - A/4 + - B/2 + - B/3 + - B/4 + interval_minutes: 30.0 + start_hpi: 4.0 + marker: TOMM20 + organelle: mitochondria + moi: 5.0 + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 diff --git a/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV.yml b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV.yml new file mode 100644 index 000000000..f25d60de9 --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV.yml @@ -0,0 +1,54 @@ +name: organelle-box-2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV +description: "Embed-only collection: 2026_07_01 A549 SEC61B/TOMM20/G3BP1 ZIKV (multi-organelle box). + Bag-of-channels DynaCLR-2D-MIP embedding. The GFP reporter varies by plate column + (col 2=SEC61B, col 3=TOMM20, col 4=G3BP1); mCherry is the pAL17 viral sensor in every + well. Per-reporter embeddings are produced by subsetting wells (fit_include_wells) on + the GFP channel. Row A=uninfected, row B=ZIKV (MOI 5, 3 hpp). + Prepared zarr on VAST with normalization + focus_slice. No annotations." +datasets_root: /hpc/projects/organelle_phenotyping/datasets + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + airtable_query: "SEARCH(\"2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV\", {dataset})" + created_by: "eduardo.hirata" + +experiments: + - name: 2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV + data_path: ${datasets_root}/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV.zarr + tracks_path: ${datasets_root}/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/tracking.zarr + # Per-reporter embeddings (bag-of-channels, in_channels=1): one channel entry + # per reporter. The GFP reporter varies by plate column, so the same zarr + # channel appears once per organelle with its own `wells` (biologically valid + # wells; empty = all wells). `dynaclr predict-triplet` runs predict once per + # entry, restricting to `wells` via fit_include_wells, writing + # embeddings/{dataset}_{marker}.zarr. Start_hpi 3.0 (3 hpp). + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + wells: [A/2, B/2] + - name: raw GFP EX488 EM525-45 + marker: TOMM20 + wells: [A/3, B/3] + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + wells: [A/4, B/4] + - name: raw mCherry EX561 EM600-37 + marker: pAL17 + wells: [A/2, A/3, A/4, B/2, B/3, B/4] + - name: Phase3D + marker: Phase3D + wells: [A/2, A/3, A/4, B/2, B/3, B/4] + perturbation_wells: + uninfected: + - A/2 + - A/3 + - A/4 + ZIKV: + - B/2 + - B/3 + - B/4 + interval_minutes: 30.0 + start_hpi: 3.0 + moi: 5.0 + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 diff --git a/applications/dynaclr/configs/collections/organelle_remodeling.yml b/applications/dynaclr/configs/collections/organelle_remodeling.yml new file mode 100644 index 000000000..b1ef610de --- /dev/null +++ b/applications/dynaclr/configs/collections/organelle_remodeling.yml @@ -0,0 +1,376 @@ +name: organelle_remodeling +description: "Merged organelle-remodeling collection: SEC61B (ER), TOMM20 (mitochondria), + G3BP1 (stress granules), CAAX (membrane), and H2B (nuclear) reporters under + ZIKV/DENV infection. One experiment per physical dataset, including five annotated + Mantis v1 datasets from 2025. All are prepared on VAST (normalization + focus_slice). + mCherry viral sensor marker matches Airtable channel_2_marker: pAL17 for ZIKV, + pAL40 for DENV. GFP reporter per Airtable. + 2026_04_21_A549_G3BP1_DENV re-concatenated to a flat store on VAST (odd 5-assemble + layout resolved); focus_slice has a few z_focus=0 frames on 12/54 FOVs (mean 2.4 t each). + Used by the organelle_remodeling model matrix (DynaCLR, MorphEm, CELL-DINO, and + DINOv3), which fans out predict per experiment and skips already-embedded datasets." +datasets_root: /hpc/projects/organelle_phenotyping/datasets + +provenance: + airtable_base_id: app8vqaoWyOwa0sB5 + created_by: "eduardo.hirata" + +experiments: + # --- DENV, SEC61B (ER) --- + - name: 2024_11_07_A549_SEC61_DENV + data_path: ${datasets_root}/2024_11_07_A549_SEC61_DENV/2024_11_07_A549_SEC61_DENV.zarr + tracks_path: ${datasets_root}/2024_11_07_A549_SEC61_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + - name: raw mCherry EX561 EM600-37 + marker: pAL40 + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [B/1, B/3] + DENV: [B/2, C/2] + exclude_fovs: [C/1/000000] + interval_minutes: 10.0 + start_hpi: 4.0 + moi: 5.0 + organelle: endoplasmic_reticulum + pixel_size_xy_um: 0.1494 + pixel_size_z_um: 0.174 + microscope: mantis_v1 + + - name: 2026_04_14_A549_SEC61B_DENV + data_path: ${datasets_root}/2026_04_14_A549_SEC61B_DENV/2026_04_14_A549_SEC61B_DENV.zarr + tracks_path: ${datasets_root}/2026_04_14_A549_SEC61B_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + - name: raw mCherry EX561 EM600-37 + marker: pAL40 + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [C/2] + DENV: [C/3, C/4, D/2, D/3, D/4] + interval_minutes: 30.0 + start_hpi: 3.0 + moi: 5.0 + organelle: endoplasmic_reticulum + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 + microscope: mantis_v2 + + - name: 2026_04_28_A549_SEC61B_DENV + data_path: ${datasets_root}/2026_04_28_A549_SEC61B_DENV/2026_04_28_A549_SEC61B_DENV.zarr + tracks_path: ${datasets_root}/2026_04_28_A549_SEC61B_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + - name: raw mCherry EX561 EM600-37 + marker: pAL40 + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [A/2] + DENV: [A/3, A/4, B/2, B/3, B/4] + interval_minutes: 30.0 + start_hpi: 3.0 + moi: 5.0 + organelle: endoplasmic_reticulum + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 + microscope: mantis_v2 + + # --- ZIKV, SEC61B (ER) --- + - name: 2026_03_24_A549_SEC61_ZIKV + data_path: ${datasets_root}/2026_03_24_A549_SEC61_ZIKV/2026_03_24_A549_SEC61_ZIKV.zarr + tracks_path: ${datasets_root}/2026_03_24_A549_SEC61_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + - name: raw mCherry EX561 EM600-37 + marker: pAL17 + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [A/2] + ZIKV: [A/3, A/4, B/2, B/3, B/4] + interval_minutes: 30.0 + start_hpi: 5.0 + moi: 5.0 + organelle: endoplasmic_reticulum + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 + microscope: mantis_v2 + + # --- ZIKV, G3BP1 (stress granules) --- + - name: 2026_04_08_A549_G3BP1_ZIKV + data_path: ${datasets_root}/2026_04_08_A549_G3BP1_ZIKV/2026_04_08_A549_G3BP1_ZIKV.zarr + tracks_path: ${datasets_root}/2026_04_08_A549_G3BP1_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + - name: raw mCherry EX561 EM600-37 + marker: pAL17 + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [D/6] + ZIKV: [B/5, B/6, C/5, C/6, D/5] + interval_minutes: 30.0 + start_hpi: 3.0 + moi: 5.0 + organelle: stress_granules + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 + microscope: mantis_v2 + + # --- DENV, G3BP1 (stress granules) --- + # zarr stem is 2026_04_21_G3BP1_DENV (no A549), unlike the dir name. + - name: 2026_04_21_A549_G3BP1_DENV + data_path: ${datasets_root}/2026_04_21_A549_G3BP1_DENV/2026_04_21_G3BP1_DENV.zarr + tracks_path: ${datasets_root}/2026_04_21_A549_G3BP1_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + - name: raw mCherry EX561 EM600-37 + marker: pAL40 + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [A/1] + DENV: [A/2, A/3, B/1, B/2, B/3] + interval_minutes: 30.0 + start_hpi: 3.0 + moi: 5.0 + organelle: stress_granules + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 + microscope: mantis_v2 + + # --- ZIKV, TOMM20 (mitochondria) --- + - name: 2026_04_10_A549_TOMM20_ZIKV + data_path: ${datasets_root}/2026_04_10_A549_TOMM20_ZIKV/2026_04_10_A549_TOMM20_ZIKV.zarr + tracks_path: ${datasets_root}/2026_04_10_A549_TOMM20_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: TOMM20 + - name: raw mCherry EX561 EM600-37 + marker: pAL17 + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [A/2] + ZIKV: [A/3, A/4, B/2, B/3, B/4] + interval_minutes: 30.0 + start_hpi: 5.0 + moi: 5.0 + organelle: mitochondria + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 + microscope: mantis_v2 + + # --- DENV, TOMM20 (mitochondria) --- + - name: 2026_04_30_A549_TOMM20_DENV + data_path: ${datasets_root}/2026_04_30_A549_TOMM20_DENV/2026_04_30_A549_TOMM20_DENV.zarr + tracks_path: ${datasets_root}/2026_04_30_A549_TOMM20_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: TOMM20 + - name: raw mCherry EX561 EM600-37 + marker: pAL40 + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [A/2] + DENV: [A/3, A/4, B/2, B/3, B/4] + interval_minutes: 30.0 + start_hpi: 4.0 + moi: 5.0 + organelle: mitochondria + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 + microscope: mantis_v2 + + # --- ZIKV, multi-organelle box (GFP reporter varies per column) --- + - name: 2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV + data_path: ${datasets_root}/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV.zarr + tracks_path: ${datasets_root}/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + wells: [A/2, B/2] + - name: raw GFP EX488 EM525-45 + marker: TOMM20 + wells: [A/3, B/3] + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + wells: [A/4, B/4] + - name: raw mCherry EX561 EM600-37 + marker: pAL17 + wells: [A/2, A/3, A/4, B/2, B/3, B/4] + - name: Phase3D + marker: Phase3D + wells: [A/2, A/3, A/4, B/2, B/3, B/4] + perturbation_wells: + uninfected: [A/2, A/3, A/4] + ZIKV: [B/2, B/3, B/4] + interval_minutes: 30.0 + start_hpi: 3.0 + moi: 5.0 + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.16995 + microscope: mantis_v2 + + # --- DENV + ZIKV on one plate, CAAX (membrane) + H2B (nuclear); NO viral sensor --- + # Distinct marker set: mCherry=CAAX, Cy5=H2B. B/2 uninfected, B/3 DENV, B/4 ZIKV. + - name: 2026_03_26_A549_CAAX_H2B_DENV_ZIKV + data_path: ${datasets_root}/2026_03_26_A549_CAAX_H2B_DENV_ZIKV/2026_03_26_A549_CAAX_H2B_DENV_ZIKV.zarr + tracks_path: ${datasets_root}/2026_03_26_A549_CAAX_H2B_DENV_ZIKV/tracking.zarr + channels: + - name: raw mCherry EX561 EM600-37 + marker: CAAX + - name: raw Cy5 EX639 EM698-70 + marker: H2B + - name: Phase3D + marker: Phase3D + perturbation_wells: + uninfected: [B/2] + DENV: [B/3] + ZIKV: [B/4] + interval_minutes: 120.0 + start_hpi: 5.0 + moi: 5.0 + organelle: membrane_nuclear + pixel_size_xy_um: 0.1133 + pixel_size_z_um: 0.174 + microscope: mantis_v2 + + # --- Annotated Mantis v1 datasets from 2025 --- + # Wells are scoped to the marker/perturbation mappings already curated in the + # DynaCLR v3 and linear-classifier evaluation collections. + - name: 2025_01_24_A549_G3BP1_DENV + data_path: ${datasets_root}/2025_01_24_A549_G3BP1_DENV/2025_01_24_A549_G3BP1_DENV.zarr + tracks_path: ${datasets_root}/2025_01_24_A549_G3BP1_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + wells: [B/1, B/2, B/3, C/2] + - name: raw mCherry EX561 EM600-37 + marker: pAL40 + wells: [B/3, C/2] + - name: Phase3D + marker: Phase3D + wells: [B/1, B/2, B/3, C/2] + perturbation_wells: + uninfected: [B/1, B/3] + DENV: [B/2, C/2] + interval_minutes: 30.0 + start_hpi: 4.0 + moi: 5.0 + organelle: stress_granules + pixel_size_xy_um: 0.1494 + pixel_size_z_um: 0.174 + microscope: mantis_v1 + + - name: 2025_01_28_A549_G3BP1_ZIKV_DENV + data_path: ${datasets_root}/2025_01_28_A549_G3BP1_ZIKV_DENV/2025_01_28_A549_G3BP1_ZIKV_DENV.zarr + tracks_path: ${datasets_root}/2025_01_28_A549_G3BP1_ZIKV_DENV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + wells: [B/4, C/4] + - name: raw mCherry EX561 EM600-37 + marker: pAL40 + wells: [B/4, C/4] + - name: Phase3D + marker: Phase3D + wells: [B/4, C/4] + perturbation_wells: + uninfected: [B/4] + DENV: [C/4] + interval_minutes: 30.0 + start_hpi: 4.0 + moi: 5.0 + organelle: stress_granules + pixel_size_xy_um: 0.1494 + pixel_size_z_um: 0.174 + microscope: mantis_v1 + + - name: 2025_07_22_A549_SEC61_TOMM20_G3BP1_ZIKV + data_path: ${datasets_root}/2025_07_22_A549_SEC61_TOMM20_G3BP1_ZIKV/2025_07_22_A549_SEC61_TOMM20_G3BP1_ZIKV.zarr + tracks_path: ${datasets_root}/2025_07_22_A549_SEC61_TOMM20_G3BP1_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + wells: [C/1, C/2] + - name: raw mCherry EX561 EM600-37 + marker: pAL17 + wells: [C/1, C/2] + - name: Phase3D + marker: Phase3D + wells: [C/1, C/2] + perturbation_wells: + uninfected: [C/1] + ZIKV: [C/2] + interval_minutes: 30.0 + start_hpi: 3.5 + moi: 5.0 + organelle: stress_granules + pixel_size_xy_um: 0.1494 + pixel_size_z_um: 0.174 + microscope: mantis_v1 + + - name: 2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV + data_path: ${datasets_root}/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV.zarr + tracks_path: ${datasets_root}/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + wells: [A/1, A/2] + - name: raw GFP EX488 EM525-45 + marker: TOMM20 + wells: [B/1, B/2] + - name: raw GFP EX488 EM525-45 + marker: G3BP1 + wells: [C/1, C/2] + - name: raw mCherry EX561 EM600-37 + marker: pAL17 + wells: [A/1, A/2, B/1, B/2, C/1, C/2] + - name: Phase3D + marker: Phase3D + wells: [A/1, A/2, B/1, B/2, C/1, C/2] + perturbation_wells: + uninfected: [A/1, B/1, C/1] + ZIKV: [A/2, B/2, C/2] + interval_minutes: 30.0 + start_hpi: 3.5 + moi: 5.0 + organelle: multi_organelle + pixel_size_xy_um: 0.1494 + pixel_size_z_um: 0.174 + microscope: mantis_v1 + + - name: 2025_08_26_A549_SEC61_TOMM20_ZIKV + data_path: ${datasets_root}/2025_08_26_A549_SEC61_TOMM20_ZIKV/2025_08_26_A549_SEC61_TOMM20_ZIKV.zarr + tracks_path: ${datasets_root}/2025_08_26_A549_SEC61_TOMM20_ZIKV/tracking.zarr + channels: + - name: raw GFP EX488 EM525-45 + marker: SEC61B + wells: [A/1, B/1] + - name: raw mCherry EX561 EM600-37 + marker: pAL17 + wells: [A/1, B/1] + - name: Phase3D + marker: Phase3D + wells: [A/1, B/1] + perturbation_wells: + uninfected: [A/1] + ZIKV: [B/1] + interval_minutes: 30.0 + start_hpi: 3.5 + moi: 5.0 + organelle: endoplasmic_reticulum + pixel_size_xy_um: 0.1494 + pixel_size_z_um: 0.174 + microscope: mantis_v1 diff --git a/applications/dynaclr/configs/evaluation/CELL-DINO-frozen/infectomics-annotated.yaml b/applications/dynaclr/configs/evaluation/CELL-DINO-frozen/infectomics-annotated.yaml new file mode 100644 index 000000000..39890bcd1 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/CELL-DINO-frozen/infectomics-annotated.yaml @@ -0,0 +1,37 @@ +# Evaluation config: CELL-DINO-frozen × infectomics-annotated (Wave 1, baseline). +# +# CELL-DINO-frozen uses the channel-adaptive ViT-L/16 weights pretrained on +# Human Protein Atlas single-cell fluorescence — biology-trained but +# *non-temporal* and *non-DynaCLR*. Tests whether microscopy SSL pretraining +# alone matches DynaCLR's contrastive recipe. +# +# `ckpt_path: null` — CELL-DINO weights are loaded from the local .pth in +# CellDinoModel.__init__. The eval orchestrator omits `ckpt_path` from the +# generated predict YAML so Lightning skips checkpoint restoration. +# +# Submit: +# sbatch applications/dynaclr/configs/evaluation/CELL-DINO-frozen/run_infectomics_annotated.sh + +base: + - ../recipes/predict.yml + - ../recipes/reduce.yml + - ../recipes/plot_infectomics.yml + - ../recipes/linear_classifiers_infectomics.yml + - ../recipes/infectomics-annotated.yml + +training_config: /hpc/mydata/eduardo.hirata/repos/viscy/applications/dynaclr/configs/evaluation/CELL-DINO-frozen/training_config_cell_dino_frozen.yaml +ckpt_path: null +output_dir: /hpc/projects/organelle_phenotyping/models/CELL-DINO-frozen/evaluations/infectomics-annotated/ + +# CELL-DINO ViT-L/16 is 302M params (~10× ConvNeXt-tiny). Lower batch_size +# from the recipe default (400) to keep activations within a single H100/A100. +# 1024-dim cls embeddings (vs 768) also push downstream PHATE memory up. +predict: + batch_size: 64 + num_workers: 4 + precision: 32-true + devices: 1 + +# Publish to its own sub-registry so the LC bundle is identifiable. +linear_classifiers: + publish_dir: /hpc/projects/organelle_phenotyping/models/linear_classifiers/CELL-DINO-frozen/infectomics/ diff --git a/applications/dynaclr/configs/evaluation/CELL-DINO-frozen/training_config_cell_dino_frozen.yaml b/applications/dynaclr/configs/evaluation/CELL-DINO-frozen/training_config_cell_dino_frozen.yaml new file mode 100644 index 000000000..1c5559fbd --- /dev/null +++ b/applications/dynaclr/configs/evaluation/CELL-DINO-frozen/training_config_cell_dino_frozen.yaml @@ -0,0 +1,190 @@ +# Synthetic "training_config" for CELL-DINO-frozen evaluation baseline. +# +# CELL-DINO-frozen uses the channel-adaptive ViT-L/16 weights pretrained on +# Human Protein Atlas single-cell fluorescence (Mougenot et al.). No +# contrastive fine-tuning, no trainable projection head. Embeddings are the +# 1024-dim cls token mean-pooled across input channels. +# +# This file is consumed by `dynaclr prepare-eval-configs` to generate a +# Lightning predict YAML. There is no Lightning checkpoint — the leaf YAML +# sets `ckpt_path: null`, so the orchestrator omits `ckpt_path` from the +# predict YAML, and CellDinoModel.__init__ loads weights from the local +# .pth file. +# +# Mirrors DINOv3-frozen/training_config_dinov3_frozen.yaml byte-for-byte +# except the encoder class_path/init_args. Same dataloader, same +# preprocessing → apples-to-apples comparison against DINOv3 frozen. + +# lightning.pytorch==2.6.1 +seed_everything: 42 +trainer: + accelerator: gpu + strategy: auto + devices: 1 + num_nodes: 1 + precision: 32-true + fast_dev_run: false + max_epochs: 100 + log_every_n_steps: 10 + enable_checkpointing: false + accumulate_grad_batches: 1 + inference_mode: true +model: + class_path: dynaclr.engine.ContrastiveModule + init_args: + encoder: + class_path: viscy_models.foundation.cell_dino.CellDinoModel + init_args: + weights_path: /hpc/projects/organelle_phenotyping/models/CELL-DINO/model_weights/weights/channel_adaptive_dino_vitl16_pretrain_cells-ef7c17ff.pth + img_size: 224 + patch_size: 16 + freeze: true + projection: null + loss_function: + class_path: pytorch_metric_learning.losses.NTXentLoss + init_args: + temperature: 0.5 + embedding_regularizer: null + embedding_reg_weight: 1 + reducer: null + distance: null + collect_stats: null + lr: 0.0001 + schedule: Constant + log_batches_per_epoch: 3 + log_samples_per_batch: 3 + log_embeddings_every_n_epochs: 10 + pca_color_keys: condition + log_negative_metrics_every_n_epochs: 2 + example_input_array_shape: + - 1 + - 1 + - 1 + - 160 + - 160 + ckpt_path: null + freeze_backbone: false + projection: null +data: + class_path: dynaclr.data.datamodule.MultiExperimentDataModule + init_args: + cell_index_path: /hpc/projects/intracellular_dashboard/organelle_dynamics/models/DINOv3-temporal-MLP-2D-BagOfChannels/DynaCLR-2D-BagOfChannels-v3.parquet + z_window: 1 + yx_patch_size: + - 256 + - 256 + final_yx_patch_size: + - 160 + - 160 + val_experiments: [] + split_ratio: 0.8 + tau_range: + - 0.5 + - 2.0 + tau_decay_rate: 0.0 + batch_size: 256 + num_workers: 1 + stratify_by: + - condition + leaky: 0.0 + temporal_enrichment: false + temporal_window_hours: 2.0 + temporal_global_fraction: 0.3 + channels_per_sample: 1 + channel_dropout_channels: null + channel_dropout_prob: 0.0 + normalizations: + - class_path: viscy_transforms.ScaleIntensityRangePercentilesd + init_args: + keys: + - channel_0 + lower: 50.0 + upper: 99.0 + b_min: 0.0 + b_max: 1.0 + clip: false + relative: false + channel_wise: false + allow_missing_keys: false + augmentations: + - class_path: viscy_transforms.BatchedRandAffined + init_args: + keys: + - channel_0 + prob: 0.8 + rotate_range: + - 3.14 + - 0.0 + - 0.0 + shear_range: + - 0.05 + - 0.05 + - 0.0 + - 0.05 + - 0.0 + - 0.05 + translate_range: null + scale_range: + - - 0.8 + - 1.2 + - - 0.8 + - 1.2 + - - 0.8 + - 1.2 + mode: bilinear + allow_missing_keys: false + - class_path: viscy_transforms.BatchedRandAdjustContrastd + init_args: + keys: + - channel_0 + gamma: + - 0.8 + - 1.2 + prob: 0.5 + invert_image: false + retain_stats: false + allow_missing_keys: false + - class_path: viscy_transforms.BatchedRandScaleIntensityd + init_args: + keys: + - channel_0 + factors: 0.5 + prob: 0.5 + channel_wise: false + allow_missing_keys: false + - class_path: viscy_transforms.BatchedRandGaussianSmoothd + init_args: + keys: + - channel_0 + sigma_x: + - 0.25 + - 0.75 + sigma_y: + - 0.25 + - 0.75 + sigma_z: + - 0.0 + - 0.0 + truncated: 4.0 + prob: 0.5 + border_type: constant + allow_missing_keys: false + - class_path: viscy_transforms.BatchedRandGaussianNoised + init_args: + keys: + - channel_0 + prob: 0.5 + mean: 0.0 + std: 0.2 + allow_missing_keys: false + sample_std: true + cache_pool_bytes: 0 + seed: 0 + include_wells: null + exclude_fovs: null + focus_channel: null + reference_pixel_size_xy_um: null + reference_pixel_size_z_um: null +optimizer: null +lr_scheduler: null +ckpt_path: null diff --git a/applications/dynaclr/configs/evaluation/MorphEm-frozen/infectomics-annotated.yaml b/applications/dynaclr/configs/evaluation/MorphEm-frozen/infectomics-annotated.yaml new file mode 100644 index 000000000..cee51cdab --- /dev/null +++ b/applications/dynaclr/configs/evaluation/MorphEm-frozen/infectomics-annotated.yaml @@ -0,0 +1,36 @@ +# Evaluation config: MorphEm-frozen × infectomics-annotated (Wave 1, baseline). +# +# MorphEm-frozen uses the ViT-S/16 DINO weights pretrained on microscopy images +# (CaicedoLab/MorphEm) — biology-trained but *non-temporal* and *non-DynaCLR*. +# Tests whether microscopy SSL pretraining alone matches DynaCLR's contrastive +# recipe. Sibling baseline to CELL-DINO-frozen / DINOv3-frozen. +# +# `ckpt_path: null` — MorphEm weights load from HuggingFace in +# MorphEmModel.__init__ (trust_remote_code; needs network on the GPU node). +# The eval orchestrator omits `ckpt_path` from the generated predict YAML so +# Lightning skips checkpoint restoration. +# +# Submit: +# sbatch applications/dynaclr/configs/evaluation/MorphEm-frozen/run_infectomics_annotated.sh + +base: + - ../recipes/predict.yml + - ../recipes/reduce.yml + - ../recipes/plot_infectomics.yml + - ../recipes/linear_classifiers_infectomics.yml + - ../recipes/infectomics-annotated.yml + +training_config: /hpc/mydata/eduardo.hirata/repos/viscy/applications/dynaclr/configs/evaluation/MorphEm-frozen/training_config_morphem_frozen.yaml +ckpt_path: null +output_dir: /hpc/projects/organelle_phenotyping/models/MorphEm-frozen/evaluations/infectomics-annotated/ + +# MorphEm ViT-S/16 is ~44M params; 384-dim cls embeddings (same as DINOv3-S). +predict: + batch_size: 128 + num_workers: 4 + precision: 32-true + devices: 1 + +# Publish to its own sub-registry so the LC bundle is identifiable. +linear_classifiers: + publish_dir: /hpc/projects/organelle_phenotyping/models/linear_classifiers/MorphEm-frozen/infectomics/ diff --git a/applications/dynaclr/configs/evaluation/MorphEm-frozen/training_config_morphem_frozen.yaml b/applications/dynaclr/configs/evaluation/MorphEm-frozen/training_config_morphem_frozen.yaml new file mode 100644 index 000000000..a4684ce4f --- /dev/null +++ b/applications/dynaclr/configs/evaluation/MorphEm-frozen/training_config_morphem_frozen.yaml @@ -0,0 +1,189 @@ +# Synthetic "training_config" for MorphEm-frozen evaluation baseline. +# +# MorphEm-frozen uses the ViT-S/16 DINO weights pretrained on microscopy images +# (CaicedoLab/MorphEm on HuggingFace). No contrastive fine-tuning, no trainable +# projection head. Embeddings are the 384-dim cls token mean-pooled across input +# channels. +# +# This file is consumed by `dynaclr prepare-eval-configs` to generate a +# Lightning predict YAML. There is no Lightning checkpoint — the leaf YAML +# sets `ckpt_path: null`, so the orchestrator omits `ckpt_path` from the +# predict YAML, and MorphEmModel.__init__ loads weights from HuggingFace +# (trust_remote_code; requires network on the GPU node). +# +# Mirrors CELL-DINO-frozen/training_config_cell_dino_frozen.yaml except the +# encoder class_path/init_args. Same dataloader, same preprocessing → +# apples-to-apples comparison against CELL-DINO / DINOv3 frozen. + +# lightning.pytorch==2.6.1 +seed_everything: 42 +trainer: + accelerator: gpu + strategy: auto + devices: 1 + num_nodes: 1 + precision: 32-true + fast_dev_run: false + max_epochs: 100 + log_every_n_steps: 10 + enable_checkpointing: false + accumulate_grad_batches: 1 + inference_mode: true +model: + class_path: dynaclr.engine.ContrastiveModule + init_args: + encoder: + class_path: viscy_models.foundation.morphem.MorphEmModel + init_args: + model_name: CaicedoLab/MorphEm + img_size: 224 + freeze: true + projection: null + loss_function: + class_path: pytorch_metric_learning.losses.NTXentLoss + init_args: + temperature: 0.5 + embedding_regularizer: null + embedding_reg_weight: 1 + reducer: null + distance: null + collect_stats: null + lr: 0.0001 + schedule: Constant + log_batches_per_epoch: 3 + log_samples_per_batch: 3 + log_embeddings_every_n_epochs: 10 + pca_color_keys: condition + log_negative_metrics_every_n_epochs: 2 + example_input_array_shape: + - 1 + - 1 + - 1 + - 160 + - 160 + ckpt_path: null + freeze_backbone: false + projection: null +data: + class_path: dynaclr.data.datamodule.MultiExperimentDataModule + init_args: + cell_index_path: /hpc/projects/intracellular_dashboard/organelle_dynamics/models/DINOv3-temporal-MLP-2D-BagOfChannels/DynaCLR-2D-BagOfChannels-v3.parquet + z_window: 1 + yx_patch_size: + - 256 + - 256 + final_yx_patch_size: + - 160 + - 160 + val_experiments: [] + split_ratio: 0.8 + tau_range: + - 0.5 + - 2.0 + tau_decay_rate: 0.0 + batch_size: 256 + num_workers: 1 + stratify_by: + - condition + leaky: 0.0 + temporal_enrichment: false + temporal_window_hours: 2.0 + temporal_global_fraction: 0.3 + channels_per_sample: 1 + channel_dropout_channels: null + channel_dropout_prob: 0.0 + normalizations: + - class_path: viscy_transforms.ScaleIntensityRangePercentilesd + init_args: + keys: + - channel_0 + lower: 50.0 + upper: 99.0 + b_min: 0.0 + b_max: 1.0 + clip: false + relative: false + channel_wise: false + allow_missing_keys: false + augmentations: + - class_path: viscy_transforms.BatchedRandAffined + init_args: + keys: + - channel_0 + prob: 0.8 + rotate_range: + - 3.14 + - 0.0 + - 0.0 + shear_range: + - 0.05 + - 0.05 + - 0.0 + - 0.05 + - 0.0 + - 0.05 + translate_range: null + scale_range: + - - 0.8 + - 1.2 + - - 0.8 + - 1.2 + - - 0.8 + - 1.2 + mode: bilinear + allow_missing_keys: false + - class_path: viscy_transforms.BatchedRandAdjustContrastd + init_args: + keys: + - channel_0 + gamma: + - 0.8 + - 1.2 + prob: 0.5 + invert_image: false + retain_stats: false + allow_missing_keys: false + - class_path: viscy_transforms.BatchedRandScaleIntensityd + init_args: + keys: + - channel_0 + factors: 0.5 + prob: 0.5 + channel_wise: false + allow_missing_keys: false + - class_path: viscy_transforms.BatchedRandGaussianSmoothd + init_args: + keys: + - channel_0 + sigma_x: + - 0.25 + - 0.75 + sigma_y: + - 0.25 + - 0.75 + sigma_z: + - 0.0 + - 0.0 + truncated: 4.0 + prob: 0.5 + border_type: constant + allow_missing_keys: false + - class_path: viscy_transforms.BatchedRandGaussianNoised + init_args: + keys: + - channel_0 + prob: 0.5 + mean: 0.0 + std: 0.2 + allow_missing_keys: false + sample_std: true + cache_pool_bytes: 0 + seed: 0 + include_wells: null + exclude_fovs: null + focus_channel: null + reference_pixel_size_xy_um: null + reference_pixel_size_z_um: null +optimizer: null +lr_scheduler: null +ckpt_path: null diff --git a/applications/dynaclr/configs/evaluation/clustering/zikv_infection_pop.yml b/applications/dynaclr/configs/evaluation/clustering/zikv_infection_pop.yml new file mode 100644 index 000000000..ebcf1c28d --- /dev/null +++ b/applications/dynaclr/configs/evaluation/clustering/zikv_infection_pop.yml @@ -0,0 +1,73 @@ +# Proof-of-principle: recover uninfected-vs-infected cell state from DynaCLR +# embeddings per marker, with little/no manual annotation. +# +# Two annotation-frugal arms per marker: +# - KNN (semi-supervised): train on the sparse human `infection_state` labels +# (GroupKFold by fov_name for honest metrics), then propagate to all cells. +# - HDBSCAN (unsupervised): sweep {raw X, X_pca, umap2d, phate2d}, score vs labels. +# Plus time evidence (t / hours_post_perturbation) and qualitative image crops +# split into infected/ vs uninfected/. +# +# See .ed_planning/dynaclr/clustering/PLAN.md + +title: "Infection-state clustering — proof of principle" +# Ground-truth label column (human annotation, sparse) baked into embedding obs. +label_column: infection_state +positive_class: infected +negative_class: uninfected +# The existing linear-classifier column, used only as a baseline comparison. +baseline_column: predicted_infection_state +# Grouping column for leakage-free CV (no track spans train/val). +group_column: fov_name + +# Supervised KNN arm. +knn: + k_grid: [5, 15, 30] + n_splits: 5 + +# Unsupervised HDBSCAN arm. +hdbscan: + # Which representations to cluster. "X_pca" = precomputed 32-d obsm, + # "umap2d"/"phate2d" = computed here. Raw 768-d "X" is intentionally + # excluded: density clustering on 28k x 768 is O(n^2) and does not finish + # in reasonable time, and it yields only noise (see PLAN dead-ends). + spaces: [X_pca, umap2d, phate2d] + min_cluster_size: [50, 100, 200] + min_samples: [5, 25] + # The plotted/montaged clustering is picked by max well_separation (§5, + # control-vs-infected well enrichment) among configs that are DISPLAYABLE: + # noise fraction and cluster count capped so a 72-cluster / 80%-noise config + # can't win via many tiny pure clusters. + select_max_noise_frac: 0.5 + select_max_clusters: 20 + +# Qualitative single-cell image samples. +samples: + n_samples: 24 # per class (infected / uninfected) + patch_size: 160 # y/x crop side length (pixels) + z_window: 5 # +/- slices MIP'd for fluorescence markers + # Show the viral-sensor (mCherry) channel beside each marker crop so the + # infection reporter is always visible. Skipped when marker == this channel. + pair_channel: "raw mCherry EX561 EM600-37" + pair_mip: true + +random_seed: 42 + +output_dir: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated/clustering_pop + +# One entry per marker. `image_zarr` is the source pixel store; `image_channel` +# is the zarr channel to crop; `mip` = MIP over the z-window (fluorescence) vs +# single focus slice (Phase3D). +datasets: + - marker: viral_sensor + embeddings: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated/embeddings/2025_07_24_A549_viral_sensor_ZIKV.zarr + image_zarr: /hpc/projects/organelle_phenotyping/datasets/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV.zarr + # viral_sensor is the mCherry channel (GFP is the organelle markers). This + # was previously mis-set to GFP, which cropped the wrong channel. + image_channel: "raw mCherry EX561 EM600-37" + mip: true + - marker: Phase3D + embeddings: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated/embeddings/2025_07_24_A549_Phase3D_ZIKV.zarr + image_zarr: /hpc/projects/organelle_phenotyping/datasets/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV.zarr + image_channel: "Phase3D" + mip: false diff --git a/applications/dynaclr/configs/evaluation/clustering/zikv_remodel_pop.yml b/applications/dynaclr/configs/evaluation/clustering/zikv_remodel_pop.yml new file mode 100644 index 000000000..20c108126 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/clustering/zikv_remodel_pop.yml @@ -0,0 +1,70 @@ +# Proof-of-principle: recover remodeled-vs-non-remodeled organelle state from +# DynaCLR embeddings per organelle marker, with little/no manual annotation. +# +# Mirrors zikv_infection_pop.yml but for `organelle_state` (remodel/noremodel). +# Each organelle is classified from ITS OWN marker's embeddings (G3BP1 remodeling +# from G3BP1 embeddings, ER remodeling from SEC61B embeddings) — the marker that +# images that organelle. TOMM20 is NOT included: it has no embedding zarr in this +# eval set (was not embedded as a single-marker file). +# +# Note the class imbalance: G3BP1 has only ~422 remodel / 9649 labeled (~4%); +# SEC61B is better at ~1834 / 5850 (~31%). Read balanced_accuracy + per-class F1, +# not raw accuracy. +# +# See .ed_planning/dynaclr/clustering/PLAN.md + +title: "Organelle-remodeling clustering — proof of principle" +# Ground-truth label column (human annotation, sparse) baked into embedding obs. +label_column: organelle_state +positive_class: remodel +negative_class: noremodel +# The existing linear-classifier column, used only as a baseline comparison. +baseline_column: predicted_organelle_state +# Grouping column for leakage-free CV (no track spans train/val). +group_column: fov_name + +# Supervised KNN arm. +knn: + k_grid: [5, 15, 30] + n_splits: 5 + +# Unsupervised HDBSCAN arm. +hdbscan: + # "X_pca" = precomputed 32-d obsm; "umap2d"/"phate2d" computed here. Raw 768-d + # "X" excluded (O(n^2), all-noise — see PLAN dead-ends). + spaces: [X_pca, umap2d, phate2d] + min_cluster_size: [50, 100, 200] + min_samples: [5, 25] + # Pick the plotted/montaged clustering by max well_separation (§5, control-vs- + # infected enrichment) among displayable configs (noise + cluster-count capped). + select_max_noise_frac: 0.5 + select_max_clusters: 20 + +# Qualitative single-cell image samples. +samples: + n_samples: 24 # per class (remodel / noremodel) + patch_size: 160 # y/x crop side length (pixels) + z_window: 5 # +/- slices MIP'd for fluorescence markers + # Show the viral-sensor (mCherry) channel beside each organelle-marker crop, + # so remodeling can be read against the infection reporter of the same cell. + pair_channel: "raw mCherry EX561 EM600-37" + pair_mip: true + +random_seed: 42 + +output_dir: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated/clustering_remodel_pop + +# One entry per organelle marker. Both G3BP1 and SEC61B are GFP-tagged reporters +# (imaged in different wells of the same plate); the well/FOV in obs disambiguates +# which organelle. Both are fluorescence -> MIP over the focus z-window. +datasets: + - marker: G3BP1 + embeddings: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated/embeddings/2025_07_24_A549_G3BP1_ZIKV.zarr + image_zarr: /hpc/projects/organelle_phenotyping/datasets/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV.zarr + image_channel: "raw GFP EX488 EM525-45" + mip: true + - marker: SEC61B + embeddings: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated/embeddings/2025_07_24_A549_SEC61_ZIKV.zarr + image_zarr: /hpc/projects/organelle_phenotyping/datasets/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV.zarr + image_channel: "raw GFP EX488 EM525-45" + mip: true diff --git a/applications/dynaclr/configs/evaluation/clustering/zikv_velocity.yml b/applications/dynaclr/configs/evaluation/clustering/zikv_velocity.yml new file mode 100644 index 000000000..d838c04ea --- /dev/null +++ b/applications/dynaclr/configs/evaluation/clustering/zikv_velocity.yml @@ -0,0 +1,29 @@ +# PHATE + diffusion-pseudotime + velocity maps for the phenotype-flow analysis (§6). +# +# Discover whether cells follow an ordered control -> remodel progression: +# - PHATE layout (preserves trajectory geometry, unlike UMAP which fragmented it), +# - diffusion pseudotime rooted at control-well cells (annotation-free), +# - measured velocity from single-cell tracks (UMAP(t+1)-UMAP(t) analog, but on PHATE), +# - validation: Spearman(pseudotime, real t) should be strongly positive. +# +# Runs the three markers requested: viral_sensor, Phase3D, G3BP1. +# See .ed_planning/dynaclr/clustering/PLAN.md §6. + +random_seed: 42 +output_dir: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated_epoch105_step84800/clustering_pop + +# control_ref subtracts the per-timepoint control-well centroid before differencing +# the velocity, removing background drift shared by all cells. Probed 2026-07-15: +# it HELPS the organelle channel (G3BP1 infected-only coherence 0.018 -> 0.052, ~3x) +# but slightly HURTS the reporter (viral_sensor 0.030 -> 0.021), whose motion is +# already mostly infection signal. So: on for organelle/label-free, off for reporter. +datasets: + - marker: viral_sensor + embeddings: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated_epoch105_step84800/embeddings/2025_07_24_A549_viral_sensor_ZIKV.zarr + control_ref: false + - marker: Phase3D + embeddings: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated_epoch105_step84800/embeddings/2025_07_24_A549_Phase3D_ZIKV.zarr + control_ref: true + - marker: G3BP1 + embeddings: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/evaluations/infectomics-annotated_epoch105_step84800/embeddings/2025_07_24_A549_G3BP1_ZIKV.zarr + control_ref: true diff --git a/applications/dynaclr/configs/evaluation/recipes/embedding_consistency_qc.yml b/applications/dynaclr/configs/evaluation/recipes/embedding_consistency_qc.yml new file mode 100644 index 000000000..d62295d62 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/recipes/embedding_consistency_qc.yml @@ -0,0 +1,43 @@ +# Embedding-consistency QC — per-marker dataset x dataset consistency matrices. +# ============================================================================= +# Pools every dataset's control-cell embeddings for one model/run/checkpoint +# (globbed via dynaclr.evaluation.paths.iter_embeddings over +# {datasets_root}/*/2-phenotyping/predictions/{model_family}/{run}/{ckpt_name}/{marker}.zarr) +# and reports two symmetric per-marker matrices + heatmaps: +# - MMD² (primary, distributional): low off-diagonal = comparable; large = +# batch effect to correct with LOT. Same statistic LOT validation uses. +# - Pearson correlation of per-dataset mean embeddings (companion, [-1,1], +# readable but centroid-only — a fast sanity check, blind to spread/shape). +# +# This step only detects and reports — it does not correct. +# +# Run: +# dynaclr embedding-consistency-qc -c recipes/embedding_consistency_qc.yml + +base: + - mmd_defaults.yml + +# Provenance tuple that identifies the embeddings to pool across datasets. +model_family: DynaCLR-2D-MIP-BagOfChannels-single-marker-fix-shuffler +run: run1 +ckpt_name: epoch146_step117600 +# datasets_root: null # null uses the canonical DATASETS_ROOT + +output_dir: /hpc/projects/intracellular_dashboard/organelle_dynamics/_qc/embedding_consistency + +# Restrict to control cells so perturbation biology cannot look like a batch effect. +obs_filter: + perturbation: uninfected + +# Measure residual batch effects independent of a global mean offset. +center_per_experiment: true + +# Temporal binning (optional; unset here = all timepoints pooled). +# WARNING: setting temporal_bin_size or temporal_bins changes ONLY the MMD +# matrix — it computes one MMD² per (condition, bin) and the matrix cell becomes +# the MEAN over bins (per-bin values kept in consistency_mmd_results.csv). The +# Pearson matrix always pools all timepoints into one mean per dataset, so once +# bins are on the two matrices describe different populations. Requires the obs +# column 'hours_post_perturbation'. +# temporal_bin_size: 6.0 +# temporal_bins: [0, 6, 12, 24] diff --git a/applications/dynaclr/configs/evaluation/recipes/embedding_consistency_qc_organelle_box.yml b/applications/dynaclr/configs/evaluation/recipes/embedding_consistency_qc_organelle_box.yml new file mode 100644 index 000000000..ab6763ae1 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/recipes/embedding_consistency_qc_organelle_box.yml @@ -0,0 +1,45 @@ +# Embedding-consistency QC — organelle-box cohort, split by microscope (mantis v1/v2). +# ============================================================================= +# Pools every organelle-box dataset's control-cell embeddings for the DynaCLR +# single-marker model and reports per-marker dataset x dataset consistency +# matrices, PARTITIONED by acquisition platform (obs["microscope"]): +# - within_mantis_v2/ : the nine 0.1133 um/px (2026) datasets among themselves +# - within_mantis_v1/ : the lone 0.1494 um/px dataset (2024_11_07) — skipped +# (degenerate: needs >=2 datasets) +# - cross_mantis_v1__mantis_v2/ : v1 vs v2 (only SEC61B is on both platforms) — +# this block IS the cross-platform batch effect LOT corrects. +# +# Requires obs["microscope"] on the embeddings (enrich-obs-from-collection from a +# collection that carries the microscope field). Detects/reports only; LOT corrects. +# +# Run: +# dynaclr embedding-consistency-qc -c recipes/embedding_consistency_qc_organelle_box.yml + +base: + - mmd_defaults.yml + +# DynaCLR single-marker model, as embedded by the organelle_remodeling matrix. +model_family: DynaCLR-2D-MIP-BagOfChannels +run: 2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler +ckpt_name: epoch105-step84800 + +output_dir: /hpc/projects/intracellular_dashboard/organelle_dynamics/qc_embeddings + +# Partition datasets by acquisition platform: within-v1, within-v2, and v1 x v2 cross. +split_by: microscope + +# Start with the cheap, readable mean-only matrix. MMD (full-distribution) and +# Fréchet (mean+covariance) stay available — add them to this list to compute them. +metrics: [pearson] + +# Time-pool the Pearson matrix: mean per 2 h hours_post_perturbation bin, then +# average the bin-means. Each HPI bin counts equally, so datasets with different +# intervals / frame counts can't look like a batch effect purely from time sampling. +pearson_hpi_bin_hours: 2.0 + +# Control cells only, so perturbation biology cannot masquerade as a batch effect. +obs_filter: + perturbation: uninfected + +# Residual batch effects independent of a global mean offset. +center_per_experiment: true diff --git a/applications/dynaclr/configs/evaluation/recipes/mmd_defaults.yml b/applications/dynaclr/configs/evaluation/recipes/mmd_defaults.yml index 8370035b4..3be70f435 100644 --- a/applications/dynaclr/configs/evaluation/recipes/mmd_defaults.yml +++ b/applications/dynaclr/configs/evaluation/recipes/mmd_defaults.yml @@ -14,6 +14,19 @@ group_by: perturbation save_plots: true +# Default biological-effect representation. Per-experiment and --pooled MMD +# use this; --combined/--over-time batch-effect modes remain in their supplied +# coordinate system so normalization cannot erase the batch effect under test. +representation: + normalization: control_mad + control_key: perturbation + control_values: [uninfected] + smooth_sigma_timepoints: 2.0 + mad_floor_quantile: 0.05 + pca_variance: 0.80 + pca_max_cells_per_dataset_class: 5000 + random_seed: 42 + mmd: n_permutations: 1000 max_cells: 2000 diff --git a/applications/dynaclr/configs/evaluation/recipes/witness_gmm_pooled_joint_pca80.yaml b/applications/dynaclr/configs/evaluation/recipes/witness_gmm_pooled_joint_pca80.yaml new file mode 100644 index 000000000..9ea354272 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/recipes/witness_gmm_pooled_joint_pca80.yaml @@ -0,0 +1,76 @@ +# Canonical pooled biological-state recipe. +# +# This is the only maintained recipe for the validated stack. Edit both sections +# so their input paths refer to the same per-marker stores. Run pooled_representation +# first to write obsm["X_normalized_pca80"], then run witness_gmm_labels. + +pooled_representation: + input_paths: + - /path/to/dataset_1_marker_embeddings.zarr + - /path/to/dataset_2_marker_embeddings.zarr + output_dir: /path/to/pooled_joint_witness_results + group_by: perturbation + comparisons: + - cond_a: uninfected + cond_b: DENV + label: uninfected_vs_DENV + condition_aliases: null + representation: + normalization: control_mad + control_key: perturbation + control_values: [uninfected] + experiment_key: experiment + marker_key: marker + hpi_key: hours_post_perturbation + smooth_sigma_timepoints: 2.0 + mad_floor_quantile: 0.05 + pca_variance: 0.80 + pca_max_cells_per_dataset_class: 1900 + random_seed: 42 + temporal_bin_size: 0.5 + save_plots: true + mmd: + n_permutations: 1000 + max_cells: 2000 + min_cells: 20 + seed: 42 + balance_samples: true + share_bandwidth_from: null + map_settings: + enabled: false + +witness_gmm_labels: + experiments: + - experiment: dataset_1 + embeddings_zarr: /path/to/dataset_1_marker_embeddings.zarr + control_filter: {perturbation: uninfected} + perturbed_filter: {perturbation: DENV} + - experiment: dataset_2 + embeddings_zarr: /path/to/dataset_2_marker_embeddings.zarr + control_filter: {perturbation: uninfected} + perturbed_filter: {perturbation: DENV} + + marker_filters: [pAL40] + label_column: infection_state + class_map: {positive: infected, negative: uninfected} + output_dir: /path/to/pooled_joint_witness_results + annotation_format: csv + + # Promoted representation and pooled-teacher contract. + embedding_key: X_normalized_pca80 + reference_sampling: balanced_by_experiment + max_reference_cells: 1900 + witness_time_bin_hours: null + bandwidth: null + + # MMD significance gate and all-cell joint/tied GMM. + mmd_n_permutations: 1000 + mmd_pvalue_threshold: 0.05 + gmm_fit_population: joint_control_perturbed + gmm_covariance_type: tied + gmm_pos_threshold: 0.8 + gate: gmm + + condition_column: perturbation + mmd_hpi_bin_hours: 2.0 + random_seed: 42 diff --git a/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/01_stage_a_sec61b_labels.yml b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/01_stage_a_sec61b_labels.yml new file mode 100644 index 000000000..6cd42629f --- /dev/null +++ b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/01_stage_a_sec61b_labels.yml @@ -0,0 +1,34 @@ +# Stage A — witness-GMM labels on SEC61B organelle embeddings (ZIKV plate). +# +# Dataset: 2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV (mantis v2). +# SEC61B embeddings: A/2 = uninfected (5582), B/2 = ZIKV (23468), t=0-66 (hpi 3-36). +# Reproduces Soorya's sec61b_timelapse_zikv_classifier.py labeling stage: +# control (X) = all uninfected cells, perturbed (Y) = all ZIKV cells (no time gate), +# 2-component GMM on the ZIKV witness scores, posterior >= 0.8 = confident remodeled. +# +# Output: /labels/SEC61B_organelle_remodeling_state.csv +# keyed by (experiment, fov_name, id) — joins onto the Phase3D zarr in Stage B. +# +# Run: dynaclr witness-gmm-labels -c 01_stage_a_sec61b_labels.yml + +witness_gmm_labels: + experiments: + - experiment: "2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV" + embeddings_zarr: "/hpc/projects/organelle_phenotyping/datasets/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/predictions/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/epoch105-step84800/SEC61B.zarr" + # Both references pooled by perturbation across all timepoints (gate-free). + control_filter: {perturbation: uninfected} + perturbed_filter: {perturbation: [ZIKV]} + # From the SEC61B organelle marker the witness axis means ER remodeling. + marker_filters: [SEC61B] + label_column: organelle_remodeling_state + class_map: {positive: remodel, negative: noremodel} + gate: gmm + gmm_pos_threshold: 0.8 + # MMD significance gate (BY-FDR across the run's (marker, condition) family). + mmd_pvalue_threshold: 0.05 + mmd_n_permutations: 1000 + bandwidth: null # median heuristic on pooled (control, perturbed) + max_reference_cells: 5000 + condition_column: perturbation + annotation_format: csv + output_dir: "/hpc/projects/organelle_phenotyping/datasets/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/witness_gmm_phase_classifier" diff --git a/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/02_stage_b_train_phase.yml b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/02_stage_b_train_phase.yml new file mode 100644 index 000000000..516db7c81 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/02_stage_b_train_phase.yml @@ -0,0 +1,34 @@ +# Stage B — train the PHASE classifier from the SEC61B witness-GMM labels +# (teacher/student: labels computed on SEC61B, classifier trained on Phase3D). +# +# The Stage-A annotation file is keyed by (fov_name, id); the Phase3D embeddings +# share the same ultrack cell keys, so the SEC61B-derived organelle_remodeling_state +# label joins onto the matching Phase3D cells. marker_filters: [Phase3D] restricts +# training to the Phase3D embeddings. Only the SEC61B wells (A/2, B/2) carry labels, +# so cells in the other wells drop out of training (NaN label) — as intended. +# +# The pipeline (StandardScaler + LogisticRegression C=1.0, class_weight=balanced) +# matches Soorya's remodeling_state_Phase3D_ZIKV_Mantisv2 classifier. +# +# Run: dynaclr run-linear-classifiers -c 02_stage_b_train_phase.yml + +embeddings_path: "/hpc/projects/organelle_phenotyping/datasets/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/predictions/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/epoch105-step84800/Phase3D.zarr" +output_dir: "/hpc/projects/organelle_phenotyping/datasets/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/witness_gmm_phase_classifier/stage_b_phase" + +label_source: annotations +annotations: + - experiment: "2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV" + path: "/hpc/projects/organelle_phenotyping/datasets/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/witness_gmm_phase_classifier/labels/SEC61B_organelle_remodeling_state.csv" +tasks: + - task: organelle_remodeling_state + marker_filters: [Phase3D] +use_scaling: true +use_pca: false +split_train_data: 0.8 +random_seed: 42 +# Track-level split — no track lands in both train/val (temporal-contrastive +# SSL leaks on cell-level splits). +split_groups_by: + - experiment + - fov_name + - track_id diff --git a/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/02_stage_b_train_sec61b.yml b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/02_stage_b_train_sec61b.yml new file mode 100644 index 000000000..f623a4029 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/02_stage_b_train_sec61b.yml @@ -0,0 +1,29 @@ +# Stage B — train the SAME-MODALITY SEC61B classifier from the witness-GMM labels. +# +# Complement to 02_stage_b_train_phase.yml (teacher/student). Here the classifier is +# trained on the SAME SEC61B embeddings the labels were computed from — a logistic +# classifier over the organelle channel that reproduces its own witness-GMM labels. +# Its purpose is a like-for-like comparison with the Phase3D classifier: same model +# type (StandardScaler + LR C=1.0, class_weight=balanced), same labels, same split, +# differing only by the input channel (SEC61B GFP vs Phase3D). +# +# Run: dynaclr run-linear-classifiers -c 02_stage_b_train_sec61b.yml + +embeddings_path: "/hpc/projects/organelle_phenotyping/datasets/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/predictions/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/epoch105-step84800/SEC61B.zarr" +output_dir: "/hpc/projects/organelle_phenotyping/datasets/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/witness_gmm_phase_classifier/stage_b_sec61b" + +label_source: annotations +annotations: + - experiment: "2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV" + path: "/hpc/projects/organelle_phenotyping/datasets/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/witness_gmm_phase_classifier/labels/SEC61B_organelle_remodeling_state.csv" +tasks: + - task: organelle_remodeling_state + marker_filters: [SEC61B] +use_scaling: true +use_pca: false +split_train_data: 0.8 +random_seed: 42 +split_groups_by: + - experiment + - fov_name + - track_id diff --git a/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/plot_timecourse_and_prob_grids.py b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/plot_timecourse_and_prob_grids.py new file mode 100644 index 000000000..f6ea0c734 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/plot_timecourse_and_prob_grids.py @@ -0,0 +1,461 @@ +"""Per-timepoint AUROC overlay + probability-split sample grids for the ZIKV plate. + +Two diagnostics for the witness-GMM phase-classifier run on +2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV, both on OUR plate: + +D1 timecourse_auroc_sec61b_vs_phase3d.{csv,png} + Per-hpi AUROC of the SEC61B organelle witness AND the Phase3D classifier on the + held-out val split, graded against the condition label (ZIKV vs uninfected) -- the + two-line style of Soorya's 03_metrics_over_time.svg. + +D2 prob_samples_{SEC61B,Phase3D}.png + Like sample_labels_by_hpi.py (rows = HPI bins), but columns are 5 P(remodel) buckets + (0/25/50/75/100 %) instead of classes, so the morphology sweep with predicted + probability is legible per timepoint. + +Both plots need a clean per-cell P(remodel) for EVERY cell, which the Stage-A CSV does not +persist, so we recompute: + SEC61B -- re-score the SEC61B embeddings with the MMD witness (control=uninfected, + perturbed=ZIKV) + fit_gmm_labels (seed 42) -> per-cell remodel posterior. + Phase3D -- apply the Stage-B pipeline joblib -> predict_proba[:, remodel]. + +P(remodel) and crop coords BOTH come from the canonical-tree embeddings (row order differs +between trees); crops come from DATA_ZARR keyed by obs (fov_name, t, y, x). + +Run: + /hpc/mydata/eduardo.hirata/repos/viscy/.venv/bin/python plot_timecourse_and_prob_grids.py +""" + +from pathlib import Path + +import anndata as ad +import joblib +import matplotlib.pyplot as plt +import numpy as np +import pandas as pd +from cmap import Colormap +from iohub import open_ome_zarr +from sklearn.metrics import roc_auc_score + +from dynaclr.evaluation.evaluate_config import WitnessGmmLabelsConfig +from dynaclr.evaluation.linear_classifiers.witness_gmm_labels import compute_marker_scores +from viscy_utils.cli_utils import load_config +from viscy_utils.evaluation.linear_classifier import group_ids_from_obs, group_val_split +from viscy_utils.evaluation.witness_gmm import fit_gmm_labels + +# ------------------------------------------------------------------ config ---- +# Canonical prediction tree (5-marker embeddings + existing labels/plots). +EMB_DIR = Path( + "/hpc/projects/intracellular_dashboard/organelle_dynamics" + "/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/predictions" + "/DynaCLR-2D-MIP-BagOfChannels" + "/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler" + "/epoch105-step84800" +) +SEC61B_EMB = EMB_DIR / "embeddings" / "2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV_SEC61B.zarr" +PHASE_EMB = EMB_DIR / "embeddings" / "2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV_Phase3D.zarr" +OUT_DIR = EMB_DIR / "labels" / "plots" + +# Stage-A config — reused verbatim so the SEC61B witness+GMM scoring here is the SAME +# reference construction / bandwidth / seed as the Stage-A labels (no re-implementation). +STAGE_A_CONFIG = Path(__file__).with_name("01_stage_a_sec61b_labels.yml") + +# Stage-B trained pipelines (from my run's tree). Phase3D = teacher/student (labels from +# SEC61B, trained on phase); SEC61B = same-modality (trained on the channel the labels came +# from) — used only for the % -remodeled plot so both markers are the SAME model type at the +# SAME threshold. NOTE the SEC61B same-modality LR is near-ceiling by construction (its labels +# were derived from these embeddings), so its curve tracks the witness gate; Phase3D is the +# informative cross-modality test. +PHASE_PIPELINE = Path( + "/hpc/projects/organelle_phenotyping/datasets" + "/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/witness_gmm_phase_classifier" + "/stage_b_phase/pipelines/organelle_remodeling_state_Phase3D.joblib" +) +SEC61B_PIPELINE = Path( + "/hpc/projects/organelle_phenotyping/datasets" + "/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV/2-phenotyping/witness_gmm_phase_classifier" + "/stage_b_sec61b/pipelines/organelle_remodeling_state_SEC61B.joblib" +) + +# Raw data zarr for crops (has focus_slice zattrs + both channels). +DATA_ZARR = Path( + "/hpc/projects/organelle_phenotyping/datasets" + "/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV" + "/2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV.zarr" +) + +# Soorya's reference timecourse (her plate) — overlaid as a faint band for context. +SOORYA_CSV = Path( + "/hpc/mydata/soorya.pradeep/scratch/label-free_classifier" + "/witness_score_testing/sec61b_timelapse_zikv/timecourse_metrics.csv" +) + +SEED = 42 +TRAIN_SIZE = 0.8 +GMM_POS_THRESHOLD = 0.8 # SEC61B witness-GMM display threshold (matches the Stage-A gate) +SPLIT_GROUPS = ("experiment", "fov_name", "track_id") + +# HPI bins for the grids: right-open 2-hour windows across 0-36h. +HPI_BINS = list(range(0, 38, 2)) +# P(remodel) buckets by nearest-25% (label -> [lo, hi)). +PROB_BUCKETS = [ + ("0%", 0.0, 0.125), + ("25%", 0.125, 0.375), + ("50%", 0.375, 0.625), + ("75%", 0.625, 0.875), + ("100%", 0.875, 1.0001), +] + +# Crop / render (from sample_labels_by_hpi.py). +Z_RANGE = (15, 45) +PATCH = 160 +N_PER_CELL = 20 +CMAP_GREEN = Colormap("green").to_mpl() +DISPLAY_PCTL = (1.0, 99.5) +SCALE_SAMPLE = 60 +# --------------------------------------------------------------------------- # + + +# ---- crop helpers (reused verbatim from sample_labels_by_hpi.py) ----------- # +def focus_z(pos, marker: str, t: int) -> int: + """In-focus z-slice for (marker, timepoint) from the FOV's ``focus_slice`` metadata.""" + fs = dict(pos.zattrs)["focus_slice"][marker] + z = fs["per_timepoint"].get(str(t), 0) + if not z: + z = int(round(fs["fov_statistics"]["z_focus_mean"])) + return int(z) + + +def crop( + pos_img, t: int, ch_idx: int, y: int, x: int, projection: str = "mip", z_slice: int | None = None +) -> np.ndarray: + """Crop one cell: PATCH box at (y,x) of channel ch at time t (MIP over Z_RANGE or focus slice).""" + half = PATCH // 2 + _T, _C, Z, Y, X = pos_img.shape + y0, y1 = max(0, y - half), min(Y, y + half) + x0, x1 = max(0, x - half), min(X, x + half) + if projection == "focus": + z = min(max(0, z_slice if z_slice is not None else Z // 2), Z - 1) + return pos_img[t, ch_idx, z, y0:y1, x0:x1] + z0, z1 = max(0, Z_RANGE[0]), min(Z, Z_RANGE[1]) + return pos_img[t, ch_idx, z0:z1, y0:y1, x0:x1].max(axis=0) + + +# ---- per-cell P(remodel) recomputation ------------------------------------- # +def sec61b_p_remodel(adata: ad.AnnData) -> np.ndarray: + """Per-cell P(remodel) + witness score for ALL SEC61B cells, via the Stage-A path. + + Runs the real Stage-A scorer (``compute_marker_scores``) with the actual Stage-A config + (``01_stage_a_sec61b_labels.yml``) so the references, bandwidth, and seed are identical to + the Stage-A labels — no re-implementation. Then fits the 2-component GMM on the perturbed + cells' scores (as ``label_marker`` does) and applies its remodel-component posterior to + every cell, giving a continuous P(remodel) for all cells (the CSV only persists confident + positives + controls). + """ + config = WitnessGmmLabelsConfig(**load_config(STAGE_A_CONFIG)["witness_gmm_labels"]) + ms = compute_marker_scores(adata, config.experiments, config) + if ms is None: + raise RuntimeError("compute_marker_scores found no control/perturbed reference cells") + scores = ms.scores + res = fit_gmm_labels( + scores[ms.perturbed_mask], pos_threshold=config.gmm_pos_threshold, random_state=config.random_seed + ) + posterior = res.gmm.predict_proba(scores.reshape(-1, 1))[:, res.remod_component] + print( + f" SEC61B (Stage-A path) GMM means={res.gmm.means_.ravel().round(4)} " + f"weights={res.gmm.weights_.round(3)} separated={res.separated}" + ) + return posterior, scores + + +def lr_p_remodel(adata: ad.AnnData, pipeline_path: Path) -> np.ndarray: + """Per-cell P(remodel) for ALL cells via a trained Stage-B logistic pipeline.""" + pipe = joblib.load(pipeline_path) + pos_idx = list(pipe.classifier.classes_).index("remodel") + X = adata.X if isinstance(adata.X, np.ndarray) else adata.X.toarray() + return pipe.predict_proba(X)[:, pos_idx] + + +# ---- val split (the shared LC-backend split — identical to Stage B) --------- # +def val_mask(obs: pd.DataFrame, y: np.ndarray) -> np.ndarray: + """Boolean mask of the held-out val cells via the shared LC split helper. + + Uses ``group_ids_from_obs`` + ``group_val_split`` — the SAME functions + ``run-linear-classifiers`` uses — so this val set is exactly Stage B's, not a + look-alike. ``y`` is the per-cell label array (used only for the non-grouped + fallback; grouped splits ignore it). + """ + groups = group_ids_from_obs(obs, list(SPLIT_GROUPS)) + _, idx_val = group_val_split(len(obs), y, groups, TRAIN_SIZE, SEED) + m = np.zeros(len(obs), dtype=bool) + m[idx_val] = True + return m + + +def per_hpi_auroc(hpi: np.ndarray, y_cond: np.ndarray, score: np.ndarray) -> pd.DataFrame: + """Per-hpi AUROC of ``score`` vs the binary condition label; skips single-class bins.""" + df = pd.DataFrame({"hpi": np.round(hpi, 1), "y": y_cond, "s": score}) + rows = [] + for h, g in df.groupby("hpi"): + yt = g["y"].to_numpy() + if yt.sum() == 0 or (1 - yt).sum() == 0: + continue + rows.append({"hpi": h, "auroc": roc_auc_score(yt, g["s"].to_numpy()), "n_cells": len(g)}) + return pd.DataFrame(rows).sort_values("hpi") + + +# ---- D1: AUROC overlay ----------------------------------------------------- # +def plot_timecourse(sec_obs, sec_scores, phase_obs, phase_p) -> None: + """Overlay per-hpi AUROC of SEC61B witness and Phase3D classifier (val, vs condition).""" + sec_y = (sec_obs["perturbation"] == "ZIKV").astype(int).to_numpy() + sm = val_mask(sec_obs, sec_y) + # SEC61B: lower witness = more ZIKV-like, so grade on -witness_score. + sec_tc = per_hpi_auroc( + sec_obs.loc[sm, "hours_post_perturbation"].to_numpy(), + (sec_obs.loc[sm, "perturbation"] == "ZIKV").astype(int).to_numpy(), + -sec_scores[sm], + ).rename(columns={"auroc": "org_auroc", "n_cells": "n_org"}) + + phase_y = (phase_obs["perturbation"] == "ZIKV").astype(int).to_numpy() + pm = val_mask(phase_obs, phase_y) + phase_tc = per_hpi_auroc( + phase_obs.loc[pm, "hours_post_perturbation"].to_numpy(), + (phase_obs.loc[pm, "perturbation"] == "ZIKV").astype(int).to_numpy(), + phase_p[pm], + ).rename(columns={"auroc": "phase_auroc", "n_cells": "n_phase"}) + + merged = phase_tc.merge(sec_tc, on="hpi", how="outer").sort_values("hpi") + OUT_DIR.mkdir(parents=True, exist_ok=True) + merged.to_csv(OUT_DIR / "timecourse_auroc_sec61b_vs_phase3d.csv", index=False) + + fig, ax = plt.subplots(figsize=(8, 5)) + ax.plot( + merged["hpi"], merged["phase_auroc"], color="mediumorchid", lw=2, marker="o", ms=3, label="Phase3D classifier" + ) + ax.plot( + merged["hpi"], + merged["org_auroc"], + color="darkorange", + lw=2, + ls="--", + marker="o", + ms=3, + label="SEC61B organelle witness", + ) + if SOORYA_CSV.exists(): + s = pd.read_csv(SOORYA_CSV) + ax.plot(s["hpi"], s["phase_auroc"], color="mediumorchid", lw=1, alpha=0.35, label="Phase3D (Soorya ref)") + ax.plot(s["hpi"], s["org_auroc"], color="darkorange", lw=1, ls="--", alpha=0.35, label="SEC61B (Soorya ref)") + ax.axhline(0.5, color="gray", ls=":", lw=1, alpha=0.7) + ax.set_xlabel("Time post-infection (hpi)") + ax.set_ylabel("AUROC") + ax.set_ylim(0, 1) + ax.set_title( + "Per-timepoint AUROC — SEC61B organelle vs Phase3D classifier\nheld-out val · ground truth = ZIKV vs uninfected" + ) + ax.legend(fontsize=8) + fig.tight_layout() + fig.savefig(OUT_DIR / "timecourse_auroc_sec61b_vs_phase3d.png", dpi=120, bbox_inches="tight") + plt.close(fig) + print( + f" D1: wrote timecourse_auroc_sec61b_vs_phase3d.{{csv,png}} " + f"(phase mean {merged['phase_auroc'].mean():.3f}, org mean {merged['org_auroc'].mean():.3f})" + ) + + +# ---- D2: probability-split grid -------------------------------------------- # +def plot_prob_grid( + marker: str, obs: pd.DataFrame, p_remodel: np.ndarray, channel: str, projection: str, cmap, plate, rng +) -> None: + """One grid: rows = HPI bins, columns = 5 P(remodel) buckets, N cells each.""" + df = obs.copy().reset_index(drop=True) + df["p_remodel"] = p_remodel + df["fov_name"] = df["fov_name"].astype(object).str.strip("/") + df["bin"] = pd.cut(df["hours_post_perturbation"], bins=HPI_BINS, right=False) + bins = [b for b in df["bin"].cat.categories if (df["bin"] == b).any()] + + ch_idx = plate.channel_names.index(channel) + + def crop_row(row) -> np.ndarray: + pos = plate[str(row["fov_name"]).strip("/")] + t = int(row["t"]) + z = focus_z(pos, marker, t) if projection == "focus" else None + return crop(pos["0"], t, ch_idx, int(row["y"]), int(row["x"]), projection, z) + + scale_rows = df.sample(min(SCALE_SAMPLE, len(df)), random_state=SEED) + pooled = np.concatenate([crop_row(r).ravel() for _, r in scale_rows.iterrows()]) + vmin, vmax = np.percentile(pooled, DISPLAY_PCTL) + vmax = max(vmax, vmin + 1) + + ncols = N_PER_CELL * len(PROB_BUCKETS) + fig, axes = plt.subplots(len(bins), ncols, figsize=(0.9 * ncols, 1.2 * len(bins)), squeeze=False) + for r, b in enumerate(bins): + for c_i, (blabel, lo, hi) in enumerate(PROB_BUCKETS): + sub = df[(df["bin"] == b) & (df["p_remodel"] >= lo) & (df["p_remodel"] < hi)] + picks = sub.sample(min(N_PER_CELL, len(sub)), random_state=int(rng.integers(1e9))) if len(sub) else sub + for k in range(N_PER_CELL): + ax = axes[r][c_i * N_PER_CELL + k] + if k < len(picks): + ax.imshow(crop_row(picks.iloc[k]), cmap=cmap, vmin=vmin, vmax=vmax) + ax.set_xticks([]) + ax.set_yticks([]) + else: + ax.axis("off") + axes[r][0].set_ylabel(f"{int(b.left)}-{int(b.right)}h", fontsize=8, color="white") + + fig.patch.set_facecolor("black") + fig.tight_layout() + fig.subplots_adjust(top=0.96) + fig.suptitle(f"{marker} — {N_PER_CELL} cells per HPI bin x P(remodel) bucket", fontsize=12, color="white", y=0.995) + for c_i, (blabel, _lo, _hi) in enumerate(PROB_BUCKETS): + left = axes[0][c_i * N_PER_CELL].get_position().x0 + right = axes[0][c_i * N_PER_CELL + N_PER_CELL - 1].get_position().x1 + fig.text((left + right) / 2, 0.97, f"P(remodel)={blabel}", ha="center", va="bottom", fontsize=11, color="white") + for c_i in range(1, len(PROB_BUCKETS)): + x_prev = axes[0][c_i * N_PER_CELL - 1].get_position().x1 + x_next = axes[0][c_i * N_PER_CELL].get_position().x0 + xdiv = (x_prev + x_next) / 2 + fig.add_artist(plt.Line2D([xdiv, xdiv], [0.02, 0.96], color="white", lw=1.2, alpha=0.6)) + + OUT_DIR.mkdir(parents=True, exist_ok=True) + out = OUT_DIR / f"prob_samples_{marker}.png" + fig.savefig(out, dpi=120, bbox_inches="tight", facecolor="black") + plt.close(fig) + print(f" D2: wrote {out} ({N_PER_CELL} cells x {len(bins)} bins x {len(PROB_BUCKETS)} buckets)") + + +# ---- D3: % remodeled vs time (both markers, ZIKV + control) ---------------- # +def _pct_pos_by_hpi(hpi: np.ndarray, is_pos: np.ndarray) -> pd.DataFrame: + """Per-hpi % positive ± Bernoulli SEM over a cell population (paper convention).""" + df = pd.DataFrame({"t": np.round(np.asarray(hpi), 1), "is_pos": np.asarray(is_pos, dtype=float)}) + rows = [] + for t, g in df.groupby("t"): + n = len(g) + p = float(g["is_pos"].mean()) + rows.append({"t": t, "pct": 100.0 * p, "sem": 100.0 * np.sqrt(p * (1 - p) / n), "n": n}) + return pd.DataFrame(rows).sort_values("t") + + +def plot_percent_remodeled(series: list[dict]) -> None: + """% cells 'remodel' (P>=thr) vs hpi, ZIKV (solid) vs control FP (dashed), for N series. + + Each ``series`` entry: ``{name, color, obs, prob, thr}``. The % is over the WHOLE well + population per hpi (positives / all cells), ± Bernoulli SEM — the reference + ``plot_remodeling_vs_time`` convention. Plots all series on one axis so the SEC61B + witness-GMM gate (label source), the SEC61B same-modality LR, and the Phase3D + cross-modality LR are directly comparable. + """ + fig, ax = plt.subplots(figsize=(11, 5.5)) + csv_rows = [] + for spec in series: + name, color, obs, prob, thr = spec["name"], spec["color"], spec["obs"], spec["prob"], spec["thr"] + obs = obs.reset_index(drop=True) + is_pos = np.asarray(prob) >= thr + hpi = obs["hours_post_perturbation"].to_numpy() + pert = obs["perturbation"].to_numpy() + for cond, style, alpha in [("ZIKV", "-", 1.0), ("uninfected", "--", 0.6)]: + m = pert == cond + if not m.any(): + continue + stats = _pct_pos_by_hpi(hpi[m], is_pos[m]) + rate = 100.0 * float(is_pos[m].mean()) + if cond == "ZIKV": + label = f"{name} ZIKV (n={int(stats['n'].sum())})" + else: + label = f"{name} control FP ({rate:.1f}%)" + ax.errorbar( + stats["t"], + stats["pct"], + yerr=stats["sem"], + marker="o", + markersize=3, + capsize=2, + linewidth=1.7 if cond == "ZIKV" else 1.0, + linestyle=style, + alpha=alpha, + color=color, + label=label, + ) + s = stats.copy() + s["series"] = name + s["condition"] = cond + csv_rows.append(s) + + ax.set_ylim(-5, 105) + ax.set_title( + "% cells 'remodel' vs time — witness-GMM gate + same/cross-modality classifiers", fontsize=11, fontweight="bold" + ) + ax.text( + 0.5, + -0.15, + "SEC61B witness-GMM = label source (thr 0.8) · SEC61B LR = same-modality, near-ceiling (thr 0.5) · " + "Phase3D LR = cross-modality teacher/student (thr 0.5)", + transform=ax.transAxes, + ha="center", + va="top", + fontsize=7, + color="0.4", + ) + ax.set_xlabel("hours post perturbation", fontsize=11) + ax.set_ylabel("% cells 'remodel'", fontsize=11) + ax.grid(True, alpha=0.3) + ax.legend(frameon=True, fontsize=7.5, ncol=3, loc="upper left") + OUT_DIR.mkdir(parents=True, exist_ok=True) + pd.concat(csv_rows, ignore_index=True).to_csv(OUT_DIR / "percent_remodeled_vs_time.csv", index=False) + fig.savefig(OUT_DIR / "percent_remodeled_vs_time.png", dpi=150, bbox_inches="tight") + plt.close(fig) + print(" D3: wrote percent_remodeled_vs_time.{png,csv}") + + +def main() -> None: + rng = np.random.default_rng(SEED) + print("Loading embeddings ...") + sec = ad.read_zarr(SEC61B_EMB) + sec.obs_names_make_unique() + phase = ad.read_zarr(PHASE_EMB) + phase.obs_names_make_unique() + + # The SEC61B classifier is trained/valid only on the SEC61B wells (A/2 uninfected, + # B/2 ZIKV). The Phase3D zarr spans all 6 wells (A/3-4, B/3-4 carry TOMM20/G3BP1, unseen + # by the classifier). Restrict Phase3D to the SEC61B wells so both models are graded on + # the same in-domain cell population and the overlay is apples-to-apples. + sec_wells = sorted({"/".join(str(f).strip("/").split("/")[:2]) for f in sec.obs["fov_name"]}) + phase_well = phase.obs["fov_name"].astype(object).map(lambda f: "/".join(str(f).strip("/").split("/")[:2])) + phase = phase[phase_well.isin(sec_wells).to_numpy()].copy() + phase.obs_names_make_unique() + print(f"Restricted Phase3D to SEC61B wells {sec_wells}: {phase.n_obs} cells") + + print("Recomputing per-cell P(remodel) ...") + # GMM posterior (label source) — drives D1 SEC61B witness AUROC + D2 SEC61B grid. + sec_p_gmm, sec_scores = sec61b_p_remodel(sec) + # Same-modality LR — drives D3 so both markers are the SAME model type at the SAME thr. + sec_p_lr = lr_p_remodel(sec, SEC61B_PIPELINE) + phase_p = lr_p_remodel(phase, PHASE_PIPELINE) + + print("D1 — AUROC overlay ...") + plot_timecourse(sec.obs, sec_scores, phase.obs, phase_p) + + print("D3 — % remodeled vs time (witness gate + SEC61B LR + Phase3D LR) ...") + plot_percent_remodeled( + [ + { + "name": "SEC61B witness-GMM", + "color": "tab:green", + "obs": sec.obs, + "prob": sec_p_gmm, + "thr": GMM_POS_THRESHOLD, + }, + {"name": "SEC61B LR", "color": "tab:olive", "obs": sec.obs, "prob": sec_p_lr, "thr": 0.5}, + {"name": "Phase3D LR", "color": "tab:blue", "obs": phase.obs, "prob": phase_p, "thr": 0.5}, + ] + ) + + print("D2 — probability-split grids ...") + with open_ome_zarr(DATA_ZARR, mode="r") as plate: + plot_prob_grid("SEC61B", sec.obs, sec_p_gmm, "raw GFP EX488 EM525-45", "mip", CMAP_GREEN, plate, rng) + plot_prob_grid("Phase3D", phase.obs, phase_p, "Phase3D", "focus", "gray", plate, rng) + print("Done.") + + +if __name__ == "__main__": + main() diff --git a/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/run_witness_gmm_phase_classifier.sh b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/run_witness_gmm_phase_classifier.sh new file mode 100644 index 000000000..92e399162 --- /dev/null +++ b/applications/dynaclr/configs/evaluation/witness_gmm/phase_classifier/run_witness_gmm_phase_classifier.sh @@ -0,0 +1,60 @@ +#!/bin/bash +# Witness-GMM phase classifier — end-to-end Stage A -> Stage B on the ZIKV plate. +# +# Reproduces Soorya's sec61b_timelapse_zikv_classifier.py flow through the +# frameworked two-stage pipeline, on dataset +# 2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV (mantis v2): +# +# Stage A dynaclr witness-gmm-labels (SEC61B organelle -> remodeling labels) +# Stage B dynaclr run-linear-classifiers (train Phase3D classifier on those labels) +# +# Both stages are CPU-only (MMD/GMM + sklearn LR). Run directly on a login/compute +# shell, or submit with `sbatch` (SLURM header below). +# +# Usage: +# bash run_witness_gmm_phase_classifier.sh # run both stages +# bash run_witness_gmm_phase_classifier.sh --stage-a # Stage A only +# bash run_witness_gmm_phase_classifier.sh --stage-b # Stage B only (needs Stage A output) +# sbatch run_witness_gmm_phase_classifier.sh # submit as a batch job +# +#SBATCH --job-name=witness_gmm_phase +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --partition=cpu +#SBATCH --cpus-per-task=8 +#SBATCH --mem=32G +#SBATCH --time=04:00:00 + +set -euo pipefail +export PYTHONNOUSERSITE=1 + +WORKSPACE_DIR="${WORKSPACE_DIR:-/hpc/mydata/eduardo.hirata/repos/viscy}" +CONFIG_DIR="$WORKSPACE_DIR/applications/dynaclr/configs/evaluation/witness_gmm_phase_classifier" +STAGE_A_CONFIG="$CONFIG_DIR/01_stage_a_sec61b_labels.yml" +STAGE_B_PHASE_CONFIG="$CONFIG_DIR/02_stage_b_train_phase.yml" +STAGE_B_SEC61B_CONFIG="$CONFIG_DIR/02_stage_b_train_sec61b.yml" + +cd "$WORKSPACE_DIR" + +RUN_A=1 +RUN_B=1 +case "${1:-}" in + --stage-a) RUN_B=0 ;; + --stage-b) RUN_A=0 ;; + "") ;; + *) echo "Unknown argument: $1"; exit 1 ;; +esac + +if [[ $RUN_A -eq 1 ]]; then + echo "## Stage A — witness-GMM labels (SEC61B organelle)" + uv run --project "$WORKSPACE_DIR" dynaclr witness-gmm-labels -c "$STAGE_A_CONFIG" +fi + +if [[ $RUN_B -eq 1 ]]; then + echo "## Stage B — train Phase3D classifier from SEC61B labels (teacher/student)" + uv run --project "$WORKSPACE_DIR" dynaclr run-linear-classifiers -c "$STAGE_B_PHASE_CONFIG" + echo "## Stage B — train SEC61B classifier from SEC61B labels (same-modality)" + uv run --project "$WORKSPACE_DIR" dynaclr run-linear-classifiers -c "$STAGE_B_SEC61B_CONFIG" +fi + +echo "## Done." diff --git a/applications/dynaclr/configs/matrix/example.yml b/applications/dynaclr/configs/matrix/example.yml new file mode 100644 index 000000000..ba4ae0b52 --- /dev/null +++ b/applications/dynaclr/configs/matrix/example.yml @@ -0,0 +1,70 @@ +# Example model matrix — reference for every `dynaclr run-matrix` feature. +# +# One matrix sweeps many models through predict → eval in parallel (SLURM). Rows +# run concurrently; stages chain within a row via --dependency=afterok. +# +# uv run dynaclr run-matrix -c --dry-run # inspect the chained commands +# uv run dynaclr run-matrix -c # default --stages predict,eval +# uv run dynaclr run-matrix -c --stages train,predict,eval # also submit training +# uv run dynaclr run-matrix -c --overwrite # re-embed all (default skips existing) +# +# Skip-existing (DEFAULT): before submitting, the matrix prunes datasets whose +# embeddings already exist for a row's (model_family, run, ckpt_name); fully-done +# rows are dropped. Append a dataset to a collection and re-run → only it embeds. +# `--overwrite` bypasses the check. +# +# `defaults:` holds fields shared across rows; each `models:` row overrides/adds. + +defaults: + ckpt_name: last # → checkpoints/last.ckpt (pin an epoch with e.g. epoch105-step84800) + collection: applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_14_A549_SEC61B_DENV.yml + eval_config: applications/dynaclr/configs/evaluation/DynaCLR-2D-MIP-BagOfChannels/infectomics-annotated.yaml + datasets_root: /hpc/projects/intracellular_dashboard/organelle_dynamics + markers: [SEC61B, Phase3D] # run exactly these markers (extras in the collection ignored) + # Z selection — EITHER focus-centered (window WIDTH, centered per-FOV on focus_slice) + # OR fixed (z_range absolute start/stop). Mutually exclusive. z_reduction mip → + # fluorescence max-projected, label-free (Phase3D) center slice. + predict_flags: + z_window: 16 + focus_channel: Phase3D + z_focus_offset: 0.3 + z_reduction: mip + reference_pixel_size: 0.1494 + reference_pixel_size_z_um: 0.174 + # z_range: [15, 45] # (alternative) fixed absolute window instead of z_window + +models: + # --- DynaCLR rows (model_type defaults to "dynaclr"): identity from the training .sh --- + # family / run / train_configs are parsed from the .sh's export PROJECT / RUN_NAME / CONFIGS. + + # (a) Train→predict→eval a NEW model (needs --stages train,predict,eval): last.ckpt derived. + - train_sbatch: applications/dynaclr/configs/training/DynaCLR-2D/DynaCLR-2D-MIP-BagOfChannels.sh + + # (b) Predict/eval an ALREADY-trained model. Lightning nests checkpoints under a + # {run}/{PROJECT}/{wandb_id}/checkpoints/ subdir (not derivable from identity) → + # give the explicit `checkpoint:` path. + - train_sbatch: applications/dynaclr/configs/training/DynaCLR-2D/DynaCLR-2D-MIP-BagOfChannels-single-marker.sh + ckpt_name: epoch105-step84800 + checkpoint: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/DynaCLR-2D-MIP-BagOfChannels/jbrwhzr3/checkpoints/epoch=105-step=84800.ckpt + + # (c) Checkpoint sweep: one train, then a predict→eval chain per checkpoint. + # - train_sbatch: applications/dynaclr/configs/training/DynaCLR-2D/DynaCLR-2D-MIP-BagOfChannels-single-marker.sh + # ckpt_names: [last, epoch105-step84800] + + # --- Foundation rows (model_type: foundation): frozen encoders, no checkpoint --- + # No train_sbatch — give explicit identity + a LightningCLI training_config whose + # model: block builds the frozen encoder (weights load inside the wrapper). Foundation + # rows never train; they share the same predict output tree + eval link as DynaCLR. + - model_type: foundation + family: MorphEm-frozen + run: frozen + ckpt_name: frozen + training_config: applications/dynaclr/configs/evaluation/MorphEm-frozen/training_config_morphem_frozen.yaml + eval_config: applications/dynaclr/configs/evaluation/MorphEm-frozen/infectomics-annotated.yaml + + # - model_type: foundation + # family: CELL-DINO-frozen + # run: frozen + # ckpt_name: frozen + # training_config: applications/dynaclr/configs/evaluation/CELL-DINO-frozen/training_config_cell_dino_frozen.yaml + # eval_config: applications/dynaclr/configs/evaluation/CELL-DINO-frozen/infectomics-annotated.yaml diff --git a/applications/dynaclr/configs/matrix/organelle_remodeling.yml b/applications/dynaclr/configs/matrix/organelle_remodeling.yml new file mode 100644 index 000000000..863ff7dc5 --- /dev/null +++ b/applications/dynaclr/configs/matrix/organelle_remodeling.yml @@ -0,0 +1,69 @@ +# Organelle-remodeling model matrix — persistent, iterated on over time. +# +# ONE merged collection (organelle_remodeling.yml = 15 physical datasets spanning +# SEC61B/TOMM20/G3BP1/CAAX/H2B under ZIKV/DENV) run through MULTIPLE models. One +# row per MODEL; run-matrix fans out predict per experiment inside the collection +# and skips already-embedded ones. +# +# dynaclr run-matrix -c applications/dynaclr/configs/matrix/organelle_remodeling.yml --stages predict --dry-run +# dynaclr run-matrix -c applications/dynaclr/configs/matrix/organelle_remodeling.yml --stages predict +# +# Progressive: skip-existing is the DEFAULT — datasets already embedded for a row's +# (model_family, run, ckpt_name) are pruned; append a dataset to the collection and +# re-run → only it embeds. --overwrite forces a full re-embed (e.g. after changing +# predict_flags). + +defaults: + collection: applications/dynaclr/configs/collections/organelle_remodeling.yml + # OUTPUT root: embeddings + 2-phenotyping/ land here (intracellular_dashboard). + # This is independent of the INPUT root — the collection's own datasets_root + # (VAST) still resolves each experiment's data_path/tracks_path for reading. + datasets_root: /hpc/projects/intracellular_dashboard/organelle_dynamics + # Organelle reporters + label-free. Sensor markers differ by virus (pAL17 ZIKV / + # pAL40 DENV); include both to embed the sensor across all datasets, or drop them + # to embed only the organelle question. + # (CAAX/H2B are the 2026_03_26 dual-virus markers; a marker only runs where present.) + markers: [SEC61B, TOMM20, G3BP1, Phase3D, pAL17, pAL40, CAAX, H2B] + # Focus-centered Z: window WIDTH (slices) centered per-FOV on the focus plane + # (from focus_slice zattrs). z_reduction mip → fluorescence max-projected, + # label-free (Phase3D) center slice. z_focus_offset = fraction below focus. + predict_flags: + z_window: 16 + focus_channel: Phase3D + z_focus_offset: 0.3 + z_reduction: mip + reference_pixel_size: 0.1494 + reference_pixel_size_z_um: 0.174 + +models: + # --- DynaCLR-2D-MIP-BagOfChannels (single-marker), already trained --- + # Lightning nests the checkpoint under {run}/{PROJECT}/{wandb_id}/checkpoints/, so + # give the explicit path (the derived {run}/checkpoints/ default is empty). + - train_sbatch: applications/dynaclr/configs/training/DynaCLR-2D/DynaCLR-2D-MIP-BagOfChannels-single-marker.sh + ckpt_name: epoch105-step84800 + checkpoint: /hpc/projects/organelle_phenotyping/models/DynaCLR-2D-MIP-BagOfChannels/2d-mip-ntxent-t0p2-lr2e5-bs256-192to160-zext11-single-marker-fix-shuffler/DynaCLR-2D-MIP-BagOfChannels/jbrwhzr3/checkpoints/epoch=105-step=84800.ckpt + eval_config: applications/dynaclr/configs/evaluation/DynaCLR-2D-MIP-BagOfChannels-single-marker-fix-shuffler/infectomics-annotated.yaml + + # --- MorphEm (frozen ViT-S foundation baseline) over the SAME datasets --- + - model_type: foundation + family: MorphEm-frozen + run: frozen + ckpt_name: frozen + training_config: applications/dynaclr/configs/evaluation/MorphEm-frozen/training_config_morphem_frozen.yaml + eval_config: applications/dynaclr/configs/evaluation/MorphEm-frozen/infectomics-annotated.yaml + + # --- CELL-DINO (frozen channel-adaptive ViT-L/16 foundation baseline) --- + - model_type: foundation + family: CELL-DINO-frozen + run: frozen + ckpt_name: frozen + training_config: applications/dynaclr/configs/evaluation/CELL-DINO-frozen/training_config_cell_dino_frozen.yaml + eval_config: applications/dynaclr/configs/evaluation/CELL-DINO-frozen/infectomics-annotated.yaml + + # --- DINOv3 (frozen ConvNeXt-Tiny foundation baseline) --- + - model_type: foundation + family: DINOv3-frozen + run: frozen + ckpt_name: frozen + training_config: applications/dynaclr/configs/evaluation/DINOv3-frozen/training_config_dinov3_frozen.yaml + eval_config: applications/dynaclr/configs/evaluation/DINOv3-frozen/infectomics-annotated.yaml diff --git a/applications/dynaclr/configs/prediction/predict_triplet_2d_from_3d.yml b/applications/dynaclr/configs/prediction/predict_triplet_2d_from_3d.yml new file mode 100644 index 000000000..a4ac50cbf --- /dev/null +++ b/applications/dynaclr/configs/prediction/predict_triplet_2d_from_3d.yml @@ -0,0 +1,86 @@ +# Sample triplet predict config: feed a 3D OME-Zarr to a 2D model. +# +# Demonstrates the TripletDataModule options that avoid materializing a separate +# 2D MIP dataset: +# - z_reduction: collapse the extracted z_range window to a single slice +# (label-free channels -> center slice, others -> max projection). +# - reference_pixel_size: rescale each patch to the model's training pixel size +# when the inference dataset was acquired at a different magnification. +# - reference_pixel_size_z_um: convert the reference-grid Z window to the +# native slice count covering the same physical depth, then resize it back +# to the reference Z grid before projection. +# +# See docs/DAGs/inference_triplet.md for the full pipeline. +# +# TODO: point to the path to save the embeddings +# TODO: point to the path to the data +# TODO: point to the path to the tracks +# TODO: point to the path to the checkpoint + +seed_everything: 42 +trainer: + accelerator: gpu + strategy: auto + devices: 1 + num_nodes: 1 + precision: 32-true + inference_mode: true + logger: false + callbacks: + - class_path: lightning.pytorch.callbacks.TQDMProgressBar + init_args: + refresh_rate: 10 + - class_path: viscy_utils.callbacks.embedding_writer.EmbeddingWriter + init_args: + output_path: #TODO point to the path to save the embeddings (e.g. /embeddings/embeddings.zarr) + embedding_key: features # "projections" for frozen-backbone MLP heads + overwrite: true + pca_kwargs: null # reductions left to a later step (dynaclr reduce-dimensionality) + phate_kwargs: null + umap_kwargs: null +model: + class_path: dynaclr.engine.ContrastiveModule + init_args: + encoder: + class_path: viscy_models.contrastive.ContrastiveEncoder + init_args: + backbone: convnext_tiny + in_channels: 1 + in_stack_depth: 1 # 2D model — pairs with data.z_reduction below + stem_kernel_size: [1, 4, 4] + stem_stride: [1, 4, 4] + embedding_dim: 768 + projection_dim: 32 + drop_path_rate: 0.0 # stochastic depth off at inference + example_input_array_shape: [1, 1, 1, 160, 160] +data: + class_path: viscy_data.triplet.TripletDataModule + init_args: + data_path: #TODO point to the path to the data (e.g. /registered_test.zarr) + tracks_path: #TODO point to the path to the tracks (e.g. /track_test.zarr) + source_channel: + - Phase3D + z_extraction_window: 10 # Reference-grid slices, centered per FOV. + focus_channel: Phase3D + z_focus_offset: 0.3 + z_reduction: mip # "mip" (max projection) or "center" (center slice). + # Label-free channels always take the center slice; + # others are max-projected. This sets the fallback only. + reference_pixel_size: 0.1494 # µm/px of the model's TRAINING dataset. Remove (or set + # null) when the inference dataset is at the same resolution. + reference_pixel_size_z_um: 0.174 # µm/slice of the model's training reference grid. + initial_yx_patch_size: [160, 160] # ignored when reference_pixel_size is set (computed from ratio) + final_yx_patch_size: [160, 160] # patch size fed to the model after rescale + z_reduction + batch_size: 32 + num_workers: 0 # REQUIRED for predict (avoids zarr-fork deadlock) + normalizations: + - class_path: viscy_transforms.NormalizeSampled + init_args: + keys: [Phase3D] + level: fov_statistics + subtrahend: mean + divisor: std + # augmentations omitted: predict must be deterministic. The datamodule still + # applies normalizations + reference_pixel_size rescale + z_reduction at predict time. +return_predictions: false +ckpt_path: #TODO point to the path to the checkpoint (e.g. /checkpoints/epoch=94-step=2375.ckpt) diff --git a/applications/dynaclr/configs/training/DynaCLR-2D/DynaCLR-2D-MIP-BagOfChannels.yml b/applications/dynaclr/configs/training/DynaCLR-2D/DynaCLR-2D-MIP-BagOfChannels.yml index 37c237f0f..7e9fe4072 100644 --- a/applications/dynaclr/configs/training/DynaCLR-2D/DynaCLR-2D-MIP-BagOfChannels.yml +++ b/applications/dynaclr/configs/training/DynaCLR-2D/DynaCLR-2D-MIP-BagOfChannels.yml @@ -1,7 +1,7 @@ # DynaCLR-2D-MIP-BagOfChannels # ============================== # 2D bag-of-channels contrastive learning with channel-wise z-reduction. -# Extracts a 20-slice z-stack around focus, randomly crops to 10 slices +# Extracts a 16-slice reference-grid z-stack around focus, randomly crops to 10 slices # (Z-invariance), then applies MIP for fluorescence and center-slice for # label-free (Phase3D, BF, DIC, Retardance). # Multi-cell-type: A549 infectomics, microglia dynamorph, ALFI mitosis. @@ -70,6 +70,10 @@ data: cell_index_path: /hpc/projects/organelle_phenotyping/models/collections/DynaCLR-2D-MIP-BagOfChannels-v3.parquet focus_channel: Phase3D reference_pixel_size_xy_um: 0.1494 + # Interpret every slice-based Z window/crop on the Mantis reference grid. + # Native stacks with different Z sampling are read at the equivalent + # physical depth and resampled before the 10-slice crop + MIP. + reference_pixel_size_z_um: 0.174 z_window: 1 z_extraction_window: 16 z_focus_offset: 0.3 @@ -141,7 +145,7 @@ data: prob: 0.5 mean: 0.0 std: 0.1 - # Random Z crop: select 10 of 20 extracted slices for Z-invariance. + # Random Z crop: select 10 of 16 reference-grid slices for Z-invariance. # Must come before ZReduction so MIP sees a variable sub-stack. - class_path: viscy_transforms.BatchedRandSpatialCropd init_args: diff --git a/applications/dynaclr/docs/DAGs/ai_ready_datasets.md b/applications/dynaclr/docs/DAGs/ai_ready_datasets.md index 8e000769f..f223c2900 100644 --- a/applications/dynaclr/docs/DAGs/ai_ready_datasets.md +++ b/applications/dynaclr/docs/DAGs/ai_ready_datasets.md @@ -1,92 +1,25 @@ -# Data Preparation DAG - -## Entry point - -`prepare run -c prepare_config.yaml` (from `airtable_utils`) discovers wells and -channels from NFS, generates all configs and SLURM scripts, and submits the pipeline. - -```bash -prepare run 2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV \ - -c /path/to/prepare_config.yaml - -# Dry-run: generate configs/scripts without submitting -prepare run 2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV \ - -c /path/to/prepare_config.yaml \ - --dry-run -``` - -## Step-by-step detail - -``` -NFS assembled zarr (intracellular_dashboard/organelle_dynamics/{dataset}/2-assemble/) - │ - ▼ -prepare run # discovers wells + channels from NFS zarr - │ airtable_utils.prepare_cli # validates dataset is in Airtable - │ airtable_utils.prepare # generates all configs and scripts - ▼ -{vast_output_dir}/ - ├── crop_concat.yml # biahub concatenate config (wells × channels) - ├── qc_config.yml # focus-slice QC config - ├── sbatch_overrides.sh # optional SLURM overrides for biahub's internal jobs - ├── 01_concatenate.sh # bash (not SLURM): runs biahub + rsync tracking - ├── 02_qc.sh # SLURM: GPU focus-slice detection - └── 03_preprocess.sh # SLURM: CPU normalization stats - │ - ▼ -bash 01_concatenate.sh # NOT a SLURM job — runs interactively - │ Step 1: conda run biahub concatenate -c crop_concat.yml -o {dataset}.zarr -m - │ biahub submits its own SLURM jobs internally via submitit; -m blocks until done - │ Step 2: rsync tracking zarr (NFS → VAST) - ▼ -{dataset}.zarr (OME-Zarr v0.5 / zarr v3, rechunked) -tracking.zarr (cell tracking results) - │ - ├──► sbatch 02_qc.sh # GPU (~30 min) - │ qc run -c qc_config.yml # focus-slice detection on Phase3D channel - │ → writes focus_slice metadata into {dataset}.zarr - │ - └──► sbatch 03_preprocess.sh # CPU, preempted partition (~4 hrs) - viscy preprocess # computes per-channel normalization stats - --data_path {dataset}.zarr - → writes normalization metadata into {dataset}.zarr -``` - -## Pipeline DAG (process dependency) - -``` -NFS zarr (assembled) - │ - ▼ -prepare run ──── generates configs + scripts - │ - ▼ -01_concatenate.sh (interactive bash, blocks until biahub SLURM jobs finish) - │ - ▼ -{dataset}.zarr + tracking.zarr - │ - ├──► 02_qc.sh (SLURM, GPU) → focus_slice metadata in zarr - └──► 03_preprocess.sh (SLURM, CPU) → normalization metadata in zarr +# Prepare an AI-ready dataset + +This workflow copies an assembled dataset from NFS to VAST, rechunks it as +OME-Zarr, copies tracking data, and adds focus and normalization metadata. + +```mermaid +flowchart TD + A["NFS assembled zarr + tracking zarr"] --> B["prepare run"] + B --> C["01_concatenate.sh
OME-Zarr on VAST + tracking copy"] + C --> D["02_qc.sh
focus_slice metadata"] + C --> E["03_preprocess.sh
normalization metadata"] + D --> F["AI-ready dataset"] + E --> F ``` -02_qc and 03_preprocess run in parallel (no dependency between them). -Both write metadata back to the same zarr; their outputs are checked by -`check_preprocessed()` before downstream training or evaluation. +`prepare run` executes `01_concatenate.sh`, waits for the internal biahub jobs, +then submits `02_qc.sh` and `03_preprocess.sh` in parallel. -## Key commands +## Required config - -| Step | Command | Input | Output | -| ----------------- | ------------------------------------------------- | ------------------ | --------------------------------------------------------------- | -| Generate + submit | `prepare run -c prepare_config.yaml` | NFS assembled zarr | scripts + configs, submits jobs | -| Status check | `prepare status -c prepare_config.yaml` | - | markdown table (NFS/VAST existence, zarr version, preprocessed) | -| Concatenate | `bash 01_concatenate.sh` | crop_concat.yml | {dataset}.zarr + tracking.zarr | -| QC | `sbatch 02_qc.sh` | qc_config.yml | focus_slice metadata in zarr | -| Preprocess | `sbatch 03_preprocess.sh` | {dataset}.zarr | normalization metadata in zarr | - - -## prepare_config.yaml format +Start from +[`applications/airtable/configs/prepare_config.yml`](../../../airtable/configs/prepare_config.yml). ```yaml nfs_root: /hpc/projects/intracellular_dashboard/organelle_dynamics @@ -94,70 +27,80 @@ vast_root: /hpc/projects/organelle_phenotyping/datasets workspace_dir: /hpc/mydata/eduardo.hirata/repos/viscy concatenate: - channel_names: null # null = auto-detect raw channels (Phase3D + "raw " prefix) + channel_names: null chunks_czyx: [1, 16, 256, 256] shards_ratio: [1, 1, 8, 8, 8] output_ome_zarr_version: "0.5" conda_env: biahub - sbatch_overrides: # optional: overrides for biahub's internal SLURM jobs - partition: preempted - mem-per-cpu: 16G qc: channel_names: [Phase3D] NA_det: 1.35 lambda_ill: 0.450 pixel_size: 0.1494 - midband_fractions: [0.125, 0.25] device: cuda - num_workers: 16 preprocess: - channel_names: -1 # -1 = all channels + channel_names: -1 num_workers: 32 - block_size: 32 slurm: qc: partition: gpu gres: gpu:1 cpus_per_task: 16 - mem_per_cpu: 4G time: "00:30:00" preprocess: - partition: preempted + partition: cpu cpus_per_task: 32 - mem_per_cpu: 4G time: "04:00:00" ``` -## Notes +Set dataset-specific acquisition parameters under `qc`. Leave +`concatenate.channel_names: null` to discover Phase3D and raw channels. + +## Run + +Check the current state: + +```sh +uv run --package airtable-utils prepare status \ + -c applications/airtable/configs/prepare_config.yml +``` + +Generate configs and scripts without execution: + +```sh +uv run --package airtable-utils prepare run \ + -c applications/airtable/configs/prepare_config.yml \ + --dry-run +``` -- `prepare run` validates the dataset exists in Airtable before generating anything. -Use `--force` to overwrite an existing VAST zarr (e.g. to upgrade from zarr v2 to v0.5). -- `01_concatenate.sh` is an interactive bash script, not a SLURM job. Run it from a login -node or an interactive session; it blocks until biahub's internal SLURM jobs finish (`-m` flag). -- `02_qc.sh` and `03_preprocess.sh` are independent — submit both immediately after -`01_concatenate.sh` completes; no need to wait for QC before running preprocess. -- Channel auto-detection (`channel_names: null`) keeps channels with prefix `Phase3D` or `raw` . -Virtual stains (`nuclei_prediction`, `membrane_prediction`) and deconvolved channels are excluded. -- `check_preprocessed()` checks for `normalization` key in zarr metadata; used by `prepare status` -and as a gate before evaluation. -- Raw channel names written to `crop_concat.yml` are repeated once per well entry — this is a -biahub concatenate requirement. +Run the complete preparation workflow: -## Path convention +```sh +uv run --package airtable-utils prepare run \ + -c applications/airtable/configs/prepare_config.yml +``` -All AI-ready data lives under `/hpc/projects/organelle_phenotyping/`: +Use `--force` only when an existing VAST zarr must be replaced. +## Outputs -| Directory | Contents | -| -------------------------- | --------------------------------------------- | -| `datasets//` | Zarr v3 store + `tracking.zarr` | -| `datasets/annotations/` | Per-experiment annotation CSVs | -| `models/collections/` | Cell index parquets (one per collection YAML) | -| `models//` | Training runs (checkpoints, WandB configs) | +```text +// +├── .zarr +├── tracking.zarr +├── crop_concat.yml +├── qc_config.yml +├── 01_concatenate.sh +├── 02_qc.sh +└── 03_preprocess.sh +``` +The dataset is ready when `prepare status` reports the expected OME-Zarr +version and both `focus_slice` and `normalization` metadata are present. -Collection YAMLs use `datasets_root: /hpc/projects/organelle_phenotyping` and -`${datasets_root}/datasets/...` placeholders — resolved at load time by `load_collection()`. +Continue with [training.md](training.md) to build a cell index, or +[inference_triplet.md](inference_triplet.md) to predict directly from the zarr +and tracking store. diff --git a/applications/dynaclr/docs/DAGs/end_to_end.md b/applications/dynaclr/docs/DAGs/end_to_end.md new file mode 100644 index 000000000..396c79b95 --- /dev/null +++ b/applications/dynaclr/docs/DAGs/end_to_end.md @@ -0,0 +1,107 @@ +# DynaCLR workflow overview + +This page is the entry point from an assembled dataset to reusable embeddings +and downstream evaluation. + +```mermaid +flowchart TD + A["Assembled image + tracking zarrs"] --> B["Prepare AI-ready dataset"] + B --> C["Collection YAML"] + C --> D["Train model
(optional when using an existing checkpoint)"] + C --> E["Predict per-marker embeddings"] + D --> E + E --> F["Frozen embedding zarrs"] + F --> G["Evaluation"] + F --> H["LOT correction"] + F --> I["Witness-GMM labels"] + F --> J["Pseudotime"] +``` + +## 1. Prepare the dataset + +```sh +uv run --package airtable-utils prepare run \ + -c applications/airtable/configs/prepare_config.yml +``` + +This produces an OME-Zarr and tracking store with focus and normalization +metadata. See [ai_ready_datasets.md](ai_ready_datasets.md). + +## 2. Create the collection + +Create or update a collection under: + +```text +applications/dynaclr/configs/collections/ +``` + +The collection is the shared input for cell-index construction and +`predict-triplet`. It must define each experiment's image store, tracking store, +channels, marker names, and well selection. + +## 3. Train or select a checkpoint + +To train a model: + +```sh +uv run dynaclr fit \ + -c applications/dynaclr/configs/training/.yml +``` + +See [training.md](training.md). Skip this stage when a compatible checkpoint +already exists. + +## 4. Predict embeddings + +```sh +uv run dynaclr predict-triplet \ + -c applications/dynaclr/configs/collections/.yml \ + --checkpoint /path/to/checkpoint.ckpt \ + --model-family \ + --run \ + --ckpt-name \ + --datasets-root /hpc/projects/intracellular_dashboard/organelle_dynamics \ + --z-range 15 45 \ + --z-reduction mip \ + --reference-pixel-size 0.1494 \ + --num-workers 0 +``` + +The command writes one zarr per experiment and marker: + +```text +//2-phenotyping/predictions/ + ///.zarr +``` + +See [inference_triplet.md](inference_triplet.md) for input requirements and +optional flags. + +## 5. Evaluate frozen embeddings + +Use the launcher when embeddings already exist: + +```sh +uv run dynaclr eval \ + --eval-config applications/dynaclr/configs/evaluation/.yaml \ + --model-family \ + --run \ + --ckpt-name +``` + +This launches the Nextflow `eval_from_embeddings` workflow. Reusing frozen +embeddings keeps prediction separate from downstream iteration. See +[evaluation.md](evaluation.md). + +## Downstream workflows + +| Workflow | Command | Documentation | +| --- | --- | --- | +| Batch evaluation | `dynaclr eval` or Nextflow `evaluation` | [evaluation.md](evaluation.md) | +| Evaluation matrix | `dynaclr run-matrix` | [evaluation_matrix.md](evaluation_matrix.md) | +| LOT correction | `dynaclr fit-lot-correction` / `apply-lot-correction` | [lot_correction.md](lot_correction.md) | +| Witness pseudo-labels | `dynaclr witness-gmm-labels` | [witness_gmm_classifiers.md](witness_gmm_classifiers.md) | +| Pseudotime scripts | staged Python CLIs | [pseudotime.md](pseudotime.md) | + +Use one stable tuple—`model-family`, `run`, and `checkpoint-name`—for the +prediction directory and every downstream invocation. diff --git a/applications/dynaclr/docs/DAGs/evaluation.md b/applications/dynaclr/docs/DAGs/evaluation.md index 5322c9202..e4cf6d517 100644 --- a/applications/dynaclr/docs/DAGs/evaluation.md +++ b/applications/dynaclr/docs/DAGs/evaluation.md @@ -1,682 +1,282 @@ -# Evaluation DAG +# Evaluate DynaCLR embeddings + +The evaluation workflow predicts or loads embeddings, then fans out +dimensionality reduction, plots, smoothness, MMD, and linear-classifier tasks. +Nextflow is the recommended runner. + +```mermaid +flowchart TD + A["Evaluation YAML"] --> B["prepare-eval-configs"] + B --> C{"Embedding source"} + C -->|"full workflow"| D["predict → split"] + C -->|"frozen embeddings"| E["embedding glob"] + D --> F["per-experiment zarrs"] + E --> F + F --> G["reduce + combined reduce"] + F --> H["smoothness"] + F --> I["MMD"] + F --> J["linear classifiers"] + J --> K["append labels and predictions"] + G --> L["plots"] + K --> L +``` -This document describes the **per-run** evaluation pipeline (one model on -one dataset). For the cross-model, cross-dataset matrix layout — including -the central linear-classifier registry that lets Wave-2 datasets fetch LC -pipelines trained on Wave-1 (infectomics-annotated) — see the companion -[`evaluation_matrix.md`](evaluation_matrix.md). +## Choose an entry point -## Running with Nextflow (recommended) +Use `evaluation` when prediction must run from a checkpoint and cell-index +parquet: -```bash +```sh module load nextflow/24.10.5 -nextflow run applications/dynaclr/nextflow/main.nf -entry evaluation \ - --eval_config applications/dynaclr/configs/evaluation/DynaCLR-2D-MIP-BagOfChannels/infectomics-annotated.yaml \ - --workspace_dir /hpc/mydata/eduardo.hirata/repos/viscy \ - -resume -``` - -`-resume` makes Nextflow skip steps whose outputs already exist. Re-run the same command after a failure — Nextflow picks up from where it left off. - -### Local test (no SLURM) - -```bash nextflow run applications/dynaclr/nextflow/main.nf \ - --eval_config applications/dynaclr/configs/evaluation/DynaCLR-2D-MIP-BagOfChannels_test.yaml \ - --workspace_dir /hpc/mydata/eduardo.hirata/repos/viscy \ - -profile local \ - -resume + -entry evaluation \ + --eval_config applications/dynaclr/configs/evaluation/.yaml \ + --workspace_dir /hpc/mydata/eduardo.hirata/repos/viscy \ + -resume ``` -## Pipeline entry point - -`dynaclr prepare-eval-configs` (also aliased as `dynaclr evaluate`) generates all YAML configs -under `output_dir/configs/` and prints a JSON manifest to stdout. Nextflow reads the manifest -to wire steps together. +Use `dynaclr eval` when per-marker embeddings already exist in the standard +prediction tree: -``` -eval_config.yaml - │ - ▼ -dynaclr prepare-eval-configs -c eval_config.yaml # writes configs/ + manifest JSON - │ - ▼ -output_dir/configs/ - ├── eval.yaml # copy of input config (for re-runs) - ├── predict.yml # GPU step: viscy predict - ├── reduce.yaml # template: dynaclr reduce-dimensionality (per-experiment) - ├── reduce_combined.yaml # CPU step: dynaclr combined-dim-reduction (joint) - ├── smoothness.yaml # template: dynaclr evaluate-smoothness (per-experiment) - ├── plot.yaml # template: dynaclr plot-embeddings (per-experiment) — only when "plot" in steps - ├── plot_combined.yaml # CPU step: dynaclr plot-embeddings (all experiments) — only when "plot_combined" in steps - ├── {block_name}.yaml # template: dynaclr compute-mmd (per-experiment, per-block) - ├── {block_name}_cross_exp.yaml # CPU step: dynaclr compute-mmd --combined (per-block) - └── linear_classifiers.yaml # CPU step (optional) +```sh +uv run dynaclr eval \ + --eval-config applications/dynaclr/configs/evaluation/.yaml \ + --model-family \ + --run \ + --ckpt-name ``` -## Step-by-step detail +Optional launcher filters: -``` -checkpoint.ckpt + cell_index.parquet - │ - ▼ -viscy predict -c predict.yml # MultiExperimentDataModule predict mode - │ EmbeddingWriter callback # normalizations + z_reduction, no augmentations - ▼ # obs: fov_name, id, t, track_id, -embeddings/embeddings.zarr # experiment, marker, perturbation, - │ (AnnData: .X=features, # hours_post_perturbation, organelle, well, microscope - │ .obs=cell metadata) - │ - ▼ -dynaclr split-embeddings \ - --input embeddings/embeddings.zarr \ - --output-dir embeddings/ - │ Splits by obs["experiment"], deletes combined zarr - │ Also writes configs/viewer.yaml (datasets: {exp: {hcs_plate, anndata}}) - │ hcs_plate read from obs["store_path"] of each split zarr - ▼ -embeddings/{experiment_A}.zarr -embeddings/{experiment_B}.zarr - ... -configs/viewer.yaml # nd-embedding viewer config (also valid input - ... # for combined-dim-reduction via datasets: key) - │ - ├──► dynaclr reduce-dimensionality # PCA only (per experiment, parallel SLURM jobs) - │ -c reduce.yaml # __ZARR_PATH__ substituted by Nextflow - │ → {experiment}.zarr (obsm: X_pca) - │ NOTE: skip PHATE here to avoid computing it twice - │ - │ (after reduce-dimensionality finishes for ALL experiments) - │ - ├──► dynaclr combined-dim-reduction # joint PCA + PHATE across all experiments - │ -c reduce_combined.yaml # fits on concatenated embeddings - │ → {experiment}.zarr (obsm: X_pca_combined, X_phate_combined) - │ - │ (after combined-dim-reduction finishes) - │ - ├──► dynaclr plot-embeddings # per-experiment PCA scatter (X_pca) — when "plot" in steps - │ -c plot.yaml # parallel SLURM jobs, one per experiment - │ → plots/{experiment}/*.pdf - │ - ├──► dynaclr plot-embeddings # all-experiments combined (X_pca_combined, X_phate_combined) - │ -c plot_combined.yaml # only when "plot_combined" in steps; one job - │ → plots/combined/*.pdf - │ - ├──► dynaclr evaluate-smoothness # temporal smoothness + dynamic range - │ -c smoothness.yaml # parallel SLURM jobs, one per experiment - │ → smoothness/{model}_per_marker_smoothness.csv # one row per marker - │ → smoothness/{model}_smoothness_stats.csv # mean ± std across markers - │ → smoothness/*.pdf # per-marker + per-model plots - │ - ├──► dynaclr compute-mmd # one SLURM job per (experiment, block) - │ -c {block_name}.yaml # __ZARR_PATH__ substituted by Nextflow - │ → mmd/{block_name}/mmd_results.csv - │ → mmd/{block_name}/kinetics.pdf - │ → mmd/{block_name}/activity_heatmap.pdf - │ - ├──► dynaclr compute-mmd --combined # pairwise cross-experiment batch effect detection - │ -c {block_name}_cross_exp.yaml # only generated when combined_mode: true - │ # For each marker shared by a pair of experiments, runs MMD per - │ # (condition, time_bin) after per-pair mean centering. - │ # Conditions are auto-discovered from data intersection. - │ → mmd/{block_name}_cross_exp/combined_mmd_results.csv - │ → mmd/{block_name}_cross_exp/kinetics.pdf - │ → mmd/{block_name}_cross_exp/activity_heatmap.pdf - │ - ├──► dynaclr run-linear-classifiers # logistic regression probe - │ -c linear_classifiers.yaml # reads per-experiment zarrs directory + annotation CSVs - │ # joins annotations on (fov_name, t, track_id); trains one LogisticRegression - │ # per (task, marker); marker_filters omitted → auto-discovers all markers - │ # writes trained pipelines to linear_classifiers/pipelines/ (in-run staging) - │ # if publish_dir is set: atomically promotes the bundle to the central - │ # LC registry as {publish_dir}/vN/ and updates the `latest` symlink. - │ → linear_classifiers/metrics_summary.csv - │ → linear_classifiers/{task}_summary.pdf - │ → linear_classifiers/pipelines/{task}_{marker}.joblib - │ → linear_classifiers/pipelines/manifest.json - │ → {publish_dir}/vN/{task}_{marker}.joblib (when publish_dir set) - │ → {publish_dir}/vN/manifest.json (when publish_dir set) - │ → {publish_dir}/latest -> vN (atomic symlink swap) - │ - ├──► dynaclr append-annotations # persist ground truth labels to per-experiment zarrs - │ -c append_annotations.yaml # reads annotation CSVs + writes task columns to zarr obs - │ # only experiments with AnnotationSource entries are processed; others skipped - │ → {experiment}.zarr (obs: infection_state, organelle_state, ...) - │ - └──► dynaclr append-predictions # apply saved classifiers - -c append_predictions.yaml # predicts on ALL cells per marker, not just annotated ones - # pipelines_dir may be either: - # (a) in-run: {output_dir}/linear_classifiers/pipelines/ (default), or - # (b) external: a `latest` symlink into the central LC registry - # (e.g., /hpc/.../linear_classifiers/{model_name}/latest) - # The symlink is resolved once at startup so the run is consistent - # even if a new bundle is published mid-run. Logs feature_space (= - # registry/{model_name}) and version (= vN) for traceability. - → {experiment}.zarr (obs: predicted_infection_state, ...) - → {experiment}.zarr (obsm: predicted_infection_state_proba, ...) - → {experiment}.zarr (uns: predicted_infection_state_classes, - predicted_infection_state_lc_version, - predicted_infection_state_lc_feature_space, - predicted_infection_state_lc_path, ...) - -checkpoint.ckpt (independent of predict/split — runs in parallel) - │ - ▼ -viscy export -c export_onnx.yml # export backbone to ONNX - │ - ▼ -model.onnx + CTC datasets ({seq}_ERR_SEG/, {seq}/, {seq}_GT/TRA/) - │ - ▼ -dynaclr evaluate-tracking-accuracy \ # ILP tracking on CTC benchmarks - -c tracking_accuracy.yaml # loops over (model, dataset, sequence) - │ builds tracksdata graph from segmentation masks - │ runs ONNX inference on cell crops → dynaclr_similarity edge cost - │ solves ILP; compares to GT via evaluate_ctc_metrics - │ set show_napari: true for interactive inspection - ▼ -tracking_accuracy/results.csv # one row per (model, dataset, sequence) -tracking_accuracy/ # grouped mean summary printed to stdout +```sh +--marker SEC61B +--datasets --datasets +--print-cmd ``` -After all enrichment steps complete, per-experiment zarrs contain: +The equivalent direct frozen-embedding entry is: -- `.obs`: embeddings metadata + annotations (`infection_state`, etc.) + predictions (`predicted_infection_state`, etc.) -- `.obsm`: `X_pca`, `X_pca_combined`, `X_phate_combined`, `predicted_{task}_proba` -- `.uns`: `predicted_{task}_classes`, `predicted_{task}_lc_version`, `predicted_{task}_lc_feature_space`, `predicted_{task}_lc_path` - -This enables plots colored by experiment, perturbation, annotation, and prediction from a single zarr. The `_lc_*` uns fields record exactly which LC bundle produced each predicted column (registry path, version tag, feature_space). - -## Central LC registry - -Linear-classifier pipelines can be **published** to a central per-model -registry instead of (or in addition to) the per-run `output_dir`. This lets -later evaluations on different datasets reuse the same trained classifiers -without retraining. - -### Layout - -``` -/hpc/projects/organelle_phenotyping/models/linear_classifiers/ -├── DynaCLR-2D-MIP-BagOfChannels/ -│ ├── latest -> v3 # symlink (relative target) -│ ├── v1/ {manifest.json, *.joblib} -│ ├── v2/ -│ └── v3/ -├── DynaCLR-2D-BagOfChannels-v3/ { same } -├── DynaCLR-classical/ { same } -├── DINOv3-temporal-MLP-2D-BagOfChannels-v1/ { same } -└── DINOv3-frozen/ { same } +```sh +nextflow run applications/dynaclr/nextflow/main.nf \ + -entry eval_from_embeddings \ + --eval_config applications/dynaclr/configs/evaluation/.yaml \ + --embeddings_glob '/*/2-phenotyping/predictions////*.zarr' \ + --workspace_dir /hpc/mydata/eduardo.hirata/repos/viscy \ + -resume ``` -The directory name (e.g. `DynaCLR-2D-MIP-BagOfChannels`) is the -**feature_space** identifier — pipelines from one model's registry are -*not* applicable to a different model's embeddings (different dim, different -distribution). The model name follows the training-config-stem convention -(see `evaluation_matrix.md` §7). +Add `-profile local` for a local smoke test. Keep `-resume` for recoverable and +incremental runs. -### Publishing (writer) +## Evaluation config -A Wave-1 leaf (training run) sets `linear_classifiers.publish_dir`: +Create model-specific leaves under +[`applications/dynaclr/configs/evaluation/`](../../configs/evaluation/) and +compose shared settings from `recipes/`. ```yaml -linear_classifiers: - publish_dir: /hpc/projects/organelle_phenotyping/models/linear_classifiers/DynaCLR-2D-MIP-BagOfChannels/ - # ... annotations, tasks, ... +base: + - ../recipes/predict.yml + - ../recipes/reduce.yml + - ../recipes/plot_infectomics.yml + - ../recipes/infectomics-annotated.yml + +training_config: /path/to/resolved-training-config.yaml +ckpt_path: /path/to/checkpoint.ckpt +cell_index_path: /path/to/cell-index.parquet +output_dir: /path/to/evaluation-output + +steps: + - predict + - split + - reduce_dimensionality + - reduce_combined + - smoothness + - mmd + - linear_classifiers + - append_annotations + - append_predictions + - plot + - plot_combined ``` -`run-linear-classifiers` writes pipelines to a temp staging directory, -atomically renames to `vN/` (next available version), then atomically -swaps the `latest` symlink. Crash-safe: a partial bundle never appears as -`vN/`. +Only list the steps required by the run. For frozen embeddings, `predict` and +`split` are not executed by `eval_from_embeddings`, even if inherited from a +shared recipe. -### Fetching (reader) +Generate and inspect the resolved step configs without launching Nextflow: -A Wave-2 leaf (evaluation on a different dataset) sets -`append_predictions.pipelines_dir`: - -```yaml -append_predictions: - pipelines_dir: /hpc/projects/organelle_phenotyping/models/linear_classifiers/DynaCLR-2D-MIP-BagOfChannels/latest +```sh +uv run dynaclr prepare-eval-configs \ + -c applications/dynaclr/configs/evaluation/.yaml ``` -`append-predictions` resolves the symlink **once** at startup and uses the -resolved `vN/` for the rest of the run, so a publish during the run does -not affect output. The resolved path's parent name (`DynaCLR-2D-MIP-BagOfChannels`) -becomes `feature_space` in the manifest log. - -### Manifest format - -```json -{ - "trained_at": "2026-04-24T15:33:21+00:00", - "pipelines": [ - {"task": "infection_state", "marker_filter": "G3BP1", "path": "infection_state_G3BP1.joblib"}, - {"task": "infection_state", "marker_filter": "SEC61B", "path": "infection_state_SEC61B.joblib"} - ] -} -``` +Generated YAMLs and the copied input config are written under +`/configs/`. -Lineage (model name + version) lives in the directory structure, not the -manifest. Reproducibility comes from pinning a specific `vN` (instead of -`latest`) in paper-rerun scripts. +## MMD block -### Pinning vs. latest +Each item under `mmd` becomes one per-experiment analysis and, when +`combined_mode: true`, one cross-experiment analysis. ```yaml -# active development — picks up the latest published bundle -pipelines_dir: /hpc/.../linear_classifiers/DynaCLR-2D-MIP-BagOfChannels/latest - -# paper rerun — frozen at submission time -pipelines_dir: /hpc/.../linear_classifiers/DynaCLR-2D-MIP-BagOfChannels/v2 -``` - -## Nextflow DAG (process dependency graph) - -``` -checkpoint.ckpt ──────────────────────────────────────────────────────────────┐ - │ │ - ▼ ▼ -PREPARE_CONFIGS EXPORT_ONNX (optional) - │ │ - ▼ ▼ -PREDICT (GPU) model.onnx + CTC datasets - │ │ - ▼ ▼ -SPLIT (CPU light) TRACKING_ACCURACY (CPU) - │ → results.csv - ├─[scatter]─► REDUCE ─[gather]─► REDUCE_COMBINED ─┐ - │ │ - ├─► APPEND_ANNOTATIONS ───────────────────────────►├─[scatter]─► PLOT (only if "plot" in steps) - │ │ [gather] ─► PLOT_COMBINED (only if "plot_combined" in steps) - ├─► LINEAR_CLASSIFIERS ─► APPEND_PREDICTIONS ─────►┘ - │ - ├─[scatter]─► SMOOTHNESS ─[gather]─► SMOOTHNESS_GATHER - ├─[scatter per (exp,block)]─► MMD ─[gather]─► MMD_PLOT_HEATMAP - └─[gather per block]─► MMD_COMBINED -``` - -Key: **scatter** = one SLURM job per experiment (parallel). **gather** = waits for all scatter jobs. - -`TRACKING_ACCURACY` is independent of the embedding pipeline — it reads directly from an ONNX -model and CTC-format data. Run it manually or as a separate Nextflow job alongside the main DAG. - -`PLOT` (per-experiment fan-out) and `PLOT_COMBINED` (single combined figure) are -**independently togglable** via `steps:`. List `plot` for per-experiment scatter only, -`plot_combined` for the joint figure only, both for both, or neither for a metrics-only -run. `APPEND_ANNOTATIONS` and `APPEND_PREDICTIONS` emit a `'skip'` signal when not -present in `steps`, so plotting always proceeds once `REDUCE_COMBINED` finishes — -whichever plotting steps are listed. - -## CTC Tracking Accuracy Benchmark - -Standalone benchmark that evaluates whether DynaCLR embeddings improve cell tracking -accuracy on [Cell Tracking Challenge](https://celltrackingchallenge.net/) datasets. -**Not part of the Nextflow embedding pipeline** — run independently after exporting an ONNX model. - -### Approach - -``` -CTC segmentation masks + raw images - │ - ▼ -tracksdata graph (RegionPropsNodes + DistanceEdges) - │ - ├── baseline: IoU edge weights (no model) - │ - └── DynaCLR: ONNX inference on cell crops - → dynaclr_similarity × spatial_dist_weight as ILP edge cost - │ - ▼ -ILPSolver → tracked graph - │ - ▼ -evaluate_ctc_metrics vs. ground truth - │ - ▼ -results.csv (model × dataset × sequence × CTC metrics) +mmd: + - name: perturbation + group_by: perturbation + comparisons: + - cond_a: control + cond_b: perturbed + label: control_vs_perturbed + temporal_bin_size: 4.0 + combined_temporal_bin_size: null + combined_mode: true + embedding_key: null + representation: + normalization: control_mad + control_values: [control] + smooth_sigma_timepoints: 2.0 + mad_floor_quantile: 0.05 + pca_variance: 0.80 + mmd: + n_permutations: 1000 + max_cells: 5000 ``` -### Usage - -```bash -dynaclr evaluate-tracking-accuracy -c tracking_accuracy_config.yaml -``` +For per-experiment and pooled biological comparisons, the representation +defaults to a Gaussian-smoothed, time-matched control median; one fixed, +normal-consistent control MAD per embedding dimension; and a pooled, +marker-specific unwhitened PCA retaining at least 80% explained variance. The +CLI records the selected component count separately for every marker under +`/representation/`. Inputs must contain `experiment`, `marker`, +`perturbation`, and `hours_post_perturbation` metadata. -### Config format +To reproduce the historical raw-embedding behavior, set: ```yaml -models: - - path: /hpc/projects/.../model_ckpt146.onnx - label: DynaCLR-classical - - path: /hpc/projects/.../model_ckpt185.onnx - label: DynaCLR-timeaware - - path: null # baseline: IoU + spatial distance only - label: baseline-iou - -datasets: - - path: /hpc/reference/group.royer/CTC/training/BF-C2DL-HSC - sequences: ["01", "02"] - - path: /hpc/reference/group.royer/CTC/training/Fluo-C2DL-Huh7 - sequences: ["01", "02"] - -crop_shape: [64, 64] # must match the model's training resolution -distance_threshold: 325.0 # spatial candidate edge threshold (pixels) -n_neighbors: 10 -delta_t: 5 # max frame gap for candidate edges -batch_size: 128 -output_dir: /path/to/tracking_accuracy_results +representation: + normalization: none + pca_variance: null ``` -### Output - -**`results.csv`** — one row per (model, dataset, sequence): - -| Column | Description | -|--------|-------------| -| `model` | Model label | -| `dataset` | CTC dataset name | -| `sequence` | Sequence number (01, 02) | -| `LNK` | CTC Linking metric | -| `TRA` | Tracking metric | -| `DET` | Detection metric | -| `CHOTA` | Cell-specific HOTA | -| `HOTA` | Higher Order Tracking Accuracy | -| `MOTA` | Multiple Object Tracking Accuracy | -| `IDF1` | ID F1 score | -| `BIO(0)` | Biological metric | -| `OP_CLB(0)` | Combined linking+bio score | - -Prints a grouped summary (mean over sequences) at the end. +`--combined` and `--over-time` do not apply this self-normalization because +their purpose is to measure cross-experiment batch effects in the supplied +coordinate system. -### Prerequisites +For standalone runs, use a generated MMD YAML: -1. Export the model to ONNX: - ```bash - viscy export -c export_onnx.yml - ``` -2. CTC datasets must have `{seq}_ERR_SEG/`, `{seq}/`, and `{seq}_GT/TRA/` subdirectories. -3. Install eval dependencies: `uv sync --all-packages --extra eval` - -## Pseudotime alignment benchmark - -Standalone benchmark that quantifies how well DTW on DynaCLR embeddings recovers per-cell biological -event onsets (e.g. infection onset from the NS3 sensor channel). **Not part of the Nextflow embedding -pipeline** — runs after `linear_classifiers` + `append_predictions` so it can use either human -`infection_state` or model-predicted `predicted_infection_state` as ground truth. - -Full pipeline (template build + DTW alignment) lives in `applications/dynaclr/scripts/pseudotime/` — -see [`pseudotime.md`](pseudotime.md). The scoring step described below consumes the Stage 2a -alignment parquet that pipeline produces. - -### Approach - -``` -embeddings/{experiment}.zarr (with ground-truth obs) - │ - ▼ -pseudotime/2-align_cells/alignments/ - {template}_{flavor}_on_{query_set}.parquet - │ (per-frame: pseudotime, estimated_t_rel_minutes, alignment_region) - ▼ -score_alignment.py --method {dtw | no_align} - │ - │ Per cell: onset_error_minutes = estimated_t_rel_minutes at the first - │ aligned positive frame preceded by a negative frame - │ Population: AUROC + F1@0 over (cell, frame) pairs in aligned region - ▼ -scoring/{template}_{flavor}_on_{query_set}_{method}_per_cell.parquet -scoring/{template}_{flavor}_on_{query_set}_{method}_summary.md -scoring/results.csv (one row per run, accumulates across runs) - │ - ▼ -compare_methods.py - │ - ▼ -scoring/compare_methods.md (paper table) -scoring/compare_methods.png (4-panel bar chart: med|Δt|, IQR, AUROC, F1@0) +```sh +uv run dynaclr compute-mmd -c mmd.yaml +uv run dynaclr compute-mmd --combined -c mmd_cross_exp.yaml +uv run python applications/dynaclr/scripts/evaluation/write_normalized_pca80.py \ + -c applications/dynaclr/configs/evaluation/recipes/witness_gmm_pooled_joint_pca80.yaml +uv run dynaclr witness-gmm-labels \ + -c applications/dynaclr/configs/evaluation/recipes/witness_gmm_pooled_joint_pca80.yaml ``` -### Method comparison philosophy - -The benchmark compares **alignment methods on the same DynaCLR embedding**, not different -embeddings. Two methods are bundled: - -- **`dtw`** — uses `estimated_t_rel_minutes` from the Stage 2a parquet (DBA template + subsequence - DTW on the embedding trajectory). -- **`no_align`** — substitutes `estimated_t_rel_minutes` with each cell's frame index relative to - its track midpoint, no learning. The lower bound DTW must beat. +The only maintained pooled biological-state example is +`applications/dynaclr/configs/evaluation/recipes/witness_gmm_pooled_joint_pca80.yaml`. Its `pooled_representation` section drives PCA80 export, and its +`witness_gmm_labels` section drives pooled MMD permutation testing plus the +joint/tied GMM teacher. -This is the right comparison for the paper claim *"DTW on DynaCLR embeddings recovers infection -onset"* — the embedding is held fixed, so the metric attributes the gain to the alignment step, -not to representation quality. +## Linear classifiers -### Usage +Classifier tasks use annotation files keyed to cells. Use a group-aware split +when tracks contribute multiple frames. -```bash -cd applications/dynaclr/scripts/pseudotime/2-align_cells -# Score one alignment parquet (DTW) -uv run python score_alignment.py \ - --datasets ../../../configs/pseudotime/datasets.yaml \ - --config ../../../configs/pseudotime/align_cells.yaml \ - --template infection_nondividing_sensor --flavor raw \ - --query-set sensor_all_07_24 \ - --truth-column infection_state --truth-positive infected \ - --method dtw - -# No-align baseline on the same parquet -uv run python score_alignment.py ... --method no_align - -# Render comparison artifacts -uv run python compare_methods.py +```yaml +linear_classifiers: + label_source: annotations + annotations: + - experiment: + path: /path/to/annotations.csv + tasks: + - task: infection_state + marker_filters: [Phase3D] + use_scaling: true + use_pca: false + split_train_data: 0.8 + split_groups_by: [experiment, fov_name, track_id] + random_seed: 42 ``` -### Output schema (`scoring/results.csv`) - -| Column | Description | -|---|---| -| `template`, `flavor`, `query_set` | identifies the alignment parquet | -| `method` | `dtw` or `no_align` | -| `truth_column`, `truth_positive` | which obs column was the ground truth | -| `n_cells_scored` | cells with a usable negative→positive transition in the aligned region | -| `median_abs_onset_error_minutes` | robust center of \|Δt_onset\|; **primary metric** | -| `iqr_abs_onset_error_minutes` | spread of \|Δt_onset\| | -| `median_signed_onset_error_minutes` | systematic bias (≈0 if unbiased) | -| `auroc` | over aligned (cell, frame) pairs ranked by warped time; chance = 0.5 | -| `f1_at_zero` | F1 at the threshold `estimated_t_rel_minutes ≥ 0` | -| `n_pairs` | aligned (cell, frame) pairs used by AUROC/F1 | -| `timestamp_utc` | run timestamp | - -### Prerequisites - -1. A built template under `1-build_template/templates/` (see `pseudotime.md`). -2. A Stage 2a alignment parquet under `2-align_cells/alignments/`. -3. The query embedding zarrs must carry the requested `--truth-column` (human `infection_state` - on 07_22/07_24; `predicted_infection_state` on 08_26/01_28 — populated by - `APPEND_PREDICTIONS`). - -## Cross-model comparison +The workflow trains one pipeline per task-marker pair. Witness-GMM labels use +the same annotation path; see +[witness_gmm_classifiers.md](witness_gmm_classifiers.md). -After running evals for multiple models, compare results with: +To publish a fitted bundle for later runs: -```bash -python applications/dynaclr/scripts/evaluation/compare_evals.py -c eval_registry.yml +```yaml +linear_classifiers: + publish_dir: /path/to/linear_classifiers/ ``` -Registry format: +To apply a published bundle without retraining: ```yaml -models: - - name: DynaCLR-v3 - eval_dir: /path/to/eval_v3 # = output_dir of that model's eval run - - name: DINOv3-MLP - eval_dir: /path/to/eval_dino -output_dir: /path/to/comparison_output -fdr_threshold: 0.05 +append_predictions: + pipelines_dir: /path/to/linear_classifiers//v2 ``` -`eval_dir` is the **`output_dir`** declared in each leaf eval YAML (where -`smoothness/`, `linear_classifiers/`, `mmd/` land). One row per model × dataset -pair you want to compare side-by-side. Auto-discovers results and produces -overlaid plots and summary CSVs for smoothness, linear classifiers, and MMD. - -The shipping registry at `applications/dynaclr/configs/evaluation/eval_registry.yaml` -is checked into git and updated as new (model × dataset) eval runs complete. - -## Key commands - -| Step | Command | Input | Output | -|------|---------|-------|--------| -| Config gen | `dynaclr prepare-eval-configs -c eval.yaml` | eval config | configs/ + manifest JSON | -| Predict | `viscy predict -c predict.yml` | checkpoint + parquet | embeddings/embeddings.zarr | -| Split | `dynaclr split-embeddings --input ... --output-dir ...` | combined zarr | per-experiment zarrs + `configs/viewer.yaml` | -| Dim reduction | `dynaclr reduce-dimensionality -c reduce.yaml` | {experiment}.zarr | zarr with X_pca | -| Combined reduction | `dynaclr combined-dim-reduction -c reduce_combined.yaml` | all {experiment}.zarr | zarrs with X_pca_combined/X_phate_combined | -| Plots (per-exp) | `dynaclr plot-embeddings -c plot.yaml` | {experiment}.zarr | plots/{experiment}/*.pdf | -| Plots (combined) | `dynaclr plot-embeddings -c plot_combined.yaml` | all {experiment}.zarr | plots/combined/*.pdf | -| Smoothness | `dynaclr evaluate-smoothness -c smoothness.yaml` | {experiment}.zarr | per_marker_smoothness.csv, smoothness_stats.csv | -| MMD (per-exp) | `dynaclr compute-mmd -c {block}.yaml` | {experiment}.zarr | mmd/{block}/mmd_results.csv | -| MMD (combined) | `dynaclr compute-mmd --combined -c {block}_cross_exp.yaml` | all {experiment}.zarr | mmd/{block}_cross_exp/combined_mmd_results.csv | -| MMD (pooled) | `dynaclr compute-mmd --pooled -c pooled.yaml` | all {experiment}.zarr | mmd_results.csv | -| Linear probe | `dynaclr run-linear-classifiers -c clf.yaml` | per-experiment zarrs + annotations | metrics_summary.csv, {task}_summary.pdf, pipelines/ | -| Append annotations | `dynaclr append-annotations -c append_annotations.yaml` | per-experiment zarrs + annotation CSVs | zarrs with obs annotation columns | -| Append predictions | `dynaclr append-predictions -c append_predictions.yaml` | per-experiment zarrs + pipelines/ | zarrs with predicted_{task} in obs/obsm/uns | -| Compare models | `python compare_evals.py -c eval_registry.yml` | multiple eval dirs | comparison CSVs + plots | -| CTC tracking | `dynaclr evaluate-tracking-accuracy -c tracking_accuracy.yaml` | ONNX model + CTC datasets | tracking_accuracy/results.csv | - -## Placeholder pattern - -Template YAMLs (`reduce.yaml`, `smoothness.yaml`, `{block}.yaml`, `plot.yaml`) contain `__ZARR_PATH__` -as a placeholder for `input_path`. `plot.yaml` also contains `__PLOT_DIR__`. Nextflow process -scripts substitute these inline with Python one-liners before calling the CLI command: - -```python -import yaml -with open('reduce.yaml') as f: - cfg = yaml.safe_load(f) -cfg['input_path'] = '/path/to/experiment.zarr' -with open('reduce_patched.yaml', 'w') as f: - yaml.dump(cfg, f, default_flow_style=False, sort_keys=False) +Pin a version directory for reproducible runs. Use `latest` only for active +iteration. A classifier bundle must be applied to the same embedding feature +space in which it was trained. + +## Outputs + +Depending on `steps`, the workflow writes: + +```text +/ +├── configs/ +├── embeddings/.zarr +├── plots/ +├── smoothness/ +├── mmd// +├── mmd/_cross_exp/ +└── linear_classifiers/ + ├── metrics_summary.csv + ├── _summary.pdf + └── pipelines/ + ├── manifest.json + └── _.joblib ``` -For `reduce_combined.yaml`, `plot_combined.yaml`, and `{block}_cross_exp.yaml`, Nextflow collects -all zarr paths and writes the `input_paths` list directly. +Enrichment steps update each embedding zarr with: -## Notes +- annotation and predicted-label columns in `.obs`; +- prediction probabilities in `.obsm`; +- class order and classifier lineage in `.uns`; +- per-experiment and combined reductions in `.obsm`. -- `MultiExperimentDataModule` supports `stage="predict"` since the eval orchestrator was added. - It uses the full cell index (no train/val split), applies only normalizations + z-reduction (no augmentations). -- `BatchedChannelWiseZReductiond` is architecturally required for 2D models even at inference time - (converts 3D z-stack → 2D MIP/center-slice). The orchestrator moves it from `augmentations` - to `normalizations` in the generated predict config. -- Dimensionality reductions (PCA, PHATE) are **not** computed inline during predict. - They run as separate CPU steps after splitting, keeping predict fast. -- The `combined-dim-reduction` step fits reductions on all experiments jointly and writes - `X_pca_combined` / `X_phate_combined` back to each per-experiment zarr. -- PHATE is not computed per-experiment by default (`reduce_dimensionality.phate: null`). Run it only jointly via `reduce_combined`. -- `configs/viewer.yaml` is generated after split and can be passed directly to `dynaclr combined-dim-reduction`. -- MMD reads `.X` (raw backbone embeddings) by default. It can also run on `X_pca` or `X_pca_combined` via `embedding_key`. -- Embeddings obs carries `organelle`, `well`, and `microscope` in addition to `experiment`, `marker`, `perturbation`, `hours_post_perturbation`. +## Direct step commands -## MMD config format +Nextflow normally patches generated configs and calls these commands: -Use `configs/evaluation/recipes/mmd_defaults.yml` as a base to avoid repeating MMD algorithm parameters: +| Step | Command | +| --- | --- | +| Split combined embeddings | `dynaclr split-embeddings --input --output-dir ` | +| Per-store reduction | `dynaclr reduce-dimensionality -c ` | +| Combined reduction | `dynaclr combined-dim-reduction -c ` | +| Plot embeddings | `dynaclr plot-embeddings -c ` | +| Smoothness | `dynaclr evaluate-smoothness -c ` | +| MMD | `dynaclr compute-mmd -c ` | +| Linear probes | `dynaclr run-linear-classifiers -c ` | +| Append labels | `dynaclr append-annotations -c ` | +| Append predictions | `dynaclr append-predictions -c ` | -```yaml -# Per-experiment (template — __ZARR_PATH__ substituted at runtime) -base: recipes/mmd_defaults.yml -input_path: __ZARR_PATH__ -output_dir: /path/to/evaluation/mmd/perturbation/ -group_by: perturbation -comparisons: - - cond_a: uninfected - cond_b: ZIKV - label: "uninfected vs ZIKV" -embedding_key: null # null = raw .X; or "X_pca", "X_pca_combined" -temporal_bin_size: 4.0 # uniform bin width in hours (null = aggregate) -# temporal_bins: [0, 6, 12, 24] # alternative: explicit bin edges (mutually exclusive) -mmd: - balance_samples: true # subsample larger group to match smaller - share_bandwidth_from: "uninfected vs uninfected" # reuse bandwidth from baseline comparison -map_settings: - enabled: true # compute mAP via copairs alongside MMD - -# Cross-experiment ({block}_cross_exp.yaml — input_paths substituted at runtime) -# No comparisons — conditions auto-discovered from data intersection. -base: recipes/mmd_defaults.yml -input_paths: [__ZARR_PATH__] -output_dir: /path/to/evaluation/mmd/perturbation_cross_exp/ -group_by: perturbation -temporal_bin_size: 4.0 - -# Pooled (standalone CLI only — not generated by orchestrator) -base: recipes/mmd_defaults.yml -input_paths: - - /path/to/exp_A.zarr - - /path/to/exp_B.zarr -output_dir: /path/to/evaluation/mmd/pooled/ -comparisons: - - cond_a: uninfected - cond_b: ZIKV - label: "uninfected vs ZIKV" -condition_aliases: - uninfected: [uninfected, uninfected1, uninfected2] # map variants to canonical name -``` +Use the generated configs rather than recreating their placeholder substitution +manually. + +## Validation + +- Confirm every requested experiment appears under `embeddings/`. +- Confirm all requested `steps` produced non-empty outputs. +- Inspect failed or skipped classifier tasks for missing labels or marker + mismatches. +- Check MMD group counts before interpreting scores. +- Rerun the same Nextflow command with `-resume` after correcting a failed step. -## MMD output columns - -### Per-experiment and pooled (`mmd_results.csv`) - -| Column | Description | -|--------|-------------| -| `experiment` | Experiment name (absent in pooled output) | -| `marker` | Organelle marker (e.g., "TOMM20", "G3BP1") | -| `cond_a` | Reference/control condition | -| `cond_b` | Treatment condition | -| `label` | Human-readable comparison label | -| `hours_bin_start` | Start of temporal bin (NaN if no binning) | -| `hours_bin_end` | End of temporal bin (NaN if no binning) | -| `n_a` | Cells from `cond_a` used after subsampling | -| `n_b` | Cells from `cond_b` used after subsampling | -| `mmd2` | Unbiased MMD² estimate | -| `p_value` | Permutation test p-value (Phipson & Smyth smoothed) | -| `q_value` | BH-corrected FDR (pooled mode only) | -| `bandwidth` | Gaussian RBF bandwidth | -| `effect_size` | mmd2 / bandwidth (scale-free) | -| `activity_zscore` | (mmd2 − null_mean) / null_std — normalized against permutation null | -| `map_value` | Mean Average Precision (NaN if map_settings.enabled=false) | -| `map_p_value` | mAP permutation p-value (NaN if map_settings.enabled=false) | -| `embedding_key` | Embedding used ("X" or obsm key) | - -### Cross-experiment (`combined_mmd_results.csv`) - -| Column | Description | -|--------|-------------| -| `marker` | Organelle marker | -| `exp_a` | First experiment in the pair | -| `exp_b` | Second experiment in the pair | -| `condition` | Condition value matched across experiments | -| `hours_bin_start` | Start of temporal bin (NaN if no binning) | -| `hours_bin_end` | End of temporal bin (NaN if no binning) | -| `n_a` | Cells from `exp_a` used | -| `n_b` | Cells from `exp_b` used | -| `mmd2` | Unbiased MMD² estimate | -| `p_value` | Permutation test p-value | -| `bandwidth` | Gaussian RBF bandwidth | -| `effect_size` | mmd2 / bandwidth | -| `activity_zscore` | (mmd2 − null_mean) / null_std | -| `embedding_key` | Embedding used | - -## Linear classifiers output columns - -| Column | Description | -|--------|-------------| -| `task` | Classification task (e.g., `infection_state`) | -| `marker_filter` | Marker used to filter cells (one row per marker per task) | -| `n_samples` | Total annotated cells used | -| `val_accuracy` | Validation accuracy | -| `val_weighted_f1` | Validation weighted F1 | -| `val_auroc` | Validation AUROC (OvR macro for multiclass) | -| `train_*` | Training set counterparts of the above | -| `val_{class}_f1` | Per-class F1 on validation set | +For multi-row submission, see +[evaluation_matrix.md](evaluation_matrix.md). For the workflow implementation, +see [the Nextflow README](../../nextflow/README.md). diff --git a/applications/dynaclr/docs/DAGs/evaluation_matrix.md b/applications/dynaclr/docs/DAGs/evaluation_matrix.md new file mode 100644 index 000000000..2aedae57e --- /dev/null +++ b/applications/dynaclr/docs/DAGs/evaluation_matrix.md @@ -0,0 +1,144 @@ +# Run a prediction and evaluation matrix + +`dynaclr run-matrix` submits prediction and evaluation jobs for multiple +collections, models, or checkpoints. Models must already be trained; the +matrix runner does not submit training jobs. + +```mermaid +flowchart TD + A["Matrix YAML"] --> B["Resolve model identity and checkpoints"] + B --> C["AI-ready preflight"] + C --> D1["row 1: predict"] + C --> D2["row 2: predict"] + C --> DN["row N: predict"] + D1 --> E1["row 1: eval"] + D2 --> E2["row 2: eval"] + DN --> EN["row N: eval"] +``` + +Rows run in parallel. Within each row and checkpoint, evaluation is submitted +with an `afterok` dependency on prediction. + +## Matrix config + +Start from +[`applications/dynaclr/configs/matrix/example.yml`](../../configs/matrix/example.yml). + +```yaml +defaults: + train_sbatch: applications/dynaclr/configs/training/DynaCLR-2D/.sh + ckpt_name: epoch105-step84800 + checkpoint: /path/to/epoch=105-step=84800.ckpt + eval_config: applications/dynaclr/configs/evaluation/.yaml + datasets_root: /hpc/projects/intracellular_dashboard/organelle_dynamics + +models: + - collection: applications/dynaclr/configs/collections/.yml + markers: [SEC61B, Phase3D] + - collection: applications/dynaclr/configs/collections/.yml + markers: [G3BP1, Phase3D] +``` + +`train_sbatch` is used only to parse the model `family`, `run`, and training +config identity from its exported variables. It is not submitted. Set +`checkpoint` explicitly when the checkpoint is nested below a logger run +directory. + +To sweep checkpoints in one row: + +```yaml +models: + - train_sbatch: applications/dynaclr/configs/training/DynaCLR-2D/.sh + collection: applications/dynaclr/configs/collections/.yml + ckpt_names: [epoch80-step64000, epoch105-step84800] + checkpoints: + - /path/to/epoch=80-step=64000.ckpt + - /path/to/epoch=105-step=84800.ckpt +``` + +`ckpt_names` and `checkpoints` must have the same length. Each pair creates an +independent `predict → eval` chain. + +Prediction parameters not represented in the matrix schema are controlled by +[`applications/dynaclr/tools/predict.sbatch`](../../tools/predict.sbatch). +Legacy `predict_flags` blocks in existing matrix examples are not currently +consumed by `run-matrix`; do not rely on them. + +## Run + +Always inspect the submission plan first: + +```sh +uv run dynaclr run-matrix \ + -c applications/dynaclr/configs/matrix/.yml \ + --dry-run +``` + +Submit prediction and evaluation: + +```sh +uv run dynaclr run-matrix \ + -c applications/dynaclr/configs/matrix/.yml \ + --stages predict,eval +``` + +Run only one stage when needed: + +```sh +uv run dynaclr run-matrix -c .yml --stages predict +uv run dynaclr run-matrix -c .yml --stages eval +``` + +On real prediction submissions, the runner verifies that every collection has +focus and normalization metadata before queueing jobs. `--skip-preflight` +bypasses this check and should only be used when readiness was verified +separately. + +## Adding datasets to one model + +Rows with the same model, run, and checkpoint write into the same embedding +tree. Their evaluation jobs do not wait for prediction jobs from sibling rows. +For this case, predict the full matrix first: + +```sh +uv run dynaclr run-matrix \ + -c applications/dynaclr/configs/matrix/.yml \ + --stages predict +``` + +After all prediction jobs finish, launch one cohort evaluation: + +```sh +uv run dynaclr eval \ + --eval-config applications/dynaclr/configs/evaluation/.yaml \ + --model-family \ + --run \ + --ckpt-name +``` + +This avoids multiple evaluations writing to the same `output_dir`. + +## Outputs + +Prediction uses the standard provenance tree: + +```text +//2-phenotyping/predictions/ + ///.zarr +``` + +Evaluation writes to the `output_dir` declared by each evaluation config. See +[evaluation.md](evaluation.md) for the output layout. + +## Validation + +- Inspect `--dry-run` output for the resolved collection, checkpoint, model + family, run, marker list, and evaluation config. +- Confirm every prediction job completed before launching a gathered cohort + evaluation. +- Give concurrent evaluation rows distinct `output_dir` values. +- Use pinned checkpoints and classifier bundle versions for reproducible + matrices. + +The maintained multi-dataset example is +[`organelle_remodeling.yml`](../../configs/matrix/organelle_remodeling.yml). diff --git a/applications/dynaclr/docs/DAGs/inference_triplet.md b/applications/dynaclr/docs/DAGs/inference_triplet.md new file mode 100644 index 000000000..59ed9e7b2 --- /dev/null +++ b/applications/dynaclr/docs/DAGs/inference_triplet.md @@ -0,0 +1,140 @@ +# Predict per-marker embeddings + +`dynaclr predict-triplet` runs a trained checkpoint directly against image and +tracking zarrs defined by a collection. It writes one AnnData zarr per +experiment and marker. + +```mermaid +flowchart LR + A["Collection YAML"] --> D["predict-triplet"] + B["Checkpoint"] --> D + C["Image + tracking zarrs"] --> D + D --> E["Per-marker embedding zarrs"] + E --> F["Evaluation and analysis"] +``` + +Use this workflow for collection-driven inference. The parquet-first prediction +path used by the full evaluation workflow is documented in +[evaluation.md](evaluation.md). + +## Inputs + +- An AI-ready image zarr with normalization metadata. +- A tracking zarr with `track_id`, `t`, `y`, and `x`. +- A checkpoint compatible with the configured model family. +- A collection YAML under + [`applications/dynaclr/configs/collections/`](../../configs/collections/). + +For each experiment, the collection must provide the data paths and channel to +marker mapping. Optional `wells` restricts a channel entry to part of a plate. + +```yaml +name: +experiments: + - name: + data_path: /path/to/dataset.zarr + tracks_path: /path/to/tracking.zarr + channels: + - name: Phase3D + marker: Phase3D + - name: raw GFP EX488 EM525-45 + marker: SEC61B + wells: [A/2, B/2] + perturbation_wells: + control: [A/1, B/1] + perturbed: [A/2, B/2] + interval_minutes: 30.0 + pixel_size_xy_um: 0.1494 + pixel_size_z_um: 0.174 +``` + +## Run + +```sh +uv run dynaclr predict-triplet \ + -c applications/dynaclr/configs/collections/.yml \ + --checkpoint /path/to/checkpoint.ckpt \ + --model-family \ + --run \ + --ckpt-name \ + --datasets-root /hpc/projects/intracellular_dashboard/organelle_dynamics \ + --z-window 10 \ + --focus-channel Phase3D \ + --z-focus-offset 0.3 \ + --z-reduction mip \ + --reference-pixel-size 0.1494 \ + --reference-pixel-size-z-um 0.174 \ + --yx-patch-size 160 160 \ + --batch-size 32 \ + --num-workers 0 +``` + +Important options: + +| Option | Purpose | +| --- | --- | +| `--markers SEC61B,TOMM20` | Predict only the listed markers. | +| `--no-labelfree` | Skip phase and brightfield channels. | +| `--z-window` | Focus-centered window width in reference-grid slices. | +| `--z-reduction mip|center` | Collapse the selected z window for a 2D model. | +| `--reference-pixel-size` | Rescale crops to the model's training pixel size. | +| `--reference-pixel-size-z-um` | Convert `--z-window` to the native slice count covering the same physical depth. | +| `--no-enrich-obs` | Do not append collection metadata to output `obs`. | + +For example, `--z-window 10 --reference-pixel-size-z-um 0.174` +defines a 1.74 µm slab. A dataset sampled at 0.288 µm/slice reads 6 native +slices rather than 10, then resizes those 6 slices back to the 10-slice +reference grid with nearest-neighbor interpolation before augmentation and +MIP. The native count is rounded to the nearest whole slice. Triplet and +MultiExperiment use this same sequence. +Physical Z normalization requires the focus-centered `--z-window` form; +absolute `--z-range` indices retain their literal native-slice meaning. + +Keep `--num-workers 0`; multiprocessing can deadlock while reading zarr during +prediction. Prediction is deterministic and does not apply training +augmentations. + +## Outputs + +```text +//2-phenotyping/predictions/ +└── / + └── / + └── / + ├── Phase3D.zarr + ├── SEC61B.zarr + └── .zarr +``` + +Each output is an AnnData store: + +- `.X`: selected embedding representation; +- `.obs`: cell identifiers, tracking fields, and collection metadata; +- `.uns`: model, run, checkpoint, collection, marker, and channel provenance. + +The output path is derived from the collection and provenance flags. Use the +same `model-family`, `run`, and `checkpoint-name` in downstream launchers. + +## Batch prediction + +For multiple collections or checkpoints, use the matrix runner instead of +writing per-dataset shell loops: + +```sh +uv run dynaclr run-matrix \ + -c applications/dynaclr/configs/matrix/.yml \ + --stages predict \ + --dry-run +``` + +Inspect the dry run, then rerun without `--dry-run`. See +[evaluation_matrix.md](evaluation_matrix.md). + +## Validation + +- Confirm one zarr was written for every requested experiment-marker pair. +- Run `uv run dynaclr info .zarr` and verify row count, feature count, + `obs` metadata, and provenance. +- Treat embeddings from different checkpoints as different feature spaces. + +Continue with [evaluation.md](evaluation.md) to evaluate the frozen outputs. diff --git a/applications/dynaclr/docs/DAGs/lot_correction.md b/applications/dynaclr/docs/DAGs/lot_correction.md new file mode 100644 index 000000000..2098ba3f2 --- /dev/null +++ b/applications/dynaclr/docs/DAGs/lot_correction.md @@ -0,0 +1,133 @@ +# Correct embedding batches with LOT + +Fit one Linear Optimal Transport (LOT) map from pooled source and target +reference cells, then apply the same fitted map to every source-domain +embedding store. + +```mermaid +flowchart TD + S["Source reference zarrs"] --> F["fit-lot-correction"] + T["Target reference zarrs"] --> F + F --> P["lot_pipeline.pkl"] + P --> A1["apply to dataset A"] + P --> A2["apply to dataset B"] + P --> AN["apply to dataset N"] + A1 --> V["MMD pre/post validation"] + A2 --> V + AN --> V +``` + +Fit once per embedding feature space and anchor channel. Application jobs are +independent and may run in parallel. + +## Inputs + +- Source- and target-domain AnnData zarrs with embeddings in `.X`. +- A reference-population filter that is available in `.obs` for both domains. +- Matching feature dimensions and model/checkpoint provenance. + +The source and target sides may contain different numbers of cells. `ns_lot` +is a compute cap for each pooled side, not a requirement to balance counts. + +## Fit config + +```yaml +source: + - zarr: /path/to/source_rep1.zarr + filter: + column: fov_name + startswith: ["C/1/"] + - zarr: /path/to/source_rep2.zarr + filter: + column: fov_name + startswith: ["C/1/"] + +target: + - zarr: /path/to/target.zarr + filter: + column: perturbation + equals: control + +channel: Phase3D +n_pca: 50 +ns_lot: 3000 +random_seed: 42 +output_pipeline: /path/to/lot_pipeline.pkl +``` + +Each side is a list of zarr/filter entries. Entries on the same side are pooled. +Omit `filter` to use every cell in a store. Set `n_pca: null` to fit LOT in the +scaled input space and `ns_lot: null` to use all reference cells. + +## Fit + +```sh +uv run dynaclr fit-lot-correction -c fit_lot.yaml +``` + +The fitted artifact contains the scaler, optional PCA, LOT transform, channel, +sampling settings, and explained-variance metadata. + +## Apply + +Apply the same pipeline to each complete dataset, not only the reference subset: + +```sh +uv run dynaclr apply-lot-correction \ + --pipeline /path/to/lot_pipeline.pkl \ + --input /path/to/source_rep1.zarr \ + --output /path/to/corrected/source_rep1.zarr +``` + +Use `--overwrite` only when the output store may be replaced. Repeat the command +for every input that must share the corrected coordinate system. + +The corrected zarr contains: + +- corrected embeddings in `.X`; +- copied `.obs` and `.uns` metadata; +- correction provenance in `.uns["lot_correction"]`. + +Existing `.obsm`, `.varm`, `.obsp`, layers, and dimensionality reductions are +dropped because they belong to the uncorrected feature space. Recompute them +from corrected `.X`. + +## Validate pre/post MMD + +Create a paired over-time MMD config: + +```yaml +output_dir: /path/to/mmd_over_time +input_paths: + - /path/to/source_rep1.zarr + - /path/to/source_rep2.zarr +corrected_paths: + - /path/to/corrected/source_rep1.zarr + - /path/to/corrected/source_rep2.zarr +group_by: perturbation +temporal_bin_size: 6.0 +``` + +Run: + +```sh +uv run dynaclr compute-mmd --over-time -c mmd_over_time.yaml +``` + +Inputs and corrected outputs are paired through `obs["experiment"]`, not list +position. The command writes `over_time_mmd_results.csv` and per-marker pre/post +kinetics plots. + +## Validation rules + +- Use reference filters that select the same type of population on both sides. +- Apply one fitted pipeline to all source datasets that must remain comparable; + do not refit per dataset. +- Fit one map per anchor channel and record that channel in the config. +- Do not use a map across different model/checkpoint feature spaces. +- Confirm at least five pooled reference cells exist on each side. +- Confirm post-correction cross-domain MMD decreases for every channel to which + the map will be applied. + +Continue downstream with [evaluation.md](evaluation.md). Re-run dimensionality +reduction after correction. diff --git a/applications/dynaclr/docs/DAGs/pseudotime.md b/applications/dynaclr/docs/DAGs/pseudotime.md index 298437487..cee1a88bc 100644 --- a/applications/dynaclr/docs/DAGs/pseudotime.md +++ b/applications/dynaclr/docs/DAGs/pseudotime.md @@ -1,731 +1,236 @@ -# Pseudotime DAG +# Run the pseudotime workflow -This document describes how the pseudotime pipeline runs. For why it -runs this way — methodology decisions, claims, falsification protocol — -see the source-of-truth discussion document at -`/home/eduardo.hirata/repos/DynaCLR/.planning/dynaclr_dtw_discussion.md`. +The pseudotime scripts select candidate tracks, build an optional template, +produce three alignment variants, compute per-channel readouts, and compare the +results. -## 1. Goals - -We measure when SEC61 (ER), G3BP1 (stress granules), and quantitative -phase morphology change relative to per-cell NS3 sensor translocation -in single A549 cells, then compare three alignment tracks to see which -sharpens the population timing readouts. - -The pipeline produces three parallel sets of organelle-remodeling -plots, one per track, indexed on the same lineage-reconnected cohort -so the side-by-side comparison is direct. +```mermaid +flowchart TD + A["Dataset and candidate configs"] --> B["0. Select candidate cohorts"] + B --> C["1. Build template"] + B --> D1["2A. Annotation alignment"] + B --> D2["2B. Classifier alignment"] + C --> D3["2C. Embedding DTW alignment"] + D1 --> E["3. Per-channel readouts"] + D2 --> E + D3 --> E + E --> F["4. Compare alignment tracks"] +``` -| Track | Anchor | Alignment | Outputs in | -|---|---|---|---| -| **A-anno** | Human `infection_state` first-positive frame | Per-cell shift, real-time | `3-organelle-remodeling/A-anno/` | -| **A-LC** | Linear classifier `predicted_infection_state` first-positive frame | Per-cell shift, real-time | `3-organelle-remodeling/A-LC/` | -| **B** | NS3 embedding band on transition window | Hybrid DTW: warp transition only, propagated to organelle and phase | `3-organelle-remodeling/B/` | +All commands below run from the repository root. The scripts write into their +stage directories under `applications/dynaclr/scripts/pseudotime/`. -Path B is the methodological contribution. Paths A-anno and A-LC are -baselines. +## Config files -### What `t_rel = 0` means +| Config | Used by | Required content | +| --- | --- | --- | +| [`datasets.yaml`](../../configs/pseudotime/datasets.yaml) | every stage | dataset paths, frame intervals, embedding patterns | +| [`candidates.yaml`](../../configs/pseudotime/candidates.yaml) | candidate selection, A-anno, A-LC, readouts | candidate sets and cohort rules | +| [`build_template.yaml`](../../configs/pseudotime/build_template.yaml) | template build and self-check | template names, input candidate set, channel, preprocessing | +| [`align_cells.yaml`](../../configs/pseudotime/align_cells.yaml) | DTW alignment and scoring | query sets, channel, dataset filters, time requirements | +| [`compare_tracks.yaml`](../../configs/pseudotime/compare_tracks.yaml) | final comparisons | candidate set, tracks, channels, output aggregation | -`t_rel = 0` is when the NS3 protease sensor crosses our chosen anchor: -either the human `infection_state` first-positive frame (Path A-anno), -the LC threshold-crossing (Path A-LC), or the NS3 embedding's -half-rise band on the transition window (Path B). All three are -landmarks downstream of viral entry, polyprotein translation, and ER -remodeling — `t_rel = 0` is a fiducial clock, not the start of -infection. See discussion §1.1 and §2.1 for biological framing. +Keep dataset IDs, candidate-set names, template names, and query-set names +consistent across the configs. -## 2. Pipeline stages +## 0. Select candidate cohorts -``` -0-select_candidates → 1-build_template → 2-align_cells → 3-organelle-remodeling → 4-compare_tracks - (lineage-reconnect, (Path B only — (Path A-anno, (per-track per-organelle (side-by-side - cohort tagging, NS3 transition A-LC, B readouts: SEC61 comparison, - manual + auto) template) alignments) cosine-distance, warp-vs-no-warp, - G3BP1 oscillation, bimodality test, - phase distance) robustness) +```sh +uv run python applications/dynaclr/scripts/pseudotime/0-select_candidates/select_candidates.py \ + --datasets applications/dynaclr/configs/pseudotime/datasets.yaml \ + --config applications/dynaclr/configs/pseudotime/candidates.yaml \ + --candidate-set ``` -Stages 0 and 1 share data across tracks. Stages 2, 3, and 4 fork -per-track, then re-converge in Stage 4 for comparison. - -## 3. Directory layout +Outputs: -``` -applications/dynaclr/ -├── configs/pseudotime/ -│ ├── datasets.yaml # datasets + embedding patterns -│ ├── candidates.yaml # candidate sets, cohort tags, lineage rules -│ ├── build_template.yaml # Stage 1 (Path B template build) -│ ├── align_cells.yaml # Stage 2 (per-track query sets) -│ └── compare_tracks.yaml # Stage 4 (cross-track headlines) -├── docs/DAGs/pseudotime.md # this file -├── src/dynaclr/pseudotime/ # library -│ ├── dtw_alignment.py # template fit + warp solver -│ ├── io.py # template-zarr layout + provenance -│ ├── alignment.py # lineage reconnection, daughter handling -│ ├── signals.py # extract annotation / LC / embedding signals -│ ├── metrics.py # onset, t_50, peak, oscillation stats -│ └── plotting.py # response curves, heatmaps, comparisons -└── scripts/pseudotime/ - ├── 0-select_candidates/ - │ ├── select_candidates.py # auto path (from annotations) - │ ├── manual_candidates.py # manual path (hand-picked tracks) - │ ├── reconnect_lineages.py # mother+daughter stitching, divides flag - │ ├── tag_cohorts.py # productive / bystander / abortive / mock - │ ├── inspect_candidates.py # per-track image montage QC - │ └── candidates/ # output: lineage-aware annotation CSVs - ├── 1-build_template/ # Path B only - │ ├── build_template.py # NS3 transition template via DBA - │ ├── evaluate_template.py # self-align sanity check - │ ├── templates/ # output: template_*.zarr - │ └── plots/ # output: build-set diagnostics - ├── 2-align_cells/ - │ ├── align_anno.py # Path A-anno: real-time shift on infection_state - │ ├── align_lc.py # Path A-LC: real-time shift on LC predictions - │ ├── align_embedding.py # Path B: hybrid DTW + warp propagation - │ ├── A-anno/alignments/ # per-track output parquets - │ ├── A-LC/alignments/ - │ └── B/alignments/ - ├── 3-organelle-remodeling/ - │ ├── readout_sec61.py # cosine-distance-from-baseline - │ ├── readout_g3bp1.py # oscillation excursion stats - │ ├── readout_phase.py # phase embedding distance - │ ├── A-anno/ # per-track per-organelle outputs - │ ├── A-LC/ - │ └── B/ - └── 4-compare_tracks/ - ├── compare_onsets.py # side-by-side SEC61, G3BP1, phase headlines - ├── compare_phase_to_fluor.py # claim (a') Spearman ρ - ├── warp_vs_no_warp.py # mandatory comparator (Path B only) - ├── bimodality_check.py # dip-test on every back-projected distribution - └── headline_figure.py # the figure-1 of the paper +```text +applications/dynaclr/scripts/pseudotime/0-select_candidates/candidates/ +├── _productive.csv +├── _bystander.csv +├── _abortive.csv +├── _unannotated_productive.csv +├── _mock.csv +└── _funnel.md ``` -Status of stage scripts as of this revision: - -- **Implemented and current:** `0-select_candidates/select_candidates.py`, - `manual_candidates.py`, `inspect_candidates.py`, - `1-build_template/build_template.py`, `evaluate_template.py`, - `2-align_cells/align_cells.py` (current name covers what becomes - `align_embedding.py` after split), per-stage plotting scripts. -- **Implemented but in worktree, not in DAG structure:** - `annotation_remodeling.py` and `prediction_remodeling.py` cover - Path A-anno and Path A-LC respectively; live in - `.claude/worktrees/cytoland-virtual-staining-examples/applications/dynaclr/scripts/pseudotime/`. - Need refresh against new directory structure and configs. -- **TODO: not yet implemented:** `reconnect_lineages.py`, - `tag_cohorts.py`, the per-track split in Stage 2, per-organelle - readouts as separate scripts, all of Stage 4. - -The current scripts still produce useful output but operate on a -single track (Path A-anno-equivalent) without lineage reconnection, -cohort tagging, or cross-track comparison. The phases below describe -the migration. - -## 4. Cohorts - -Stage 0 emits one annotation CSV per cohort. All four cohorts share the -schema. Each cell carries a `cohort` column. - -| Cohort | Definition | Used by | -|---|---|---| -| `productive` | Lineage with NS3 rise within imaging window; manual `[t_before, t_after]` and `t_key_event` from `manual_candidates.py` | Primary cohort; all three tracks | -| `bystander` | Lineage in infected wells with no NS3 rise across imaging duration | Negative control for claims (a), (a'), (b) | -| `abortive` | Lineage with NS3 channel embedding present, no rise | Claim (b) bifurcation comparison (caveat: censored-data category, see discussion §3.2) | -| `mock` | Lineage from uninfected wells | Per-frame null distribution for organelle distance comparisons | - -Mock cells do not get a synthetic `t_zero`. Every mock cell × frame -contributes to an FOV-stratified per-frame null distribution. - -## 5. Stage 0 — Select candidates and reconnect lineages - -Stage 0 emits per-cohort annotation CSVs. Two entry points feed it: -`select_candidates.py` (auto, from existing annotations) and -`manual_candidates.py` (hand-picked tracks). - -The single output artifact is `{cohort}_annotations.csv`, one row per -`(dataset_id, fov_name, lineage_id, t)` over the per-cell crop window. - -### 5.1 Auto path - -```bash -cd applications/dynaclr/scripts/pseudotime/0-select_candidates -uv run python select_candidates.py \ - --datasets ../../../configs/pseudotime/datasets.yaml \ - --config ../../../configs/pseudotime/candidates.yaml \ - --candidate-set infection_transitioning_nondiv -``` +Review the funnel report and cohort row counts before continuing. -Filters tracks per `config["candidate_sets"][NAME]["filter"]` (anchor -label, anchor_positive, min_pre/post_minutes, crop_window_minutes), -expands each track into per-frame rows, copies real annotation labels -onto each row, then runs lineage reconnection (§5.3) and cohort tagging -(§5.4) before writing. +Template construction uses a separate artifact contract: +`_annotations.csv`. The current +`select_candidates.py` command writes cohort CSVs, not this template annotation +file. Use an existing reviewed annotation file in the candidates directory or a +manual candidate generator before Stage 1. -### 5.2 Manual path +## 1. Build and check a template -```bash -cd applications/dynaclr/scripts/pseudotime/0-select_candidates -python manual_candidates.py +```sh +uv run python applications/dynaclr/scripts/pseudotime/1-build_template/build_template.py \ + --datasets applications/dynaclr/configs/pseudotime/datasets.yaml \ + --config applications/dynaclr/configs/pseudotime/build_template.yaml \ + --template ``` -Each track spec is `{t_before, t_after, labels: {...}}` in a Python -dict. Frames in `[t_before, t_after]` get the positive label if they -fall inside an interval. The CSV schema is the only contract with -downstream stages. - -### 5.3 Lineage reconnection - -**TODO: implement.** Stitches mother + daughter chains into single -lineages using `parent_track_id`. Daughter handling regime-dependent: - -- Division before `t_zero`: keep both daughters as paired observations; - siblings are biologically equivalent at infection. -- Division after `t_zero`: keep the daughter with more pre-`t_zero` - footage. Daughters at this stage carry different viral loads. -- Division during the transition window: tag as separate cohort outside - DTW alignment; mitotic ER fragmentation distorts templates. - -Each lineage record carries `divides ∈ {none, pre, during, post}` and -`lineage_id` (replaces `track_id` as the canonical unit downstream). - -### 5.4 Cohort tagging - -**TODO: implement.** For each lineage, derive `cohort` from the -NS3 channel signal: - -- `productive`: lineage has manual `t_key_event` and survives window - cropping in Stage 1. -- `bystander`: in infected well, no NS3 rise across imaging window. -- `abortive`: NS3 channel embedding present, no rise. -- `mock`: from uninfected wells. +The selected template entry identifies its input candidate set and channel. +The script reads: -### 5.5 Inspect - -```bash -cd applications/dynaclr/scripts/pseudotime/0-select_candidates -uv run python inspect_candidates.py \ - --datasets ../../../configs/pseudotime/datasets.yaml \ - --config ../../../configs/pseudotime/candidates.yaml \ - --candidate-set infection_transitioning_nondiv +```text +applications/dynaclr/scripts/pseudotime/0-select_candidates/candidates/ + _annotations.csv ``` -Renders a per-cell-anchored image montage with `t_key_event` marked. -Also writes a sidecar `_qc.csv` with per-track stats (n_frames, -pre_frames, post_frames, fov, divides) for non-visual QC. - -## 6. Stage 1 — Build NS3 transition template (Path B only) - -Stage 1 produces the canonical NS3 transition template against which -Path B aligns query cells. Paths A-anno and A-LC do not use a -template. - -### 6.1 Template build +and writes: -```bash -cd applications/dynaclr/scripts/pseudotime/1-build_template -uv run python build_template.py \ - --config ../../../configs/pseudotime/build_template.yaml \ - --template infection_nondividing_zikv +```text +applications/dynaclr/scripts/pseudotime/1-build_template/templates/ + template_.zarr ``` -The builder: - -1. Reads `productive` cohort annotations. -2. Crops each lineage to `[t_zero - h_pre, t_zero + h_post]` real-time. - Default: `h_pre = 240 min`, `h_post = 360 min` (`540 min` for G3BP1 - readout downstream). See discussion §3.6. -3. Pulls NS3 channel embeddings within the transition sub-window - `[t_zero - k_pre, t_zero + k_post]`. Default: `k_pre = 60 min`, - `k_post = 120 min`. Use the 10 min/frame cohort if available - (target frame count ≥ 12 for DBA stability). -4. Computes per-cell pre-baseline = mean NS3 embedding across pre-window - frames. Cosine distance against this per-cell baseline. -5. Runs DBA on the transition sub-window only. -6. Saves the template zarr. - -### 6.2 Template zarr contents - -| Path | Type | Description | -|---|---|---| -| `template` | (T, D) array | DBA template in the transition window | -| `time_calibration` | (T,) array | mean `t_relative_minutes` per template position | -| `template_labels/{col}` | (T,) array | per-position label fractions | -| `tau_event_band` | (2,) array | `[τ_lo, τ_hi]`: half-rise band of `||dT/dτ||`. The event identifier per discussion §3.4. | -| `lineage_ids` | list (attrs) | `[dataset_id, fov_name, lineage_id]` per build-set lineage | -| `aggregator` | str (attrs) | `"dba"` | -| `template_duration_minutes` | float (attrs) | `time_calibration[-1] - time_calibration[0]` | -| `build_frame_intervals_minutes` | dict (attrs) | per-dataset frame interval at build time | -| `viscy_git_sha`, `dtaidistance_version`, `scikit_learn_version`, `numpy_version` | str (attrs) | provenance | - -**TODO:** add `tau_event_band` to the zarr writer. Currently the -template stores derivative-argmax as a point. - -### 6.3 Self-consistency check - -```bash -uv run python evaluate_template.py \ - --config ../../../configs/pseudotime/build_template.yaml \ - --template infection_nondividing_zikv -``` - -Re-aligns the build-set lineages onto the template they built. Sanity -check, not generalization evidence. - -## 7. Stage 2 — Align query cells per track +Run the self-alignment check: -Stage 2 forks per track. Each track produces an alignment parquet with -the same schema (described in §7.4) so Stage 3 readouts and Stage 4 -comparisons read uniformly. - -### 7.1 Path A-anno: annotation-anchored real-time shift - -**TODO: implement** as `align_anno.py`. Replaces `annotation_remodeling.py`'s -alignment step (currently in -`.claude/worktrees/.../annotation_remodeling.py`). - -Per-cell `t_zero` = first frame where `infection_state == "infected"` -in the manual or auto annotation CSV. Real-time per-cell shift: -`t_rel = (t - t_zero) * frame_interval_minutes`. No DTW, no template, -no warping. - -```bash -# TODO: command shape -uv run python align_anno.py \ - --datasets ../../../configs/pseudotime/datasets.yaml \ - --config ../../../configs/pseudotime/align_cells.yaml \ - --query-set zikv_07_24 +```sh +uv run python applications/dynaclr/scripts/pseudotime/1-build_template/evaluate_template.py \ + --datasets applications/dynaclr/configs/pseudotime/datasets.yaml \ + --config applications/dynaclr/configs/pseudotime/build_template.yaml \ + --template \ + --flavor raw ``` -### 7.2 Path A-LC: LC-anchored real-time shift +Use `--flavor pca` to check the stored PCA flavor. -**TODO: implement** as `align_lc.py`. Replaces `prediction_remodeling.py`. +## 2. Generate alignments -Per-cell `t_zero` = first frame of the longest run of `predicted_infection_state == "infected"` in the NS3 channel embedding zarr. `min_run` parameter (default 3) prevents single-frame flickers from defining `t_zero`. +Annotation-anchored alignment: -```bash -# TODO: command shape -uv run python align_lc.py \ - --datasets ../../../configs/pseudotime/datasets.yaml \ - --config ../../../configs/pseudotime/align_cells.yaml \ - --query-set zikv_07_24 --min-run 3 +```sh +uv run python applications/dynaclr/scripts/pseudotime/2-align_cells/align_anno.py \ + --datasets applications/dynaclr/configs/pseudotime/datasets.yaml \ + --config applications/dynaclr/configs/pseudotime/candidates.yaml \ + --candidate-set \ + --anchor-label infection_state \ + --anchor-positive infected ``` -### 7.3 Path B: hybrid DTW + warp propagation - -Existing `align_cells.py` covers most of this. Renaming and feature gaps -listed below. - -```bash -cd applications/dynaclr/scripts/pseudotime/2-align_cells -uv run python align_embedding.py \ - --datasets ../../../configs/pseudotime/datasets.yaml \ - --config ../../../configs/pseudotime/align_cells.yaml \ - --template infection_nondividing_zikv \ - --query-set zikv_07_24 \ - --min-match-minutes 360 --max-skew 0.7 -``` +Classifier-anchored alignment: -The aligner: - -1. Loads `templates/template_{name}.zarr` and the cohort-tagged query - annotations. -2. Pulls NS3 channel embeddings, applies build-time L2 normalization - (no refit at alignment time). -3. For each query lineage, runs subsequence DTW on the transition - sub-window. The template (length T) must match fully; the query - floats. Returns a warp path, best-match window `[q_start, q_end]`, - cost, and `path_skew`. -4. **TODO:** propagate the warp path to organelle and phase channel - embeddings within the transition sub-window. Pre and post stay in - real-time. -5. **TODO:** for each cell, back-project the τ_event band to a - per-cell real-time interval. Report median + IQR per cohort. -6. Applies guards (§7.6) and writes alignment parquet. -7. Writes a sidecar `{template}_on_{qset}.drop_log.json` with - per-filter drop counts. - -### 7.4 Alignment parquet schema (all three tracks) - -One row per `(dataset_id, fov_name, lineage_id, t)`. Per-lineage columns -are repeated on every frame so downstream scripts can filter rows -without a separate join. - -| Column | Type | Per-frame? | Tracks | Notes | -|---|---|---|---|---| -| `dataset_id`, `fov_name`, `lineage_id`, `t` | ids | yes | all | identifiers (`lineage_id` replaces `track_id`) | -| `cohort` | str | yes | all | `productive` / `bystander` / `abortive` / `mock` | -| `divides` | str | yes | all | `none` / `pre` / `during` / `post` | -| `t_zero` | int | per-lineage | all | per-cell anchor frame | -| `t_rel_minutes` | float | yes | all | real-time minutes from `t_zero` | -| `track_path` | str | per-lineage | all | `A-anno` / `A-LC` / `B` | -| `pseudotime` | float ∈ [0, 1] | yes | B only | warp-path template position | -| `alignment_region` | str | yes | B only | `pre` / `aligned` / `post` | -| `t_rel_minutes_warped` | float | yes | B only | back-projected real-time at template position; equals `t_rel_minutes` outside `aligned` | -| `dtw_cost` | float | per-lineage | B only | raw DTW cost | -| `length_normalized_cost` | float | per-lineage | B only | `dtw_cost / len(warp_path)` | -| `path_skew` | float ∈ [0, 1] | per-lineage | B only | mean deviation from ideal diagonal | -| `match_q_start`, `match_q_end` | int | per-lineage | B only | absolute query frames bounding the matched window | -| `template_id` | str | per-lineage | B only | UUID linking to template zarr | - -`t_rel_minutes` is shared across tracks. For Paths A, it's the only -time coordinate. For Path B, it covers pre/post windows; the transition -window also has `t_rel_minutes_warped` from the back-projection. - -### 7.5 Diagnostic plots per track - -```bash -# Path B only — same scripts as before, renamed -uv run python rank_by_cost.py --query-set zikv_07_24 -uv run python plot_top_n_montage.py --query-set zikv_07_24 --top-n 30 --worst-n 10 -uv run python plot_pcs_aligned.py --query-set zikv_07_24 --top-n 50 +```sh +uv run python applications/dynaclr/scripts/pseudotime/2-align_cells/align_lc.py \ + --datasets applications/dynaclr/configs/pseudotime/datasets.yaml \ + --config applications/dynaclr/configs/pseudotime/candidates.yaml \ + --candidate-set \ + --pred-column predicted_infection_state \ + --positive-value infected \ + --min-run 3 ``` -### 7.6 Guards and frame-rate invariance +Template/embedding alignment: -DTW with generous psi can collapse the template onto a single query -frame. Five guards prevent and surface this: - -| Guard | CLI flag | Default | Rejects | -|---|---|---|---| -| Non-finite cost | always on | — | tracks too short for the solver | -| Path skew | `--max-skew` | 0.7 | degenerate non-diagonal warps (primary gate per discussion §3.8 #2) | -| Length-normalized cost gate | `--cost-gate` | sweep | stereotypy filter; sweep `{0, 10, 20, 30, 50}%` | -| Minute-based match window | `--min-match-minutes` | 360 | template compressed onto tiny real-time window | -| Pre/post headroom | per query-set YAML | 0 | lineages without real footage on either side | - -Path skew is the primary gate; cost is secondary (per discussion §3.8 -#2: skew rejects DTW failures without rejecting biological variance). -**TODO:** wire path-skew-as-primary into the existing two-pass filter -(currently cost-only). - -`--max-psi-minutes` defaults to half the template duration, read from -template attrs. Per-track psi is `round(max_psi_minutes / dataset_frame_interval_minutes)`. - -## 8. Stage 3 — Organelle remodeling readouts - -Stage 3 forks per track per organelle. Each readout reads its track's -alignment parquet and produces population curves and per-cell timing -metrics. The plotting scripts in this stage replace the current -`plot_organelle_remodeling.py` with per-organelle scripts. - -### 8.1 SEC61 readout - -**TODO: implement** as `readout_sec61.py`. Cosine distance of SEC61 -embedding from per-cell baseline (= mean SEC61 embedding across -pre-window frames). Per-cell trajectory binned by `t_rel_minutes`. -Population curve = binned median + IQR. - -```bash -uv run python readout_sec61.py \ - --track {A-anno|A-LC|B} \ - --query-set zikv_07_24 +```sh +uv run python applications/dynaclr/scripts/pseudotime/2-align_cells/align_embedding.py \ + --datasets applications/dynaclr/configs/pseudotime/datasets.yaml \ + --config applications/dynaclr/configs/pseudotime/align_cells.yaml \ + --template \ + --flavor raw \ + --query-set \ + --candidate-set \ + --min-match-minutes 360 \ + --max-skew 0.8 ``` -Headline metric: real-time `t_rel` at which productive median exceeds -FOV-paired mock 95th percentile. Reported per-cohort. For Path B, -back-projected real-time IQR is reported alongside. - -### 8.2 G3BP1 readout - -**TODO: implement** as `readout_g3bp1.py`. Oscillation-aware metrics -on the post-window (real-time, never warped per discussion §3.6 — stress -granule kinetics are sub-frame, warping by NS3 is meaningless): - -| Metric | Definition | -|---|---| -| `excursion_count` | Number of distance threshold crossings in the post-window | -| `dwell_time_minutes` | Total time above threshold | -| `largest_excursion_amplitude` | Max distance above baseline | -| `largest_excursion_duration` | Duration of the longest contiguous excursion | - -Threshold = mock pulsation 95th percentile, FOV-stratified. - -### 8.3 Phase readout - -**TODO: implement** as `readout_phase.py`. Cosine distance of phase -embedding from per-cell baseline. Same structure as SEC61. +The outputs are: -For claim (a'), phase per-cell `t_50` (or onset-time metric) is -extracted and matched to the same cell's SEC61 fluorescence onset -time. The pair `(t_phase, t_sec61)` per cell feeds Stage 4's Spearman -ρ comparison. - -### 8.4 Per-cell timing metrics - -**TODO: implement** per-track `compute_timing_metrics.py`. Replaces -the existing single-track version under `3-organelle-remodeling/`. +```text +applications/dynaclr/scripts/pseudotime/2-align_cells/ +├── A-anno/alignments/.parquet +├── A-LC/alignments/.parquet +└── B/alignments/