DynaCLR models staging: requeue (#478) + rescale/LOT (#424) + temporal straightening (#493)

.claude/skills/airtable-build-collection/SKILL.md

@@ -32,6 +32,7 @@ Key fields in the Datasets table:
 | `channel_0_marker` .. `channel_N_marker` | Protein marker for each channel |
 | `t_shape`, `c_shape`, `z_shape`, `y_shape`, `x_shape` | Array dimensions |
 | `pixel_size_xy_um`, `pixel_size_z_um` | Physical pixel sizes |
+| `microscope` | Acquisition platform (e.g. `mantis_v1`, `mantis_v2`); derive from pixel size if absent |
 
 ## Usage
 
@@ -155,13 +156,18 @@ experiments:
     moi: <moi or 0.0>
     pixel_size_xy_um: <from airtable>
     pixel_size_z_um: <from airtable>
+    microscope: <from airtable microscope field; else derive: mantis_v1 / mantis_v2>
 ```
 
 Key notes:
 - `data_path` should be the zarr store root (up to `.zarr`), NOT the FOV-level path from Airtable
 - `perturbation_wells` uses `uninfected` / `<perturbation>` keys inferred from the `perturbation` field
 - `channels` lists only the channels needed for training (not all channels in the zarr)
 - `marker` at the experiment level is the primary marker for this experiment entry
+- `microscope` labels the acquisition platform — always populate it. Use the Airtable microscope
+  field when present; otherwise derive from acquisition grid: `pixel_size_xy_um == 0.1494` →
+  `mantis_v1`, `0.1133` → `mantis_v2`. Downstream QC (embedding-consistency-qc) partitions
+  datasets by `obs["microscope"]`, so a missing/empty value silently collapses the v1/v2 split.
 
 ### Step 7: Save and Validate
 

.ed_planning/visreg/temporal-regularization/PLAN.md

@@ -0,0 +1,59 @@
+# Temporal regularization of DynaCLR embeddings — plan
+
+**Status (2026-07-27):** implementation, PR-review hardening, and local integration validation complete; remote CI remains.
+
+## Objective
+
+Keep NT-Xent for state separation, then add two losses on encoder features:
+
+- a shared next-state predictor, `L_pred`, to reward transition structure shared across cells;
+- a three-frame curvature loss, `L_curv`, to make distance along a track better reflect progression.
+
+The losses create temporal structure; they are not gated on straightness already being present in a frozen contrastive embedding.
+
+## Locked design
+
+- Emit fixed `K`-frame, `tau`-spaced sequences as row-major `B*K` tensors.
+- A valid sequence uses one exact `global_track_id` and marker. It stops at divisions and never switches siblings.
+- Apply temporal losses directly to encoder features; do not add a separate temporal head.
+- Stop-gradient the predictor target at `t+1`.
+- Keep temporal weights independently schedulable.
+- Default to track-consistent stochastic augmentation. Retain `independent` and `none` as explicit ablations.
+- Mask incomplete sequences while keeping every configured module in the distributed backward graph.
+
+## Milestones
+
+- [x] Sequence sampling, rectangular batching, and validity masks.
+- [x] Curvature loss and shared predictor.
+- [x] Lightning integration, schedules, and configuration.
+- [x] Local end-to-end smoke coverage.
+- [x] PR review hardening and regressions.
+- [x] Run the final combined local suite.
+- [ ] Confirm remote CI after push.
+- [ ] Compare biology and representation metrics against the NT-Xent baseline.
+
+## PR review resolutions
+
+| Priority | Risk | Resolution |
+| --- | --- | --- |
+| P1 | A lineage stencil could jump between siblings. | Match every frame by exact `global_track_id` and invalidate at division boundaries (`061d6e31`). |
+| P1 | Flattened `B*K` frames received independent random transforms. | Reuse one transform realization per track; expose `consistent`, `independent`, and `none` modes (`061d6e31`). |
+| P1 | An all-invalid rank left predictor parameters unused in DDP. | Forward the empty masked tensor through the predictor and backpropagate a graph-connected zero (`651d693c`). |
+| P2 | `positive_cell_source=self` skipped the lookup needed by sequence emission. | Build the lineage/timepoint lookup whenever sequences are enabled (`061d6e31`). |
+| P2 | Half-open HPI bins omitted a maximum on an exact boundary. | Share an edge builder that always adds a terminal bin (`f9426fab`). |
+| P3 | Zero, negative, or non-finite HPI widths were accepted. | Reject invalid widths during Pydantic validation (`f9426fab`). |
+
+## Acceptance checks
+
+- Exact-track and division-boundary sampler regressions pass.
+- Self-positive sequence emission passes.
+- Track-consistent flip regression passes.
+- Predictor parameters receive non-`None`, zero gradients for all-invalid batches.
+- HPI terminal-boundary and invalid-width regressions pass.
+- Dataset/datamodule, engine, witness-GMM, and final integration suites pass.
+
+Local combined validation passed on 2026-07-27. The two skipped inference-reproducibility tests require external HPC data and CUDA; there were no failures.
+
+## Follow-up experiment
+
+Train matched seeds for NT-Xent-only versus NT-Xent + `L_pred` + `L_curv`. Compare contrastive retrieval, collapse indicators, temporal prediction, per-track progression, and held-out biological separation before promoting the temporal objective to a default recipe.

.ed_planning/visreg/temporal-regularization/straightening-on-dynaclr.md

@@ -0,0 +1,63 @@
+# Straightening on DynaCLR — exploration
+
+**Status (2026-07-27):** promoted to implementation; review findings incorporated.
+
+## Question
+
+Can DynaCLR learn a common biological progression without erasing state organization?
+
+## Working hypothesis
+
+Straightening alone is a per-track geometric prior; it does not directly reward a transition shared across cells. The selected objective therefore combines:
+
+`L = L_NT-Xent + lambda_pred * L_pred + lambda_curv * L_curv`
+
+The shared predictor supplies the common-transition pressure, curvature regularization makes local motion easier to interpret, and NT-Xent resists constant-state collapse.
+
+## Implementation findings
+
+### Identity is stricter than lineage
+
+A lineage may contain a parent and multiple daughters. Sampling any same-lineage row at each timepoint can create a synthetic path that changes physical cells. Temporal stencils must match the anchor's exact `global_track_id`; a division makes the stencil invalid.
+
+### Augmentation is part of the temporal model
+
+Applying random transforms to flattened `B*K` frames independently injects artificial motion. The default must reuse one random realization across the `K` frames of each track while allowing different realizations across tracks. Independent and augmentation-free modes remain useful ablations.
+
+### Masked loss still has distributed semantics
+
+On a rank with no valid sequences, returning a zero connected only to encoder output leaves predictor parameters unused. Passing the empty input through the predictor produces zero gradients for its parameters and keeps DDP iteration state consistent.
+
+### Evaluation bins need explicit boundary semantics
+
+HPI loops use half-open intervals `[lo, hi)`. `np.arange(start, max + width, width)` still omits `max` when it is exactly a boundary. The edge builder must create one additional terminal edge, and widths must be finite and positive before arithmetic.
+
+## Decisions retained
+
+- Operate on encoder features, matching the representation being regularized.
+- Use three or more points for curvature; two points provide smoothing only.
+- Keep fixed-frame spacing initially for a small, auditable implementation.
+- Treat velocity-based phenotype splitting as a possible discovery signal, not automatically as representation damage.
+- Evaluate biology and collapse jointly; straightness by itself is not a success criterion.
+
+## Rejected shortcuts
+
+- Same-lineage sampling without exact track identity.
+- Independent stochastic transforms as the default temporal input.
+- Skipping configured modules on empty masked batches under DDP.
+- Using frozen-embedding straightness as a gate for whether training may induce temporal structure.
+
+## Evidence added by review hardening
+
+- Exact-track selection chooses one sibling consistently and invalidates parent-to-daughter stencils.
+- `self` positives can emit sequences without a missing lookup.
+- Repeated frames remain identical after track-consistent random flips.
+- Every predictor parameter receives a zero, non-`None` gradient on all-invalid batches.
+- Boundary-aligned maximum HPI values fall inside a bin; invalid widths fail at config construction.
+
+## Open empirical questions
+
+- Does the predictor learn biology rather than acquisition-time drift?
+- Which temporal weight schedule preserves contrastive retrieval best?
+- Is `K=3` sufficient, or do longer stencils improve robustness enough to justify their sampling cost?
+- Do consistent spatial transforms improve temporal metrics without weakening useful augmentation diversity?

.gitignore

@@ -73,6 +73,3 @@ lightning_logs/
 checkpoints/
 
 plot_related/
-
-# Local-only planning docs (not for upstream)
-applications/dynaclr/docs/DAGs/evaluation_matrix.md

applications/dynaclr/README.md

@@ -48,6 +48,18 @@ uv run --package dynaclr viscy predict -c examples/configs/predict.yml
 sbatch examples/configs/fit_slurm.sh
 ```
 
+### End-to-end pipeline (new dataset → embeddings → eval)
+
+For the full "dataset → per-marker embeddings → downstream evals" workflow — including the
+one-call `dynaclr predict-triplet` (writes the dataset-centric tree
+`<dataset>/2-phenotyping/predictions/{model}/{run}/{ckpt}/{marker}.zarr`), the decoupled
+Nextflow `eval_from_embeddings` entry, and the **model matrix** for running many models through
+train → predict → eval in parallel — see:
+
+- [`docs/DAGs/end_to_end.md`](docs/DAGs/end_to_end.md) — the pipeline overview + Quickstart runbook
+- [`nextflow/README.md`](nextflow/README.md) — the two Nextflow eval entries
+- [`tools/README.md`](tools/README.md) — the model matrix (`dynaclr run-matrix`) + launchers
+
 The YAML config determines which model and data module to use via `class_path`:
 
 ```yaml
@@ -78,12 +90,21 @@ DynaCLR also provides evaluation-specific commands via `dynaclr <command>`:
 
 | Command | Description |
 |---------|-------------|
+| `predict-triplet` | One-call per-marker embedding inference from a collection + checkpoint → dataset-centric tree |
+| `predict-batch` | Run `predict-triplet` for one model with AI-readiness preflight |
+| `eval` | Evaluate frozen embeddings selected by model/run/checkpoint |
+| `run-matrix` | Chain train → predict → eval for many models through SLURM dependencies |
+| `split-embeddings` | Split a combined embeddings zarr per experiment/marker (`--route-by-dataset` → the dataset tree) |
+| `embedding-consistency-qc` | Build per-marker cross-dataset MMD² and correlation matrices |
+| `run-linear-classifiers` | Train linear classifiers on embeddings (batch, CSV metrics) |
+| `append-annotations` / `append-predictions` | Join annotation / predicted-label columns onto per-experiment zarrs |
 | `train-linear-classifier` | Train a linear classifier on cell embeddings |
 | `apply-linear-classifier` | Apply a trained linear classifier to new embeddings |
 | `append-obs` | Append columns from a CSV to an AnnData zarr obs (with optional prefix, e.g. `annotated_`, `feature_`) |
 | `reduce-dimensionality` | Compute PCA, UMAP, and/or PHATE on saved embeddings |
 | `evaluate-smoothness` | Evaluate temporal smoothness of embedding models |
 | `compare-models` | Compare previously saved smoothness results |
+| `compute-mmd` | MMD between perturbation / experiment groups (cross-experiment consistency) |
 | `info` | Print summary of an AnnData zarr store |
 
 ```bash

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2024_11_07_A549_SEC61_DENV.yml

@@ -0,0 +1,41 @@
+name: organelle-box-2024_11_07_A549_SEC61_DENV
+description: "Profiling/handoff collection: 2024_11_07 A549 SEC61 DENV (single-marker
+  organelle box). Channels embedded as bag-of-channels samples: SEC61B (GFP),
+  viral_sensor (mCherry), Phase3D. Prepared zarr on VAST with timepoint_statistics +
+  focus_slice. Sourced from Airtable base app8vqaoWyOwa0sB5 (dataset
+  '2024_11_07_A549_SEC61_DENV'). FOV C/1/000000 exists in the zarr but has no tracking
+  CSV and is excluded at build time."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "{dataset}=\"2024_11_07_A549_SEC61_DENV\""
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2024_11_07_A549_SEC61_DENV
+    data_path: ${datasets_root}/2024_11_07_A549_SEC61_DENV/2024_11_07_A549_SEC61_DENV.zarr
+    tracks_path: ${datasets_root}/2024_11_07_A549_SEC61_DENV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: SEC61B
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - B/1
+        - B/3
+      DENV:
+        - B/2
+        - C/2
+    exclude_fovs:
+      - C/1/000000
+    interval_minutes: 10.0
+    start_hpi: 4.0
+    marker: SEC61B
+    organelle: endoplasmic_reticulum
+    moi: 5.0
+    pixel_size_xy_um: 0.1494
+    pixel_size_z_um: 0.174

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_08_A549_G3BP1_ZIKV.yml

@@ -0,0 +1,38 @@
+name: organelle-box-2026_04_08_A549_G3BP1_ZIKV
+description: "Embed-only collection: 2026_04_08 A549 G3BP1 ZIKV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: G3BP1 (GFP), viral_sensor (mCherry), Phase3D.
+  Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_08_A549_G3BP1_ZIKV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_08_A549_G3BP1_ZIKV
+    data_path: ${datasets_root}/2026_04_08_A549_G3BP1_ZIKV/2026_04_08_A549_G3BP1_ZIKV.zarr
+    tracks_path: ${datasets_root}/2026_04_08_A549_G3BP1_ZIKV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: G3BP1
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - D/6
+      ZIKV:
+        - B/5
+        - B/6
+        - C/5
+        - C/6
+        - D/5
+    interval_minutes: 30.0
+    start_hpi: 3.0
+    marker: G3BP1
+    organelle: stress_granules
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_10_A549_TOMM20_ZIKV.yml

@@ -0,0 +1,38 @@
+name: organelle-box-2026_04_10_A549_TOMM20_ZIKV
+description: "Embed-only collection: 2026_04_10 A549 TOMM20 ZIKV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: TOMM20 (GFP), viral_sensor (mCherry), Phase3D.
+  Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_10_A549_TOMM20_ZIKV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_10_A549_TOMM20_ZIKV
+    data_path: ${datasets_root}/2026_04_10_A549_TOMM20_ZIKV/2026_04_10_A549_TOMM20_ZIKV.zarr
+    tracks_path: ${datasets_root}/2026_04_10_A549_TOMM20_ZIKV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: TOMM20
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - A/2
+      ZIKV:
+        - A/3
+        - A/4
+        - B/2
+        - B/3
+        - B/4
+    interval_minutes: 30.0
+    start_hpi: 5.0
+    marker: TOMM20
+    organelle: mitochondria
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_14_A549_SEC61B_DENV.yml

@@ -0,0 +1,38 @@
+name: organelle-box-2026_04_14_A549_SEC61B_DENV
+description: "Embed-only collection: 2026_04_14 A549 SEC61B DENV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: SEC61B (GFP), viral_sensor (mCherry), Phase3D.
+  Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_14_A549_SEC61B_DENV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_14_A549_SEC61B_DENV
+    data_path: ${datasets_root}/2026_04_14_A549_SEC61B_DENV/2026_04_14_A549_SEC61B_DENV.zarr
+    tracks_path: ${datasets_root}/2026_04_14_A549_SEC61B_DENV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: SEC61B
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - C/2
+      DENV:
+        - C/3
+        - C/4
+        - D/2
+        - D/3
+        - D/4
+    interval_minutes: 30.0
+    start_hpi: 3.0
+    marker: SEC61B
+    organelle: endoplasmic_reticulum
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_21_A549_G3BP1_DENV.yml

@@ -0,0 +1,41 @@
+name: organelle-box-2026_04_21_A549_G3BP1_DENV
+description: "Embed-only collection: 2026_04_21 A549 G3BP1 DENV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: G3BP1 (GFP), viral_sensor (mCherry), Phase3D.
+  Source zarrs are the intracellular_dashboard preprocess outputs (5-assemble data +
+  4-track ultrack labels); no prepared VAST copy exists. Predict uses fov_statistics
+  normalization, so timepoint_statistics is not required. Annotations exist (infection,
+  organelle, cell_division, cell_death)."
+datasets_root: /hpc/projects/intracellular_dashboard/organelle_dynamics
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_21_A549_G3BP1_DENV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_21_A549_G3BP1_DENV
+    data_path: ${datasets_root}/2026_04_21_A549_G3BP1_DENV/1-preprocess/5-assemble/2026_04_21_G3BP1_DENV.zarr
+    tracks_path: ${datasets_root}/2026_04_21_A549_G3BP1_DENV/1-preprocess/4-track/2026_04_21_G3BP1_DENV.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: G3BP1
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - A/1
+      DENV:
+        - A/2
+        - A/3
+        - B/1
+        - B/2
+        - B/3
+    interval_minutes: 30.0
+    start_hpi: 3.0
+    marker: G3BP1
+    organelle: stress_granules
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995