DynaCLR models staging: requeue (#478) + rescale/LOT (#424) + temporal straightening (#493) - #486
Open
edyoshikun wants to merge 98 commits into
Open
DynaCLR models staging: requeue (#478) + rescale/LOT (#424) + temporal straightening (#493)#486edyoshikun wants to merge 98 commits into
edyoshikun wants to merge 98 commits into
Conversation
…staging Scale-aware patch extraction (packages/viscy-data): - Add _read_pixel_size() helper to read X pixel size from OME-Zarr metadata - Add reference_pixel_size parameter to TripletDataModule: when set, computes initial_yx_patch_size from the pixel-size ratio so the same physical area is covered at inference time - Replace BatchedRescaleYXd (removed per review) with existing BatchedZoomd using scale_factor=(1.0, final_y/initial_y, final_x/initial_x) and mode="bilinear" with antialias; import is lazy to avoid a hard dep LOT batch correction (applications/dynaclr): - Add dynaclr.evaluation.lot_correction submodule with core logic (fit, apply, save, load), Pydantic configs, and Click CLI entry points - Register fit-lot-correction and apply-lot-correction in cli.py via LazyCommand - Add pot and joblib to optional-dependencies.eval in pyproject.toml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Plots n cell patches side-by-side: - Left: raw patch at initial_yx_patch_size (larger physical area) - Right: same patch bilinearly downscaled to final_yx_patch_size (model input) Both columns share the same percentile contrast window so differences are spatial. Physical scale (µm × µm) is shown in each panel title. Usage: python visualize_triplet_rescaling.py --data-path /path/to/data.zarr --tracks-path /path/to/tracks --source-channel Phase3D --z-range 0 5 --final-yx-patch-size 224 224 --reference-pixel-size 0.325 --output rescaling_comparison.png Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…meta
_collate_norm_meta iterated every normalization level and called
torch.stack on each stat, but timepoint_statistics is nested
{timepoint: {stat: tensor}} rather than flat {stat: tensor}. Any zarr
carrying timepoint_statistics (alongside fov/dataset stats) crashed
batch collation in TripletDataModule with "expected Tensor as element 0
... but got dict". Stack within each timepoint sub-dict instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…etDataModule
Lets a 3D OME-Zarr feed a 2D model without materializing a separate MIP
dataset, and centers the extracted Z window on each FOV's focus plane.
- z_reduction ("mip"/"center"): collapse the extracted z_range to one
slice via BatchedChannelWiseZReductiond. Label-free channels (resolved
by parse_channel_name) take the center slice; others are max-projected.
on_after_batch_transfer expects Z=1 when reduction is on.
- z_extraction_window/z_focus_offset/focus_channel: resolve a per-FOV
focus-centered window from each position's
focus_slice[ch].fov_statistics.z_focus_mean (fallback z_total//2), all
windows the same width. z_range stays as an explicit override; exactly
one of z_range / z_extraction_window must be given. Per-FOV windows are
resolved at setup() and looked up per patch in the dataset.
Tests cover both reduction strategies (discriminating center vs MIP),
the normalize-then-reduce order, per-FOV focus resolution, and the
z_range/z_extraction_window XOR guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Documents the TripletDataModule predict path (zarr + tracking, not parquet) and adds a runnable sample config demonstrating z_reduction + reference_pixel_size to feed a 3D dataset to a 2D model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Notebook-style script (no CLI) that loads a 3D OME-Zarr + tracking, extracts a per-FOV focus-centered Z window, collapses it via z_reduction, applies a random affine so anchor/positive diverge, and visualizes a couple of batches to a PNG. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Declare viscy-transforms in viscy-data's triplet extra and hoist the BatchedZoomd/BatchedChannelWiseZReductiond imports to module top; the inline imports were masking a missing runtime dependency that would ImportError for anyone using z_reduction or reference_pixel_size. - Raise in _resolve_per_fov_z_ranges when a FOV's Z is smaller than z_extraction_window, instead of silently emitting a narrower window that breaks cross-FOV batch stacking. - Remove a dead duplicate break in test_focus_centered_z_range. - Correct the inference DAG doc: embeddings live in .X (the embedding_key array), mirrored to obsm["X_backbone"]/["X_projections"]. uv.lock intentionally omitted: the workspace glob currently entangles the untracked applications/eet package, so a clean regen of the single viscy-transforms edge is not possible until eet is committed or removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the NumPy/scipy.cdist backend with PyTorch so the pooled RBF kernel matrix is built once on the available device (CUDA if usable, else CPU) and reused across all permutations. Device selection probes a trivial kernel launch and falls back to CPU, so a present-but-unusable GPU (e.g. compute capability older than the torch build) does not crash. Public API is unchanged: median_heuristic/compute_mmd_unbiased return the same bandwidth-convention values, and mmd_permutation_test still returns (mmd2, p_value, null_distribution) so dynaclr's effect-size and activity z-score computations keep working. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add witness_function to the MMD module: the per-point empirical witness w(z) = mean_i k(z, x_i) - mean_j k(z, y_j), the RKHS direction along which distributions P (X) and Q (Y) differ. Positive scores lean toward X, negative toward Y. Reuses the GPU RBF-kernel helpers and bandwidth (median-heuristic) convention already in the module, and chunks over query points to bound the intermediate kernel matrices. Tests cover distribution separation, X<->Y antisymmetry, and chunk-size invariance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds `label_source: witness` to the linear-classifier eval as a second option to annotation-CSV labels. Per marker, cells are pooled across experiments, control/perturbed references are built from per-experiment wells, the MMD witness is fit and scored per cell, and scores are gated (sign + dead-zone) into control/perturbed pseudo-labels. Everything downstream (train_linear_classifier, publish, append-predictions, plots) is unchanged — only the label source differs. - evaluate_config.py: WitnessLabelSource, WitnessSettings, label_source switch on LinearClassifiersStepConfig (annotations stays the default). - witness_labels.py: well->reference mask (path-component match so C/1 != C/10), witness fit/score, dead-zone gating. - orchestrated.py: branch label assembly into _annotation_run_specs / _witness_run_specs / _build_labeled_adata; training loop unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Five integration tests exercising the real code path: - well-prefix match rejects C/1 vs C/10 confusion - witness scores separate control/perturbed clusters - dead-zone drops ambiguous near-origin cells - empty AnnData when a reference group has no cells - end-to-end run_linear_classifiers witness mode → metrics + summary PDF Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors linear_classifiers_infectomics.yml but with label_source: witness. Per-experiment control/perturbed wells replace annotation CSVs; wells are templates to edit per plate layout. One classifier per marker (G3BP1, SEC61B, Phase3D, viral_sensor), dead_zone 0.1, track-level split. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- witness_score_classifiers.md: full DAG (why, step-by-step, gating rule, config, notes) plus embedded visuals. - visuals/: three Graphviz flow diagrams (data flow, gating, pipeline) as .dot + png + pdf, and four mock example-output plots (witness-score histogram, per-marker metrics bar, ROC, F1-over-time) with a reproducible generator (mock_witness_plots.py). Mock plots are watermarked synthetic. - Cross-links from evaluation.md and linear_classifiers/README.md so the new DAG is discoverable from the annotation path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The witness label is a deterministic function of the embedding, so
evaluating a witness-trained classifier against that same label is
circular — it reports a trivial ~1.0 (verified on real infectomics
embeddings: 1.000 vs witness label, 0.712 vs true infection_state).
WitnessSettings gains eval_against (default "infection_state") and
eval_class_map ({control: uninfected, perturbed: infected}). In witness
mode, val metrics are recomputed on the val split against the ground-truth
obs column via the class map; the summary records eval_source. When the
column is absent, metrics fall back to the witness label flagged as
eval_source="witness_label". Adds two tests (annotation eval + fallback).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Score the witness-trained classifier against infection_state (not the self-referential witness label) with the control→uninfected / perturbed→infected class map, matching the leakage fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- witness_score_classifiers.md: add "Evaluation: avoid the circularity trap" section with the real per-marker metrics table (scored vs infection_state), eval_against/eval_class_map in the config block, and step 7 in the flow; refresh the Notes caveat (metrics are honest now, check eval_source; note class imbalance). - witness_dataflow.dot/png/pdf: add the EVALUATE-vs-annotations node. - mock_witness_plots.py + bar/ROC png/pdf: use real run values (viral_sensor 0.87, SEC61B 0.76, Phase3D 0.58, G3BP1 0.49) instead of invented all-high numbers; watermark clarifies bar/ROC are real, hist/F1 synthetic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Witness mode produced only the standard metrics/ROC/F1 plots — nothing
showing HOW the pseudo-labels were chosen. build_witness_labels now
attaches the gating diagnostic to the returned AnnData (obs["witness_score"],
obs["witness_ref"] = control_well/perturbed_well/other, uns["witness_gating"]
with threshold, bandwidth, counts, and the pre-gating score/ref arrays).
_save_task_plots leads the {task}_summary.pdf with a per-marker gating page:
the witness-score histogram split by well-of-origin, the dropped dead-zone
band, the sign cut, and labeled/dropped counts. Verified on real SEC61B
embeddings — control/perturbed wells separate weakly, most labeled cells are
"other" near zero, which explains the modest AUROC at a glance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related extensions to the witness LC label source:
1. Filter-based references. WitnessLabelSource gains control_filter /
perturbed_filter (arbitrary obs filters) as an alternative to
control_wells / perturbed_wells. obs_filter_mask supports scalar (==),
list (isin), and range dicts ({lt,le,gt,ge}, one bound = half-line, two =
window), with a well/fov_name key routing to the well-prefix match. This
enables contrasts like early-vs-late timepoints or control-vs-perturbed-at-
late as weak-label sources. Validator: wells XOR filter per side, no mixing.
2. Annotation-scored eval. WitnessSettings.eval_annotations lets a witness run
(weak labels for training) be scored against real infection_state joined
from annotation CSVs when the embeddings obs lacks the column, instead of
the perturbation proxy. _join_eval_annotations does the per-experiment join.
Tests: obs_filter_mask forms + filter-based late-window integration (8 green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tness label The leakage fix recomputed the metrics CSV against ground-truth infection_state but left the ROC + F1-over-time pages drawing from the trainer's witness-label val arrays — so the summary PDF showed a circular AUROC=1.000 while the metrics bar/CSV correctly showed ~0.80. _evaluate_witness_against_annotations now also returns the annotation-scored y_val / y_val_proba / classes / val_hours (aligned to the has-ground-truth subset), and the orchestrator swaps them into the plotting outputs so the whole PDF is consistent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The witness axis measures how much a MARKER's embedding changes between the
references, so its meaning is marker-dependent: viral_sensor -> infection,
organelle markers (SEC61B/TOMM20/G3BP1) -> remodeling. Scoring every marker
against infection_state was wrong for the organelle markers.
WitnessSettings.marker_eval maps a marker to {eval_against, eval_class_map},
overriding the top-level target for that marker (absent markers fall back).
_resolve_witness_eval applies the override per run; eval_source in the summary
records the actual target used. Lets one run score viral_sensor vs
infection_state and SEC61B vs organelle_state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the sign+dead-zone witness gate and its circular annotation grading
with a two-stage, annotation-format pipeline (Soorya Pradeep's recipe,
framework-ized). A witness->GMM label IS an annotation: its meaning is named by
the modality it is computed from (viral_sensor -> infection_state; organelle ->
organelle_remodeling_state), so it flows through the existing annotation training
path unchanged.
Stage A (new `dynaclr witness-gmm-labels`): witness score -> per-condition
2-component GMM -> writes an annotation file (named state column + real class
vocabulary). Negatives = all control-well cells; positives = perturbed cells
with GMM posterior >= threshold; unimodal (near-noise) markers skipped.
Stage B: the existing `run-linear-classifiers` (label_source: annotations),
untouched. Teacher/student is free — point Stage B at a different modality's
zarr (e.g. SEC61 labels -> train on phase), joined by cell key.
- New shared `viscy_utils.evaluation.witness_gmm.fit_gmm_labels` (+ tests).
- Rename witness_labels{,_test}.py -> witness_gmm_labels{,_test}.py; recipe ->
witness_gmm_labels_infectomics.yml.
- Remove WitnessSettings, _gate_scores, _evaluate_witness_against_annotations,
_resolve_witness_eval, _join_eval_annotations, the label_source: witness
branch, and the superseded witness_score_classifiers DAG + visuals.
- New DAG witness_gmm_classifiers.md; update evaluation.md + LC README.
- Validated: fit_gmm_labels reproduces Soorya's saved GMM on
2026_03_24_A549_SEC61_ZIKV (bandwidth 238~240, remod weight 0.655~0.649,
confident-pos 59.1%~58.4%).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove generated mock visualization PNG/PDF files from version control. Files remain on disk but are no longer tracked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Curvature loss (1 - cos of consecutive latent velocities) and a shared next-state predictor MLP for temporal-dynamics learning, both acting on the encoder embedding z. Stop-grad predictor target; DDP-safe zero for all-invalid sequence batches. Includes unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add emit_sequence to the datamodule/dataset: sample K consecutive same-track same-marker frames (fixed frame stride) via the lineage lookup, emit a (B*K,C,Z,Y,X) 'sequence' batch key with a valid mask. Engine encodes it in one forward and adds lambda_curv*L_curv + lambda_pred*L_pred on z, with cosine-annealed weights. All defaults disabled so existing configs are unaffected. Includes a fast-dev smoke config. Also fix a pre-existing z_focus NaN guard in _slice_patch that only caught Python float NaN, not numpy float, crashing int(NaN) on parquets with unpopulated z_focus rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the GroupShuffleSplit-vs-train_test_split decision and the split_groups_by group-id construction into group_val_split() and group_ids_from_obs() in linear_classifier.py. train_linear_classifier and orchestrated.py's val-index replay now call them, removing three copies of the same split logic. Pure extract-method: same (seed, groups, y) yields identical indices, so training and val-metric behavior are unchanged (verified: Phase3D val AUROC 0.915, SEC61B 1.000, and a bit-for-bit idx_val match to the prior inline split). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
End-to-end two-stage recipe on 2026_07_01_A549_SEC61B_TOMM20_G3BP1_ZIKV: Stage A (witness-gmm-labels on SEC61B organelle) then Stage B (run-linear-classifiers). Two Stage-B configs share the same labels and params, differing only by input channel: 02_stage_b_train_phase (teacher/student, trains on Phase3D) and 02_stage_b_train_sec61b (same-modality). run_*.sh drives both stages (CPU-only, bash or sbatch). Reproduces Soorya's per-timepoint AUROC band. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…odeled) plot_timecourse_and_prob_grids.py produces three diagnostics for the ZIKV plate: D1 per-timepoint AUROC overlay (SEC61B witness vs Phase3D classifier), D2 P(remodel)-split sample grids (HPI bins x 5 probability buckets) per marker, and D3 %-remodeled-vs-time (witness-GMM gate + SEC61B/Phase3D logistic classifiers, ZIKV vs control false-positive). Reuses the shared LC split helpers and the real Stage-A compute_marker_scores path so the plotted scores match Stage A/B exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Raw per-marker embedding zarrs now write to
{dataset}/2-phenotyping/predictions/{family}/{run}/{ckpt}/embeddings/{marker}.zarr
so the {ckpt} dir is a container for downstream analysis (labels, plots, umap,
...) as sibling folders. embedding_store + eval_launch glob both updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bs enrichment
Expand the per-reporter triplet predict path (dynaclr predict-triplet):
- foundation model_type: instantiate the model: block of a LightningCLI
training_config (frozen encoder loads its own weights) instead of loading a
ContrastiveModule checkpoint; --checkpoint / --training-config gate by type.
- focus-centered Z: --z-window WIDTH centered per-FOV on the focus plane
(--focus-channel / --z-focus-offset), mutually exclusive with fixed --z-range;
--reference-pixel-size-z-um converts the reference-grid slice count to the
native count covering the same physical depth.
- obs enrichment: append collection metadata (experiment/marker/perturbation/
organelle/microscope/hours_post_perturbation/interval_minutes) to each
embedding's obs so triplet and parquet embeddings are consumed identically;
--no-enrich-obs opts out. New enrich-obs-from-collection subcommand backfills
obs onto already-written embeddings.
- exclude_fovs from the collection experiment is forwarded to predict.
Update the {ckpt}/embeddings/{marker}.zarr path expectation in
test_output_path_is_dataset_centric (embeddings/ nesting from d89f9ed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lections
run-matrix / predict-batch gain:
- foundation rows (model_type: foundation): resolve from explicit identity +
training_config (no train_sbatch parsing), never emit a train stage, and
forward --model-type/--training-config through predict-batch and predict.sbatch
instead of a checkpoint.
- focus-centered Z + physical Z reference (z_window/focus_channel/z_focus_offset/
reference_pixel_size_z_um) forwarded as the trailing predict positionals;
z_range and z_window validated mutually exclusive.
- skip-existing (default) via resolve_datasets_to_run: prune datasets whose
per-marker embeddings already exist for a row's (family, run, ckpt_name);
drop fully-done rows, write a pruned temp collection for partial ones. A dir
without group metadata (crash mid-write) counts as incomplete and re-runs.
--overwrite bypasses the check; default --stages is predict,eval.
Point _mark_done at the {ckpt}/embeddings/ path so completion detection matches
the nested layout (d89f9ed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the merged organelle_remodeling collection (9 experiments: SEC61B/TOMM20/ G3BP1 reporters under ZIKV/DENV, one experiment per dataset, all prepared on VAST with normalization + focus_slice). Includes 2026_03_24_A549_SEC61_ZIKV (re-concatenated + QC'd clean, mantis_v2 at 0.1133/0.16995 um); 04_21 G3BP1 DENV stays deferred. Repoint the matrix configs at the new progressive-collection workflow: - organelle_remodeling.yml matrix: one row per MODEL (DynaCLR-2D-MIP-BoC single marker + MorphEm-frozen foundation baseline) over the shared collection; focus-centered Z (z_window 16, focus_channel Phase3D, reference_pixel_size_z_um 0.174); output root on intracellular_dashboard. - example.yml: document model_type foundation rows, focus-centered vs fixed Z, and skip-existing defaults. - 07_01 collection: add the Phase3D channel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ription 2026_04_21_A549_G3BP1_DENV was re-concatenated to a flat store on VAST (odd 5-assemble layout resolved) and verified predict-ready. Note the residual z_focus=0 frames (12/54 FOVs, mean 2.4 timepoints each). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add audit_focus_slice() + a `qc audit-focus` CLI command that reads already-
written focus_slice.{channel}.per_timepoint metadata and reports where in-focus
detection landed at a stack edge (z == 0 or z == Z-1), the signature of a failed
search on a 3D stack. Z-depth-aware: a 2D acquisition (Z == 1) trivially focuses
at slice 0 for every timepoint, so it is reported as 2D with zero flags rather
than misclassifying every frame as failed.
Returns per-FOV suspect counts, dataset totals, and the valid (non-edge) focus
distribution — the summary previously computed ad hoc while vetting datasets for
focus-centered predict.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
embedding_store writes .../{ckpt}/embeddings/{marker}.zarr since the embeddings/
nesting landed (d89f9ed), but iter_embeddings still globbed .../{ckpt}/*.zarr —
so it returned [] and ALL cross-dataset pooling (embedding-consistency-qc, MMD,
linear classifiers) silently found zero embeddings. Add the missing EMBEDDINGS_DIR
segment; fix the test_paths fixtures that encoded the pre-nesting layout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ling, temporal-std Extend the per-marker embedding-consistency QC: - split_by: partition datasets by a per-dataset obs column (e.g. microscope) into within-group + cross-group blocks; degenerate blocks (single-dataset within, or empty cross side) are skipped with a log line. - metrics knob: choose any of pearson/mmd/frechet (default all three). MMD pass is skipped when not requested. - Fréchet (mean+covariance) matrix: 2-Wasserstein between Gaussian summaries of each dataset's control cells — richer than Pearson-of-means, cheaper than full MMD. - Time-aware Pearson (pearson_hpi_bin_hours): mean per hours_post_perturbation bin then mean-of-means, so uneven time sampling can't masquerade as a batch effect; plus a per-dataset temporal-std drift diagnostic (std of the control centroid across HPI bins). - Annotated, auto-scaled correlation heatmaps. Organelle-box recipe (metrics: [pearson], split_by: microscope, 2 h HPI bins). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tions; build-collection skill Add per-experiment microscope (mantis_v1 for the 0.1494 um/px dataset, mantis_v2 for the 0.1133 ones) to organelle_remodeling so obs enrichment writes it and the consistency QC can split v1/v2. Normalize the 2024_11_07 control label mock→uninfected in both organelle_remodeling and the source organelle-box collection so a single uninfected filter catches every dataset's controls. Update the airtable-build-collection skill to always populate microscope (field table, schema, derivation rule). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
plot_corr_matrix auto-scaled vmin to each matrix's own off-diagonal minimum, so
heatmaps for different markers used different colour scales and couldn't be
compared by eye. Use a shared fixed vmin (default 0.8). Fix the title
("mean-embedding" -> "control-cell summary correlation").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add control_normalization.py: robust z-score of embeddings against the control (uninfected) population of the same plate and HPI bin. Stores only per-(plate, bin) median/IQR statistics and applies (x - median) / IQR on the fly rather than materializing a normalized array. - control_reference_stats: per-experiment median + IQR of control cells, pooled across control wells, in fixed-width HPI bins anchored at 0h. - control_reference_stats_multi: same at several bin widths (default 1h + 2h). - ControlReference.apply: robust z-score against the matched bin, nearest occupied bin as fallback; zero-IQR dimensions floored to 1.0. - Raises if a plate has cells but no controls (no silent drop). 8 unit tests cover recovery of the control center, control/perturbed separation, nearest-bin fallback, zero-IQR flooring, multi-bin widths, and the control-less-plate guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Standalone README next to control_normalization.py documenting the empirical finding so a teammate knows when NOT to use it: it flattens control temporal drift (−32 to −53% control-centroid std across 3 plates) but LOWERS held-out infection/remodeling classifier AUROC on every ZIKV DynaCLR classifier tested (−0.008 to −0.032), because flattening control-relative time removes infection-trajectory signal. Per-dimension only (not multivariate batch correction). Full write-up under .ed_planning/dynaclr/batch_correction/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ssifiers Add a control_normalize flag (+ control_normalize_bin_hours, default 2.0) to LinearClassifiersStepConfig and wire it into run-linear-classifiers: when set, embeddings are robustly z-scored per plate/HPI-bin against control cells before training/scoring, via viscy_utils.evaluation.control_normalization. Off by default and backward-compatible. It is off because the held-out A/B showed it lowers infection/remodeling AUROC (see control_normalization_README); the flag exists to reproduce that result and for cases where matched-time cross-condition comparability is the actual goal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This reverts commit 13d0547.
edyoshikun
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Staging integration branch for the DynaCLR model work. Aggregates three feature branches by merge:
Integration order: #478 (already present) → latest #424 → #493. The merges completed without conflicts; Git applied #493 on top of its shared #424 ancestry.
Local focused tests passed for the temporal predictor/loss and the touched DynaCLR dataset, datamodule, engine, and witness-label paths. GitHub CI was triggered by the updated branch.