Add an "N/A values" setting to unify not-applicable covariate values

[vagisha]

Rationale

When a covariate column mixes ways of writing "not applicable" (na, NA, n/a, N/A, or an empty cell), Octopus treated each form as its own covariate group, with its own color and its own place in the balance. That fragments a single "not applicable" category into several groups. Separately, a genuinely blank cell and a typed "N/A" were folded together, so "not measured" and "not applicable" could not be told apart.

This work adds one global choice, shown only when the data actually mixes these forms, for which forms (na, NA, n/a, N/A, blank cell) should count as N/A, and keeps a genuinely blank cell distinct when asked.

Changes

• At upload, when any column mixes two or more N/A forms, an "N/A values" checklist appears. Every form starts checked, so all fold into one N/A group. The literal N/A entry is always checked and cannot be unchecked. The choice is global and applies to every column.
• A blank cell is no longer automatically treated as N/A for covariate key generation. By default it forms its own group displayed with no value. Checking its box in the list merges it into N/A.
• Display follows one rule.
  • Group views (the Covariate Summary, the Plate Details modal, and the Excel legend) show the folded N/A label.
  • Per-sample views (plate cells, the hover tooltip, the placement CSV, and the Sample Details sheet) show the raw
    typed value, so a downloaded file always matches the original user input.
• The choice is saved in the layout file, so reloading a saved layout reproduces the same groups. A layout must record a color for every group, so saving without colors is refused, and a loaded layout is rejected if its groups and colors do not match exactly.
• Changing any checkbox invalidates the current layout, the same as changing the selected covariates, so the user has to regenerate the layout.
• Also fixed the full-size plate view to show a covariate row even when its value is blank, and QC samples now show their QC value in both plate views when the QC column is not a covariate.

Data with none of these forms in any covariate groups behaves exactly as before.

Tests

• utils.test.ts, covariateKey.properties.test.ts - key builder under the
  default and folded policies, plus a property test over blank, na, NA, n/a,
  N/A, the delimiter, the escape character, and the missing marker
• naDetection.test.ts - all-column detection: exact per-column forms,
  whitespace treated as blank, single-form columns not flagged
• naPolicyRoundTrip.test.ts - all 16 checklist combinations, each checked
  against an independent reference for the groups and for an exact
  save/parse/validate/load round trip, plus a group-without-color rejection
• layoutIO.test.ts - policy persisted and validated, colors required on save
  and load, groups and colors must match exactly
• ConfigurationForm.test.tsx - checklist render, disabled N/A, uncheck updates
  the policy
• PlateDetailsModal.test.tsx, excelExport.test.ts - folded group shows N/A,
  missing group shows blank, per-sample views show the raw value
• e2e/na-value-handling.spec.ts - save a folded-with-distinct-blank layout,
  reload, and confirm the same group count with the blank box restored
• Full run clean: tsc, eslint, 321 unit tests, 51 Playwright tests, prod build

Manual test file: test-data/na-value-handling-demo.csv (Dose mixes na/NA/n/a/N/A and blank, so the checklist appears).

Co-Authored-By: Claude noreply@anthropic.com

[Copilot]

Pull request overview

This PR introduces a global “N/A values” policy to unify not-applicable covariate spellings (e.g., na, NA, n/a, N/A) into a single canonical group when desired, while allowing genuinely blank cells to remain distinct unless explicitly folded. It threads this policy through covariate key generation, group-level display (legend/summary/modal/Excel legend), layout persistence/validation, and updates plate views to consistently show blank covariate rows and QC values.

Changes:

• Add NaPolicy + detection utilities, and route covariate key/group display through a single “effective value” policy.
• Persist/validate the policy in layout files and tighten layout save/load requirements (notably covariate colors).
• Update UI and exports to apply folded-vs-raw display rules, with tests covering detection, grouping, and round-trips.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test-data/na-value-handling-demo.csv	Adds a demo dataset that mixes N/A spellings and blanks to exercise the new checklist behavior.
src/utils/utils.ts	Implements NA detection, policy-driven effective values, missing marker, and policy-aware covariate key generation.
src/utils/types.ts	Adds NaPolicy and DEFAULT_NA_POLICY, threads naPolicy into CovariateConfig.
src/utils/layoutIO.ts	Persists naPolicy in layout settings; makes covariateColors required and validates exact color/group correspondence.
src/utils/excelExport.ts	Applies policy-driven display for legend/group rollups while keeping per-sample views raw/blank.
src/tests/utils.test.ts	Adds unit coverage for missing marker behavior and effectiveValue/effectiveDisplayValue.
src/tests/PlateDetailsModal.test.tsx	Verifies modal rendering for folded spellings vs distinct blank under default policy.
src/tests/naDetection.test.ts	Adds tests for all-column NA-type detection and ambiguity rules.
src/tests/layoutRoundTrip.properties.test.ts	Ensures generated settings include naPolicy for layout/property scenarios.
src/tests/layoutIO.test.ts	Updates layout IO tests for required colors and policy persistence/validation.
src/tests/excelExport.test.ts	Updates Excel legend expectations for default blank rendering and folded policies.
src/tests/distributeToBlocks.test.ts	Updates group key expectations to reflect the missing marker under default policy.
src/tests/covariateKey.properties.test.ts	Expands property testing across multiple NA policies and updates the clean-data preservation property.
src/tests/ConfigurationForm.test.tsx	Adds component tests for the “N/A values” checklist rendering and toggling semantics.
src/hooks/useRandomization.ts	Threads naPolicy through randomization calls.
src/hooks/useCovariateColors.ts	Uses policy-driven display values in group summary labeling.
src/components/PlatesGrid.tsx	Passes qcColumn through to plate rendering to support QC display in both views.
src/components/PlateDetailsModal.tsx	Uses effectiveDisplayValue for group rollup rendering, parameterized by naPolicy.
src/components/Plate.tsx	Shows QC value when QC column isn’t a covariate; ensures blank covariate rows render in full-size view; tooltip uses raw values.
src/components/ConfigurationForm.tsx	Adds “N/A values” checklist UI controlled by naDetection + naPolicy.
src/App.tsx	Adds naPolicy state, initializes/toggles it, threads it through processing/export/layout save/load, and drives checklist via naDetection.
src/algorithms/repeatedMeasuresDistribution.ts	Ensures group-aware randomization key generation honors naPolicy.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated no new comments.