Skip to content

feat(abtest): hypothesis metadata for pre-registration (Change Set A) - #46

Merged
imjlk merged 20 commits into
mainfrom
feat/abtest-advanced-experimentation
Jul 25, 2026
Merged

feat(abtest): hypothesis metadata for pre-registration (Change Set A)#46
imjlk merged 20 commits into
mainfrom
feat/abtest-advanced-experimentation

Conversation

@imjlk

@imjlk imjlk commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Summary

Change Set A from ABTEST_ADVANCED_EXPERIMENTATION_FOLLOWUP.md — hypothesis pre-registration for A/B tests.

What changed

  • HypothesisMetadata domain module (hypothesis.ts): structured objective, primary metric, expected lift (relative/absolute), owner, experiment scope (family key, attribution/exclusion windows). Canonical SHA-256 checksum + locking for pre-registration integrity.
  • AbTest type extensions: optional hypothesis and assignmentProvenance fields.
  • Schema/validator: assignmentProvenance in abTestSchema and persistence validator.
  • Graph anchor: hypothesis tests (197 paths).

Not in this PR

  • Stratified split, collision guard, preview/seed send, decision snapshots — subsequent change sets B-E.
  • Hypothesis wiring into createTest/launchAbTest input — next commit.

Checklist

  • format, check (197 paths), build, test (200 abtest + 46 full)
  • Sampo changeset (@listmonk-ops/abtest minor)

Reviewer notes

  • @codex — please review the HypothesisMetadata type design, checksum canonicalization, validation rules (strict vs draft), and experimentFamilyKey format.

Summary by CodeRabbit

  • New Features
    • Added optional pre-registered A/B test hypothesis metadata with deterministic checksum locking and verification.
    • Added assignment provenance to reflect whether assignments are manifest-bound or unavailable.
    • Added opt-in recipient-domain/provider stratification to compute and persist a constrained quota matrix for holdout.
    • Updated the CLI (including interactive flow) to accept --hypothesis and --enable-stratification.
  • Bug Fixes
    • Strengthened load-time validation for persisted hypothesis and stratification data, including checksum tamper detection.
  • Documentation
    • Expanded A/B test docs (EN/KO) covering hypothesis locking and stratification usage.
  • Compatibility
    • Existing A/B tests continue to work since all additions are optional.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant