-
Notifications
You must be signed in to change notification settings - Fork 0
fix(abtest): resolve cross-PR review findings #51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
780b0a6
8be2fd0
174e52a
43c60c3
9f2ef54
596750c
76d61b1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| npm/@listmonk-ops/abtest: patch | ||
| --- | ||
|
|
||
| Fix unresolved review findings across Change Sets A-E: deep-clone locked hypotheses to prevent caller-mutation checksum invalidation, include revenue columns in reports when revenue_per_recipient is the primary metric, reject missing group keys in stratification, and remove dead duplicate validation. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,6 +46,8 @@ export interface ExperimentReport { | |
| clickRate: number; | ||
| conversionRate: number; | ||
| openRate: number; | ||
| revenue?: number; | ||
| revenuePerRecipient?: number; | ||
|
Comment on lines
+49
to
+50
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The new report fields carry monetary totals and per-recipient values without their currency, even though conversion events require an ISO 4217 currency and Useful? React with 👍 / 👎. |
||
| }>; | ||
| srmPassed?: boolean; | ||
| srmPValue?: number; | ||
|
|
@@ -116,6 +118,11 @@ export function buildExperimentReport( | |
| clickRate: r.clickRate, | ||
| conversionRate: r.conversionRate, | ||
| openRate: r.openRate, | ||
| revenue: r.revenue, | ||
| revenuePerRecipient: | ||
| r.revenue !== undefined && r.sampleSize > 0 | ||
| ? r.revenue / r.sampleSize | ||
| : undefined, | ||
|
Comment on lines
+123
to
+127
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When revenue-bearing results pass through the public Useful? React with 👍 / 👎. |
||
| }; | ||
| }); | ||
|
|
||
|
|
@@ -258,12 +265,34 @@ export function reportToMarkdown(report: ExperimentReport): string { | |
|
|
||
| lines.push("## Variant Results"); | ||
| lines.push(""); | ||
| lines.push("| Variant | Sample | Open Rate | Click Rate | Conversion Rate |"); | ||
| lines.push("|---------|--------|-----------|------------|-----------------|"); | ||
| const hasRevenue = report.variants.some((v) => v.revenue !== undefined); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When an exported Useful? React with 👍 / 👎. |
||
| const headers = [ | ||
| "Variant", | ||
| "Sample", | ||
| "Open Rate", | ||
| "Click Rate", | ||
| "Conversion Rate", | ||
| ]; | ||
| if (hasRevenue) { | ||
| headers.push("Revenue", "Rev/Recipient"); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This adds user-visible revenue columns to the Markdown experiment report, but neither AGENTS.md reference: AGENTS.md:L235-L237 Useful? React with 👍 / 👎. |
||
| } | ||
| lines.push(`| ${headers.join(" | ")} |`); | ||
| lines.push(`|${headers.map(() => "---------").join("|")}|`); | ||
| for (const v of report.variants) { | ||
| lines.push( | ||
| `| ${v.variantName} | ${v.sampleSize} | ${v.openRate.toFixed(2)}% | ${v.clickRate.toFixed(2)}% | ${v.conversionRate.toFixed(2)}% |`, | ||
| ); | ||
| const cells = [ | ||
| v.variantName, | ||
| String(v.sampleSize), | ||
| `${v.openRate.toFixed(2)}%`, | ||
| `${v.clickRate.toFixed(2)}%`, | ||
| `${v.conversionRate.toFixed(2)}%`, | ||
| ]; | ||
| if (hasRevenue) { | ||
| cells.push( | ||
| v.revenue?.toFixed(2) ?? "N/A", | ||
| v.revenuePerRecipient?.toFixed(4) ?? "N/A", | ||
| ); | ||
| } | ||
| lines.push(`| ${cells.join(" | ")} |`); | ||
| } | ||
| lines.push(""); | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -219,7 +219,12 @@ export function computeStratifiedQuotas(params: { | |||||||||||||||||
| []; | ||||||||||||||||||
|
|
||||||||||||||||||
| for (const [index, groupKey] of groupOrder.entries()) { | ||||||||||||||||||
| const exactCount = groupExactCounts[groupKey] ?? 0; | ||||||||||||||||||
| const exactCount = groupExactCounts[groupKey]; | ||||||||||||||||||
| if (exactCount === undefined) { | ||||||||||||||||||
|
Comment on lines
+222
to
+223
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win Check for an own property, not just an undefined lookup.
Proposed fix for (const [index, groupKey] of groupOrder.entries()) {
+ if (!Object.prototype.hasOwnProperty.call(groupExactCounts, groupKey)) {
+ throw new Error(
+ `Stratified quota invariant: group "${groupKey}" is in groupOrder but missing from groupExactCounts`,
+ );
+ }
const exactCount = groupExactCounts[groupKey];
- if (exactCount === undefined) {
- throw new Error(
- `Stratified quota invariant: group "${groupKey}" is in groupOrder but missing from groupExactCounts`,
- );
- }📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||
| throw new Error( | ||||||||||||||||||
| `Stratified quota invariant: group "${groupKey}" is in groupOrder but missing from groupExactCounts`, | ||||||||||||||||||
| ); | ||||||||||||||||||
| } | ||||||||||||||||||
| const ideal = (stratumSize * exactCount) / totalAudience; | ||||||||||||||||||
| ideals.push({ groupKey, ideal, index }); | ||||||||||||||||||
| rowQuotas[groupKey] = Math.floor(ideal); | ||||||||||||||||||
|
|
||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Return defensive copies of stored tests.
This clone detaches
config.hypothesis, but Lines 355-356 store and return the same mutableabTest;getTest,getAllTests, andsnapshotTestsalso expose internal references. A caller can mutate the returnedhypothesisand invalidate its checksum after creation. Return deep defensive copies, or encapsulate/freeze stored state, at every service boundary and add a regression test for this mutation path.🤖 Prompt for AI Agents