Add example7: negative growth rates and the zero boundary (#193) - #243
Conversation
Adds a subsection to example1's Censoring section on the cost of saturation: once the curve is well below the bound the likelihood is flat, so a lower asymptote expressed only in that region is unidentified and its posterior is the prior. Worked contrast from a microalgal growth test where the population fell below the counting limit -- substitution at the bound gives a posterior SD of 0.014, left-censoring widens it 160-fold to a value the prior chose, and interval censoring against an extinction floor restores a usable estimate. Points at prior-to-posterior contraction as the diagnostic an interval hides. Replaces example6's statement that cens() is unavailable, which dev implements, and fixes a doubled roxygen marker in nsec.R.
Re-checked every number against the rebuilt analysis/phase3_parameters.csv, phase3_diagnostics.csv and phase4_bot_contraction.csv. The bot posteriors, contractions, LOD arithmetic and the 160-fold widening all reproduce. Three did not: - the substituted interval is a tenth the width of the interval-censored one (0.056 vs 0.587), not a twentieth; - arm C reports 5 divergent transitions in 8000 draws, not none -- which still makes the point, since nothing there would stop a routine check; - the interval-censored fit needed a longer warmup and smaller step size to converge, so it did not vary only the censoring declaration. Disclosed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A vignette on what the common conventions for handling negative growth rates do to the reported toxicity estimates, written up as a study rather than as a summary. Covers: why the ErCx subscript comes from OECD TG 201 and how the absolute ECx places 100% effect at zero response; the NSEC definition and why it depends on the width of the control posterior; the six approaches with the bnec() code for each and why nec4param alone rather than a model set; the twelve simulation scenarios with their true parameters and a panel figure of each; and the results. Two findings the earlier internal summary had wrong. NSEC is not merely unaffected-but-unscoreable: as the noise falls, A, C and D converge on the true nec of 1.3 while B1, B2 and B3 level off at 1.91, 1.99 and 3.04, overestimating it by 47%, 53% and 134%. ErC50 is biased low by the same conventions, so the two endpoints are distorted in opposite directions and the NSEC direction is the non-conservative one. Separately, the prior sensitivity check was only ever run on arm A; run across every approach with a free plateau it shows bot moving 4.6-5.1x under A, C and D but only 1.2x under B1 -- prior-insensitivity there being a symptom of the substituted zeros, not of a well-estimated parameter. Reported endpoints move at most 7.8% in any approach. Also drops two items that were Claude-derived rather than substantive: the "untestable hypothesis" framing, and a caveat that real datasets cannot show bias, which is true of all case studies and not worth stating. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Title is now "Modelling growth data and other potentially negative response values". Terminology. ECx and NSEC are toxicity estimates, not endpoints. In OECD TG 201 the endpoint is the measured response -- growth rate and yield -- and the subscript in ErC50 names which endpoint the estimate came from. Corrected throughout, including the data column, and the distinction is now stated where the notation is introduced. Structure. Headings restructured into Overview / Background / Methods / Results / Discussion / References, matching examples 1-4, and the essayistic headings replaced with descriptive ones. Prior sensitivity moved from Discussion into Results. Corrections raised in review: - dropped the paragraph on the ecx(type = "absolute") guard, since the documented and implemented behaviour of the type argument currently disagree (issue #195) and that text would go stale. The as-used model set is now recorded explicitly instead. - fixed the nec4param rationale, which implied bot anchors the absolute ECx scale. It does not: type = "absolute" runs from the predicted control response to zero and never involves bot. bot matters only because constraining it moves nec and beta, and the ECx is read off the resulting curve. - nec3param is algebraically identical to nec4param with a constant(0) prior on bot, so B2/B3 fit the same model rather than an approximation to it, and a direct brms specification would give the same posterior. Both now stated. - the case studies fix nec4param for comparability, which is a limitation rather than a design choice; unlike the simulation there is no generating model to justify it. Said so, and flagged re-running them under the model-averaged workflow. - corrected the divergence attribution. Flooring does not cause sampling problems (B1 0.06, B3 0.03 divergences per fit); pinning the asymptote does (B2 5.67). That the floored fits report nothing amiss is what makes them dangerous. - NSEC is not "often the value actually reported"; it is a recommended estimate under ANZG guidance. Cites Warne et al. 2025 (added to the bib), fisherfox2023 and fisher2023ieam. - \@ref() cross-reference removed: html_vignette is not bookdown, so it rendered literally. - concentration axes on both curve figures are now log scale, with controls dropped and that stated. - recorded why 240 iterations (MCSE on coverage) and that 500 is the usual choice for work of this kind. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds arms E (floor, scale by max, Beta, nec3param) and F (floor, no scaling, Gamma, nec3param) to the methods section, with runnable bnec() calls. Framing moves from six approaches to eight in three groups: measurement retained, boundary imposed explicitly, boundary imposed by the family's support. States why E and F stand outside the shared-prior scheme (taking bnec()'s defaults is the practice under examination), that check_data() performs the boundary nudge and is deliberately not pre-empted, and that dividing by the maximum cannot move an absolute ECx because both sides of f(x) = max(f) * (1 - x/100) carry the scaling. Results for these arms are not yet included; the sweep is running. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extends the results to arms E and F now that phase 7 stage 1 is complete. The palette is eight validated colours, the two-panel split is three, and the exclusion, divergence, precision and NSEC tables all cover E and F. Adds a section on what the two family-floored approaches do, including the F-versus-B3 contrast that isolates the likelihood: the displacement is in nec, not beta, which rules out the variance-structure explanation an earlier draft was heading toward. Why it reverses with delta is stated as unexplained rather than guessed at. The case studies stay at six approaches: on real data there is no true value to score E and F against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Regenerates the three embedded result blocks from the completed sweep and updates every figure quoted in prose. Nothing in the argument changes: the approach ordering is the same, and the coverage MCSE is now about 1.0 percentage point rather than 1.4, so the paragraph apologising for 240 iterations is gone. E's record is now stronger and worth stating exactly: not one divergent transition in any of its 6,000 fits, while carrying the largest ErC50 bias in the study at the highest precision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Everything the vignette reported until now fixed one equation in advance. That is right for scoring a convention -- the generating model is the one fitted -- but it is not the workflow bayesnec recommends, and the eight-curve figure suggests why it might matter: the constrained curve cannot follow the data, so it flattens its descent and moves nec, and the estimates follow. A separate sweep tested it: five approaches averaged over the declining candidate set, three scenarios, 200 simulated datasets each, paired to the single-model results by seed. No approach ever lost its whole candidate set, so nothing was excluded from any arm. ErC50 is rescued everywhere (B3 -10.2 -> -3.0%, E -14.8 -> -5.5%, F -15.7 -> +0.9%), with RMSE falling alongside coverage and intervals widening only 1.1-1.6x -- accuracy, not width. The NSEC divides on precision: repaired at realistic noise, reversed at high precision, where B3 reaches -62.7% with coverage 0.01 and worse RMSE. That is this vignette's own organising principle applied to the workflow rather than the convention -- averaging removes the estimation component of the bias and exposes the misspecification underneath. ErC10 is where averaging can cost you, and not only for the floored arms: arm A goes +19.0 -> -11.3% at realistic noise. Also: the "no model averaging was performed" paragraph now reads as a tested decision; the "does the NSEC escape? no, emphatically" passage is qualified as a property of the single-model analysis; and the Overview separates "do not floor" from "do not fix one equation", which are now independently supported. Numbers are generated, never typed -- negative-sgr's analysis/phase10_vignette_numbers.R emits every figure the section quotes, and an HTML comment in the section says so. Not for dev yet. The averaging results depend on bayesnec #216 (model-averaged ecx()/nsec() resample with an unseeded sample()); on dev as it stands those numbers are not reproducible between calls. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The model-averaging section called the weights "stacking". They are pseudo-BMA with Bayesian bootstrap -- bnec()'s default via define_loo_controls(), which this study never overrode. Corrected, and now stated rather than assumed, because the choice changes what a near-zero weight means: stacking zeroes redundant models as readily as wrong ones, while pseudo-BMA weights track expected log predictive density, so nec4param at a thousandth of the weight is evidence that it fits far worse rather than that another shape covers it. Also notes that re-running with method = "stacking" will move the weights and that they should not then be compared with these. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Speculative advice about a method the reader has not chosen, in a section that is about what the results mean rather than how to vary them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The results were transcribed into the vignette as literals, with their provenance given only as a code comment naming a path outside this repository. A reader could not tell where a number came from, and nothing recorded which state of the study produced it. Adds a "Where these numbers come from" section to the simulation design: open-AIMS/negative-sgr, pinned at commit 0181d66e, with the specific tables a reader can regenerate and check against -- phase5_metrics.csv for the eight-approach comparison, phase10_metrics.csv for the model-averaging section. States the two things the code on the page cannot show: that the figures are frozen at that commit and describe bayesnec as it behaved there, and that the compendium rather than this vignette is the citable record. The Phase 10 comment now names the same pinned commit and says to move the pin when the sweep is re-run. Bib entry carries the URL and the commit in howpublished, where the CSL style renders it; a note field is dropped. Prose applied identically to the rendered .Rmd, which is checked to match the .orig outside chunk bodies. precompile.R was not run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…4param case Headings were conversational where the published vignettes use noun phrases. Renamed to match that baseline: "The eight approaches" -> "Zero-handling approaches", "Where these numbers come from" -> "Provenance of the reported results", "The twelve scenarios" -> "Simulation scenarios", "What the two family-floored approaches actually do" -> "Mechanism of the family-floored approaches", "Does model averaging rescue the floored approaches?" -> "Model averaging", "What to take from this" -> "Interpretation", and in example1 "What saturation costs" -> "Limits of the censored likelihood". The two cross-references to renamed sections were updated with them. The case for a fixed functional form was made three times -- once under "Why nec4param and not a model-averaged set", again under "The model set actually fitted", and a third time in a paragraph restating that it had been tested. It also read as a refusal to model-average when a later section does exactly that. Now stated once, with a forward reference to the "Model averaging" section, and the second block reduced to what is additive: which model each approach uses, and the #195 caution for anyone reproducing it. Bold lead-ins normalised from "Why ..." to declarative noun phrases, and the "worth stating / worth being explicit / read in order" constructions removed. Prose applied identically to the rendered .Rmd; both render clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Independent reviewRead the vignette in full rather than the PR description, re-derived the simulation design from the vignette's own code, cross-checked every quoted percentage against the transcribed literals, and verified the package-behaviour claims against Verdict: the argument is sound and the transcription is clean. Two things block merge, and both are about the branch this lands on rather than about the science. Blocking1. B2 and B3 do not run on The pinned-asymptote arms are shown as: prior_fixed_bot$prior[prior_fixed_bot$nlpar == "bot"] <- "constant(0)"
fit_B2 <- bnec(y ~ crf(x, "nec4param"), data = dat, ..., prior = prior_fixed_bot)
Either state is incorrect, so this needs deciding rather than deferring: merge #244 first and drop the note, or add the 2. " From Limitations:
Scope: this closes #193 but doesn't do what #193 askedA maintainer decision rather than a fix. #193's plan specified the shipped What is delivered is arguably stronger — a scored simulation beats an unscoreable case study — but closing #193 leaves Numeric and factual issues
Figures
# the actual dataset these eight models were fitted to, rather than a fresh draw,
# so that the points shown are the points the curves were estimated fromTrue for A, B2 and (in effect) C only. The caption's "open points are the nine negative values that the conventions act on" is defensible as a display choice, but a reader will read B1/B3/E/F as failing to fit their own data. Say that the intact measurements are shown throughout for comparison, and fix the comment. All eight curves are drawn from posterior-mean parameters, not the posterior mean curve. For a nonlinear model those differ, and most for B2, whose posterior is widest. Disclosed indirectly ("Posterior means from the eight fits") but worth one clause in the caption.
Housekeeping
Verified correctWorth recording what held up:
Recommendation: approve on substance. Resolve (1) by settling merge order with #244, correct (2) — it is a factual claim about the package in the package's own docs — and take the numeric fixes. (3) is a call for the maintainer, but if #193 closes here, the dispersion demonstration and 🤖 AI-assisted review generated with Claude Code |
|
Note from the #246 review, for whichever of these lands second. #246 makes a Once #246 is in 🤖 Generated with Claude Code |
Merges dev, which carries #244, so the constant(0) prior route the pinned-asymptote arms use now works through bnec() without a hand-written init list. The practical note saying otherwise is replaced. The distributional-sigma limitation said bnec() has no route to one. It does -- disp("loglinear"), which is the form a response crossing zero gets since disp("power") is refused there. Reworded, and the claim that the misspecification cannot change the ordering is narrowed: it holds for ErC50, which is mean structure, but the account of why B1 sits below the intact analysis at high noise runs entirely through a global sigma, and that channel closes under a dispersion sub-model. Numbers: the NSEC bullet quoted 37-134% "relative to the intact analysis" but 134% is against the true nec, so both baselines are now given; B1's ErC50 coverage at the finest precision is 0.002, not zero; the model-averaging "before" column is the paired n = 200 recomputation rather than the n = 500 figure quoted earlier, which is now stated because the section invites the cross-check. check_data() messages the zero shift under Gamma but not under Beta, so in approach E the last step of the flooring is silent -- which is the argument, not an exception to it. The model-averaging preamble borrowed B2's diagnosis for B3/E/F, whose data are floored and whose curves fit them comfortably; that is the point about a silent failure. example-curves plotted one intact copy of the data in all eight panels, including the four fitted to floored values. Each panel now shows what its own fit was given, with D's truncation recovered from A's stored posterior rather than hard-coded, and the caption records that the curves are drawn from posterior mean parameters. example1's bot table now names its source: alga, r_salina under contaminant A, which is where the 3871 cells/mL comes from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Correction to the review above, and one real findingTwo things in my earlier comment need withdrawing, and one narrower finding survives. Withdrawn: "
|
…dies onto alga
The Limitations section said bnec() had no route to a distributional sigma
and that the misspecification could not change the ordering. Both were
wrong. disp("loglinear") exists -- it is the form a response crossing zero
gets, since disp("power") is refused there -- and the ordering is
dispersion-dependent for one arm.
Checked on scenario 8: 50 datasets, six Gaussian arms, fitted twice each
and paired. The homoscedastic half reproduces the compendium's scenario-8
contrasts before anything else is read from it. Result splits in two.
Flooring is unaffected -- B1 -9.0 to -8.5%, B3 -10.7 to -9.8%, coverage
still 0.50 and 0.44 -- so the case against it stands on its own. B2 is
largely exonerated: bias -12.7 to -1.2%, NSEC 1.49 to 1.06 times the
intact analysis, divergent transitions 3.22 to 0.12 per fit. All three of
its symptoms had one cause, and it was the pooled sigma rather than the
pinned asymptote. The B1/B3 NSEC contrast flips with it, 1.01 to 1.21 and
1.05 to 1.28, closing the compression channel exactly as the mechanism
predicts. Unanticipated: floored data and a dispersion sub-model sample
badly together, B3 at 13.8 divergences per fit against none homoscedastic.
Recorded in notes/, following disp_validation.md and the normalisation
sweep: the note carries the design and the table, the script and the 600
rows sit beside it. The compendium stays as it is -- it is the record of
that study, not of this check.
Case studies now fit alga live rather than quoting the compendium. All
four cells reproduce exactly and 35 of 41 fitted ratios land within 2% of
the published values, which is what makes the rework safe. Arm C needed
defining properly on data with a counting limit: a below-limit row is
censored at the limit, not at zero, because that is the tighter statement
and the one the test actually supports. With that, the four outliers
resolve to the published values exactly. Fits are seeded, per example1.
vignettes/example7.Rmd is stale for the case-study section until
precompile runs; Results and Discussion are hand-synced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The #242 checker printed hits[1], which for a long message is the bare "#> Error:" knitr emits before wrapping the text onto the following #> lines. The precompile/example7 run failed with exactly that and the log said nothing about the cause. Carry the continuation lines through, and say how many chunks errored rather than leaving the count ambiguous. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The workflow added in #215 has never successfully precompiled a vignette with fitting chunks -- example7 was the first to select one and it failed in 14 seconds, all 24 bnec() calls dying with "Boost not found; call install.packages('BH')". BH, RcppEigen and StanHeaders are LinkingTo dependencies of rstan: installing rstan's binary does not pull them, and precompiling compiles a Stan model at run time, so it needs the headers. R-CMD-check has carried BH since 2021 for exactly this reason. This workflow was written later and did not inherit it. Found because the error was legible -- the previous commit stopped the checker truncating the message to a bare "#> Error:". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rendered vignette now comes from a real precompile against the branch package (2.1.3.20 in a scratch library, not the stale 2.1.3.7 installed), so the case-study section reports the fits the source describes. Three things the first precompile turned up, none of which any automated check catches -- it exited clean with no errored chunks and would have shipped all three. Stan writes chain progress to stdout, which knitr captures: 2304 lines of "Chain 2: Gradient evaluation took ..." more than doubled the vignette. example1's fitting chunks carry results = "hide" for exactly this. The real-ratios figure was destroyed by going live. The transcribed B2 lower bound was exactly 0, which log10 maps to -Inf and ggplot drops; a fitted one is tiny but positive, so the axis honestly spanned four decades and compressed every estimate onto the reference line. Those estimates are already flagged unusable, so the bar is clipped to the panel and the open symbol carries the meaning. The prose still quoted the compendium while the figure came from the fit. All eleven figures are now computed in a helper chunk and interpolated, so a refit cannot leave the text describing an earlier one. They land close to the transcribed values -- 0.74 to 0.75, 0.43 to 0.43, 0.26 to 0.25, 2.9 to 2.7, 116 to 116 -- which is a further check on the transcription. Also drops six vignette-fig-*.png committed at the repo root by bb97754. Regenerating a figure knitted from the root and git add -A swept up the strays; .Rbuildignore has no rule for them, so they would have shipped in the tarball. precompile.R's unlink() removes them, as it always did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # DESCRIPTION # NEWS.md
precompile.R writes vignette-fig-*.png to the root, moves them into vignettes/ and deletes them, so they are transient. An ad-hoc knit of a single chunk leaves them behind, and git add -A then commits six stray copies of files that already live in vignettes/ -- which is what bb97754 did. .Rbuildignore has no rule for them, so they would have shipped in the tarball. Anchored to the root so the tracked vignettes/ copies are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Handoff — state at
|
Two of the three open items are dischargedChecked while working through the queue; the third is still RF's. 1. CI is verified. The handoff note records R-CMD-check and pkgdown as 2. The follow-up issue is opened: #283, covering the dispersion sub-model 3. The 14,700-word length is a judgement for RF and is untouched. One thing that has changed under this branch since the handoff
The branch is 66 commits behind |
Applies the section 12 register rulings and the section 13 heading and structure rulings to the vignette. No number, estimate or claim is changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
# Conflicts: # DESCRIPTION
The Gaussian zero-bounded exclusion (#206) and the absolute-ECx refusal are gone, so the vignette can no longer argue that the defaults protect the analyst by excluding those equations. The argument is now that the curve shape and the treatment of the data are separate decisions, and only the second causes the bias. Adds arm G, which fits nec3param directly on the intact measurements, and demonstrates the retained candidate set with models(c(-Inf, Inf)) rather than asserting it. Prose cut from 11,674 to 8,199 words. No reported figure from the simulation changed. The rendered vignette is not rebuilt here; it needs a precompile against current dev. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
|
The vignette has been rewritten against the landed stack and the PR body now describes the new state rather than the old one. What moved. A new arm. G fits Two claims I had written were contradicted by the recomputed figures and are fixed: the floored analyses sit below the intact analysis on The rendered vignette is stale and is not rebuilt here. A full single-vignette precompile against a 2.1.3.31 build succeeded (341 minutes, 28 fits, no errored chunks) and was discarded because it predated those two corrections. Rebuilding belongs in a full-package precompile rather than a second single-vignette run — see the note in the body about |
Two what-clauses, one of them also second person, and two bare counts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
Numbered sections; simulation before case studies; provenance moved to the end. Arm G dropped -- B2 is the same model and now carries that role. The fixed-equation design is stated as a device for attribution rather than a model of practice, and the consequences are drawn from the model-averaged sweep instead. Adds Table 4 and Figure 6, the paired single-against-averaged Phase 10 results. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
Sections 5 to 7 fold into a single numbered simulation study; the case studies, recommendations, limitations and provenance renumber to 6 to 9. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
Matches the register of the human-written vignettes: first person plural for what the study did, impersonal for what it measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
The rewrap pass that reflowed the edited paragraphs also reflowed the YAML header, which knitr then rendered as body text. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
Names bayesnec rather than 'the package'; drops redundant framing; restates the two dispersion mechanisms in place; makes the generated sigma gradient prominent; direct labels replace the colour keys on the sweep figures; the 70-row example dataset moves to a hidden chunk. The NSEC section is framed against ErC10 rather than ErC50. Table 4 is replaced by Figures 6 and 7, which cover all three estimates and all three Phase 10 cells. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
…egime split The case studies now carry the two family-imposed approaches, which is how data of this kind have most often been analysed. Arm D is dropped there: it is defined by reference to arm A's fitted curve rather than being an independent choice, and the simulation already scores it. Twenty-eight case-study fits, against twenty-four before. The stops-short regime becomes one paragraph and Table 3 goes, which removes the two-regime framing from the rest of the section. Re-running the sweep on a cleaner design is tracked in #296. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
D cuts the concentration series at a concentration, dropping whole treatment groups; the wording invited reading it as deleting negative observations. That is a fifth practice and is not among the eight, which Section 8 now says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oaD1fLGjAyQ5eEUwFwGwy
Replaces the eight lettered arms with the six named conventions of the redesigned study, which fits each with bnec() defaults and averages over the family's declining set rather than holding the equation at nec4param. Results come from open-AIMS/negative-response-conventions at 5103be3: 7 cells x 100 realisations x 6 conventions, 4,200 model-averaged fits. Drops the fixed-equation/averaging split and the dispersion sub-model section, which the new design has no counterpart for, and the bot prior sensitivity check, which assumed a shared prior the new design does not use. Adds the model-weight result: given the measurements, averaging recovers the generating equation at 0.88 of the weight, and under flooring a zero-asymptote shape takes 0.96. The case studies now run the same six conventions over the same candidate set. vignettes/example7.Rmd is NOT regenerated by this commit: the precompile is deferred until #309 settles which initial values are accepted, and until the branch takes the new default nec prior from dev (#304). Both change the case-study fits. The superseded negative-sgr bib entry is kept alongside the new one so the currently generated vignette still resolves its citation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBmDd2uv8yuhdu8RAUM2Bo
|
The PR body has been rewritten and no longer describes the same study. The vignette is rebuilt against the redesigned simulation (#296), in which every convention is fitted with Removed from the vignette, and so from the body: the fixed-equation versus model-averaging split, the dispersion sub-model section, and the Two things a reviewer should know before reading the diff. |
The case studies move into the compendium alongside the simulation. Four datasets by six conventions, each averaged over thirteen or fourteen equations, is about a fifteen-hour precompile, which is not something to run on a workstation whenever a sentence changes. Section 6 now transcribes their results the way Section 5 already transcribes the simulation's, and the code an analyst would write is shown rather than run. What that costs is stated in Section 9 rather than left to be discovered: the case-study estimates no longer track whatever version of bayesnec they are published beside. Table 3, which describes the four datasets themselves, is still computed at build time from the alga data, so that much does. Section 8's paragraph on the ec50 prior is rewritten. It recorded that the prior placed the true ErC50 above the 99th percentile in every cell and that this was the first thing to check if the default changed. It has changed: the prior is now a lognormal on the log of the predictor, and the true ErC50 sits between the 80th and the 95th percentile. The superseded behaviour is kept as a note for anyone comparing the two versions of the study. One bug fixed. rng_ratio() and furthest() called sapply(others, rr, ds, est), which passes the element as the FIRST argument, so every lookup silently swapped the convention and the dataset and returned nothing. Caught by knitting the section against synthetic case data before the real results exist. The transcribed case block is a placeholder until the re-run finishes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBmDd2uv8yuhdu8RAUM2Bo
…nger support The transcribed blocks are regenerated from the compendium at c83c13d: 4,200 simulation units and 24 case-study units against bayesnec 25466e5, no failures. The precompile is now seconds rather than fifteen hours, because nothing is fitted. Four assertions did not survive the new numbers and are corrected rather than left standing. The Gamma was said to be within one per cent on ErC50 in every precision cell. It is +4.3% at the noisiest and converges to +0.2%, so it now reads as converging, and its ErC10 is described as getting worse rather than as failing to improve, which is what it does. Section 5.5 said the retaining conventions converge on the true nec from above, "as the definition requires". They no longer do: under the new nec prior they start on it. The overshoot the metric's definition predicts at finite noise is small enough to be lost in the Monte Carlo error of 100 realisations, so what identifies a correctly specified approach here is that it stays on the truth as the residual error falls, not the direction it arrives from. The figure caption and the coverage paragraph follow. `censored` was called the one convention holding nominal coverage for all three estimates across the sweep. `deleted` does as well. And the case studies no longer show a substantial change on every dataset. On c_proliferum every other convention returns about three-quarters of the `measured` ErC50; on the other three they are within 13%. Read straight that looks like evidence the choice is safe, so the paragraph says what a single real dataset can and cannot separate instead of implying the effect is general. Added: a table of the model weight held by equations whose tail ESS fell below 400. On r_salina more than half the weight under `measured` and `deleted` sits on such equations, and those are reference analyses rather than conventions under test, so the ratios there are the weaker evidence. The simulation has no such problem -- across all 4,200 fits the largest such figure is under half a per cent. The superseded negative-sgr citation is dropped; nothing refers to it now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBmDd2uv8yuhdu8RAUM2Bo
Section 5.3 explained the Gamma's ErC10 displacement by saying an identity-link Gamma has one coefficient of variation to cover both the control scatter and the floored block, and that the compromise falls on the shoulder. A posterior predictive check on the saved fit for the noisiest cell says that is not what happens. The Gamma does not compromise: it ties the residual SD to the mean, so against a generated SD that is constant at 0.0114 it predicts 3.5 times too much spread at the control, too little at the shoulder and none at all at the floored values -- a top-to-bottom ratio of five thousand against a true value of one. The floored block therefore takes almost all the weight in the likelihood and the shoulder is what gives way. The same check establishes something the vignette had not said: under a Gaussian family the floored data are described by a correctly specified variance model. Both `measured` and `floored` predict 0.86 to 0.93 of the generated spread at every concentration, with top-to-bottom ratios of 1.02 and 0.98. So flooring damages the mean structure and not the variance, and under a Gamma the two failures compound. This was asked for and not checked: the study recorded convergence diagnostics and no fitted objects, so the claim had stood on reasoning since it was written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBmDd2uv8yuhdu8RAUM2Bo
e33982a to
6dda3d3
Compare
… fit Section 5.3 explained the Gamma's ErC10 displacement from a posterior predictive check on a single saved fit per cell. Differences read off one realisation are inside what one realisation produces, and two of the figures were materially wrong: the control ratio is 2.51 rather than 3.5, and the control-to-floor gradient 181 rather than five thousand. The section now transcribes results/ppc_spread.csv, which summarises 100 realisations of each cell and convention, and reads its figures out of that block rather than having them typed. `measured` is correctly specified by construction, so its control ratio of 0.98 to 1.00 with 95% of realisations in [0.81, 1.18] is stated as the reference every other convention is read against -- without it there is no way to tell a departure from one that is misspecification from one that is the estimator. Two claims are added that the single fit could not support. `floored` under a Gaussian returns a control ratio of 0.91 to 1.12 and a gradient of 1.04, inside the reference, so flooring damages the mean structure and not the variance. And the Beta's control ratio is 1.08 to 1.10, also inside the reference, with a gradient of 1.7 to 2.7 -- so whatever displaces its ErC10 is not its variance model. The vignette says that rather than leaving the reader to assume the two bounded families fail the same way. The data frame is named `ppc`: `spread` is already a helper for the range of bias across the Gaussian conventions, and shadowing it broke the knit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bnec() derives its model weights with pseudo-BMA and a Bayesian bootstrap whenever loo_controls is not supplied, and the compendium never supplies it. Table 2 and its chunk comment described them as stacking weights. The method is now named where the conventions are introduced, so the claim is checkable rather than implicit. The Monte Carlo standard errors in Section 8 were typed and two had gone stale against the re-run: the NSEC median is 2.2% rather than 2.1% and the largest is 10.5% rather than 11.6%. They are read from the transcribed block now, which gains an mcse_pct column, so a re-run updates them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The body has been updated and the Status section rewritten. Two things had gone stale. The headline table still carried the figures from the first run of the compendium, before it was re-run against the current default priors — for example And the Status section said Two corrections went into the vignette in the same pass. The model weights were described as stacking weights; |
Section 8 said the sub-model is never exercised and that re-running the sweep
under disp("loglinear") is tracked in #283. Both statements are now false: the
sweep was run on the same cells and realisations, and the results are not
reported because a sub-model does not repair the conventions that impose the
zero boundary through the response distribution, and adding one to a Gaussian
gives its variance a gradient the data do not have.
The paragraph states that rather than claiming the question is untouched, and
points at the compendium, whose CLAUDE.md records why the tables are uncited.
Section 5.3 is unaffected: the posterior predictive check that identifies the
Gamma's variance misspecification is a diagnostic and remains cited.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only the NSEC changes: #325 seeks the crossing from the control upward, so a draw crossing between the control and the next grid point is no longer lost and the lower bound falls. Every ErC10, ErC50 and model weight is identical. Two corrections found while checking the prose against the new tables. Section 6.2 reported the largest mean weight on an equation failing a diagnostic as under half a per cent; that figure is the mean over all 4,200 fits, and the largest is 5.2%, in ctl under gamma. Section 9 now names the NSEC change as a third difference from the CRAN release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PafHJiefzAfJfMTs197d3P
# Conflicts: # .github/workflows/precompile-vignettes.yaml
Outcome of the re-runThe study has been re-run against the pinned Only the NSEC changed. #326 seeks the crossing from the control upward, with the
Every conclusion the vignette draws is unaltered. Corrections made while checking the proseSection 6.2 reported the largest mean weight on an equation failing a diagnostic Section 9 now names the NSEC change as a third difference from the CRAN release, EvidenceCompendium open-AIMS/negative-response-conventions Verification, and what it establishesThe regeneration of the transcribed blocksThe blocks were regenerated by script rather than by hand. The script was first The equivalence test run before the studyAll 42 fixed priors are identical between the superseded pin and the new one, so The estimation path against current
|
What
example7, a vignette on responses that can legitimately take negative values — specific growth rate above all — and on what the common practice of removing those negatives does to the toxicity estimate that gets reported.Closes #193.
Why it matters
Replacing a negative growth rate with zero is standard practice, in this package and in others, and the reasons are long-standing: a negative value cannot be expressed as a percentage of the control without exceeding 100, and several commonly used equations cannot generate a negative mean at all.
On simulated data, where the truth is known and every convention is fitted the way an analyst would fit it —
bnec()defaults, model-averaged over the family's declining candidate set — the conventions that keep the measurement recover the truth and the conventions that replace it do not. Across a precision sweep from a 9.6% to a 1.9% control coefficient of variation:measuredcensoreddeletedflooredbetagammaThe bias of a convention that replaces the measurement does not shrink as the experiment improves, so its interval narrows around a displaced centre and coverage collapses. A laboratory that adds replicates and reduces its control variability produces an analysis that is more confident and no less wrong.
gammaconverges on ErC50 and instead displaces ErC10, from -8.7% to -25.0% with coverage of 0.01, so these conventions cannot be summarised across estimates either.None of it is visible in the diagnostics. Every one of the 4,200 fits converged and returned all three estimates, and equations failing a convergence check hold 0.4% of the model weight averaged over the study, 5.2% at the worst single cell and convention.
The clearest single result is what the model weights say. Given the measurements as recorded, averaging identifies the generating equation and does so more sharply as the experiment improves —
nec4paramtakes 0.51 of the weight at a 9.6% control CV and 0.85 at 1.9%. Under flooring it takes 0.000, and the six equations whose lower asymptote is fixed at zero take 0.98. A model weight describes the data supplied to the fit, not the process that produced the measurements, and on floored data those are different questions.A posterior predictive check says where the Gamma's failure is. These data are generated with a residual standard deviation constant along the curve, so a correctly specified family should predict that value everywhere.
measureddoes, and so doesfloored. An identity-link Gamma over-states the spread at the control by a factor of 2.5 and predicts a control-to-floor gradient of 90 to 181 against a true value of one, because it ties the dispersion to the mean. So flooring damages the mean structure and not the variance; under a Gamma the two failures compound.The vignette also records why nothing in the current defaults requires the substitution. A Gaussian mean is unconstrained, the six zero-asymptote equations are retained under a Gaussian family from #206, and an absolute ECx is reported for them, so the curve shape TG 201 describes can be fitted on the measurements as recorded and weighed against the free-asymptote shapes by the same model averaging as everything else.
And it says where the question does not arise: in the cell whose concentration series stops at the zero crossing, where about two observations in seventy are negative, the four Gaussian conventions span 0.4 percentage points of ErC50 bias between them.
Evidence
Simulation and case studies: open-AIMS/negative-response-conventions. Seven cells × 100 realisations × 6 conventions = 4,200 model-averaged fits for the simulation, 24 for the case studies, run as SLURM arrays against
bayesnecat a pinneddevcommit inside the containerbayesnec's ownhpc/image.lockidentifies. No unit failed and no estimate was unidentified. Model weights arebnec()'s default, pseudo-BMA with a Bayesian bootstrap.Every figure in the vignette is transcribed from that compendium's result tables and read out of them by inline code, so a re-run changes the numbers without touching prose.
Status
Ready. The study has been re-run against the pinned
devcommit, the vignette's transcribed blocks regenerated from the new tables, and both the source and the generated vignette pushed.The re-run was 4,224 model-averaged fits with no failures. Only the NSEC changed. #326 seeks the crossing from the control upward, so a draw crossing between the control and the next grid point is no longer discarded: the point estimate changes by 0.1 to 0.5 percentage points and the lower bound by about three times that. Every ErC10, every ErC50 and all 42 rows of the model-weight table are identical to the digit, which is why
results/ppc_spread.csvwas not regenerated.Two corrections were made while checking the prose against the new tables. Section 6.2 reported the largest mean weight on an equation failing a diagnostic as under half a per cent; that figure is the mean over all 4,200 fits, and the largest for any one cell and convention is 5.2%, under
gammainctl. Section 9 now names the NSEC change as a third difference from the CRAN release, alongside the priors and the initial-value search.devhas moved since the pinned commit, and the estimation path has not:R/ecx.R,R/nsec.R,R/bnec.R,R/fit_bayesnec.R,R/inits_functions.RandR/priors.Rare all byte-identical between the pin anddevatf54e4bdc. The changes since are post-fit helpers and RNG restoration, so the merged state reproduces the numbers reported here.What changed in this rewrite, and what was removed
The study behind the vignette was redesigned (#296). The previous version held the equation fixed at
nec4paramfor every arm, which answers a question about the likelihood rather than the question a reader has, and on altered data fitted an equation that did not generate them — so part of what it measured was misspecification the convention did not cause. Every convention now callsbnec()with its own default priors and averages over its family's declining set.Consequences for the document:
measured,censored,deleted,floored,beta,gamma.deletedis new and was recorded as an untested practice in the previous version;botprior sensitivity check is removed: it tested a shared prior the new design does not use;algawith substituted rows excluded, one of the four datasets shows a gradient and the 8.1 figure was a single dose-group ratio on that one dataset;bayesnecthey are published beside.Sampler settings for the case studies match the simulation —
iter = 4000,warmup = 2000,adapt_delta = 0.99,max_treedepth = 12. Measured on the two worst-mixing fits, that takes tail ESS from 127 and 57 to 2,327 and 1,892. In the full averaged fit no equation holding any weight fails either diagnostic.A dispersion sweep was run and is deliberately not reported. It asked whether letting a family's dispersion vary along the curve repairs the bounded families. It does not, the Beta needs no such correction and is made worse by one, and adding one to a Gaussian manufactures a variance gradient the data do not have. The code and tables stay in the compendium, and its
CLAUDE.mdrecords why they are uncited.🤖 Generated with Claude Code