bidsgate: inject a known lesion or atrophy truth into BIDS data and score what a pipeline recovers

[CedricConday]

When a structural pipeline's outputs shift between releases there is usually no truth to check against (smriprep#230 is one example of a regression noticed only after the fact). I have released a small tool for that: bidsgate injects a known change into real BIDS T1w/FLAIR images and writes a normal BIDS derivative with the truth next to it, so any BIDS app runs on it unchanged, then scores what the app recovered with a one-file HTML scorecard.

Two injectors so far: lesions (ellipsoids of 30 to 1500 mm3, FLAIR-bright and T1w-dark relative to the subject's own white matter, placed inside a brain mask with a volume gate) and global atrophy (a smooth radial contraction by a known factor, default 5 percent). Scoring gives Dice, lesion-wise sensitivity by size, false positives beyond a 2 mm margin and volume ratio, or for atrophy the recovered fraction of the injected change.

First public result, on lesion segmentation rather than nipreps: LST-AI v2.0.0rc1 on six OpenNeuro ds007908 controls found 64 of 72 injected lesions (mean Dice 0.82), and seven of the eight misses were among the four lowest-placed lesions of their subject.

https://github.com/CedricConday/bidsgate (pip install bidsgate)

Nothing in it is evidence about any disease; it is a test of software. The question I would like to put to this community: would a recovery number per smriprep release be useful, and if so, what should be injected first: global atrophy, regional atrophy given a mask, or something else? I am happy to run it against a release pair and post the scorecard here if that is welcome.

[CedricConday]

The longitudinal half of the same idea is now out. mscard (https://github.com/CedricConday/mscard, pip install mscard) takes two or more visits (T1w and FLAIR, lesion masks from LST-AI or your own) and writes one page per subject: new, enlarging, shrinking and resolved lesions, slowly expanding lesion candidates, lesion volume change and brain volume change from the deformation field. Before the page is written, the subject's own baseline is turned into a synthetic follow-up with bidsgate (2 % contraction, twelve new lesions, repositioning, gain drift, noise), the same pipeline measures it, and every number on the page carries the recovery of that truth as a badge.

On the 24 usable longitudinal subjects of the public MSLesSeg dataset with expert masks: 1 false enlarging or shrinking call in 551 baseline lesions, 0 false resolved, brain-volume recovery 1.07 to 1.34 (median 1.18), and a median 1.07 SEL candidates per lesion with no true expansion. With LST-AI segmenting both visits the badges go red where they should: on P2 the segmenter's own visit-to-visit variability reads as 6 of 15 lesions changing. Gallery: https://cedricconday.github.io/mscard/. For the morphometry side the same caveat as before applies: the contraction is uniform and the calibration cannot see a skull-strip difference between visits, so the page runs a second check with the brain mask eroded 6 mm and flags rim-dominated numbers.