STAR alignment params: consider Ribo-seq-tuned defaults per sample type

[pinin4fjords]

Context

Surfaced while integrating Rp-Bp (#169). Rp-Bp's upstream pipeline calls STAR with Ribo-seq-tuned settings; ours uses a single STAR config shared between Ribo-seq and RNA-seq samples for paired analyses. Detail from Rp-Bp's src/rpbp/defaults.py:

Parameter	Rp-Bp default	nf-core/riboseq default
outFilterMismatchNmax	1	10
outFilterMismatchNoverLmax	0.04	0.3
outFilterType	BySJout	Normal
sjdbOverhang (index)	33 (max read - 1 for ~30 nt footprints)	100
seedSearchStartLmaxOverLread	0.5	unset
winAnchorMultimapNmax	100	50
alignIntronMin/Max	20 / 100000	21 / 0

outFilterMismatchNmax 1 + outFilterMismatchNoverLmax 0.04: with ~30 nt Ribo-seq footprints, our default 10/0.3 allows up to 33% of the read to mismatch. That's lax for Ribo-seq; tight is right.

sjdbOverhang 33: STAR docs recommend max_read_length - 1. Building the index with 100 means junction detection is biased toward longer overhangs than Ribo-seq reads can support.

Why this matters

• Permissive alignment passes more low-quality reads into Ribo-TISH / Ribotricer / RiboCode / Rp-Bp / PRICE, blurring their periodicity statistics.
• Specifically for Rp-Bp: Bayes-factor distributions on the same FASTQs will differ from upstream Rp-Bp because the BAMs differ. The pipeline is documented to be more permissive (see Rp-Bp section of usage.md which exposes --extra_star_align_args as the workaround), but a default that's correct out of the box is better than a documented workaround.

Why we can't just adopt Rp-Bp's settings pipeline-wide

The same STAR alignment serves both Ribo-seq and RNA-seq samples (for paired translational efficiency analysis). RNA-seq reads at 100-150 nt with outFilterMismatchNmax 1 would filter out far too many; the same STAR index with sjdbOverhang 33 would miss long-overhang junctions only RNA-seq can resolve.

Options to consider

1. Per-sample-type STAR args — apply Ribo-seq-tuned args only when meta.sample_type == 'riboseq'. Smaller blast radius. Requires teaching FASTQ_ALIGN_STAR (and the shared STAR_GENOMEGENERATE) to branch on sample type.
2. Apply Ribo-seq STAR tuning only to the second STAR pass (the --extended_orf_analysis hybrid transcriptome alignment, which is already Ribo-seq-only). Smaller change. Doesn't help default-mode Rp-Bp.
3. Two STAR indices — one with sjdbOverhang 33 for Ribo-seq, one with 100 for RNA-seq. Doubles index storage but cleanly separates concerns.
4. Stay with the current defaults + document override — what's in the modernisation PR today.

Pointers

• rpbp upstream STAR defaults: https://github.com/dieterich-lab/rp-bp/blob/master/src/rpbp/defaults.py (star_options dict, lines ~58-72)
• Where the override lives currently: --extra_star_align_args per-run override is already wired through conf/modules.config STAR_ALIGN block.
• This is not blocking the modernisation PR (#160-#171 + #170); it's a separate refinement to consider once the modernisation lands.

[pinin4fjords]

Source-verified the gap against rp-bp's own STAR defaults (rp-bp 4.0.1, src/rpbp/defaults.py) while wiring Rp-Bp in #186, so concrete targets for this issue:

The pipeline shares a single STAR_ALIGN block (conf/modules.config) across Ribo-seq and RNA-seq, so footprints are aligned with general-purpose settings. Gaps for short (~28nt) Ribo-seq reads vs rp-bp:

• Mismatch filtering (the big one): pipeline uses STAR defaults outFilterMismatchNmax 10 / outFilterMismatchNoverLmax 0.3; rp-bp uses 1 / 0.04. At 0.3, a 28nt footprint can carry ~8 mismatches and still align.
• Not set in the pipeline: outFilterType BySJout, outFilterIntronMotifs RemoveNoncanonicalUnannotated, seedSearchStartLmaxOverLread 0.5, winAnchorMultimapNmax 100, alignIntronMin 20 / alignIntronMax 100000.
• The pipeline already sets alignEndsType EndToEnd; rp-bp leaves STAR's default Local (so we are actually stricter on read ends than rp-bp there, no change needed).

Suggested design: all of the above are alignment-time params, so set them per sample type via sample_type == 'riboseq'-scoped ext.args (tighten Ribo-seq, leave RNA-seq on current defaults). sjdbOverhang 33 is the exception: rp-bp inserts the GTF/sjdb on the fly at alignment, but our index bakes the GTF in (STAR_GENOMEGENERATE), so matching it needs either on-the-fly sjdb at STAR_ALIGN or a Ribo-seq-specific index (clean only for Ribo-seq-only runs).

Blast radius: this changes every genome-BAM caller's input and all pipeline snapshots, so it needs full re-validation across the matrix. It affects Rp-Bp (#186) too; the #186 caveat points here, and --extra_star_align_args is the interim opt-in.