Skip to content

surject changes: support paired diploid placements - #5055

Open
Sagorikanag wants to merge 2 commits into
surject-stack-04-diploid-long-readfrom
surject-stack-05-diploid-paired
Open

Sagorikanag wants to merge 2 commits into
surject-stack-04-diploid-long-readfrom
surject-stack-05-diploid-paired

Conversation

@Sagorikanag

@Sagorikanag Sagorikanag commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Changelog Entry

  • vg surject --diploid-map can jointly surject grouped paired-end placements, using paired alignment evidence and an optional or learned fragment-length model.

Description

This extends diploid surjection to paired-end reads with multiple graph placements per fragment.

For --diploid-map -i, consecutive interleaved placement pairs for one fragment are processed together. Each alternative remains tied to its original mate pair, including supplementary alignments, so mate fields and SA links are rebuilt within the correct candidate.

Paired selection:

  • generates unselected surjection candidates for each mate
  • keeps supplementary pieces attached to their principal alignment
  • evaluates compatible mate pairs on the same target path with inward-facing orientation
  • scores pairs using the two alignment scores plus an optional fragment-length term
  • selects one global primary pair while preserving secondary alternatives
  • caps each mate's output MAPQ by that mate's original source MAPQ
  • preserves source confidence in aq and paired haplotype confidence in hp / hq

If no compatible pair exists, the best alternative from one original source pair is emitted as an improper pair with MAPQ 0 rather than combining mates from different source placements.

Fragment-length behavior can be controlled with:

  • --fragment-mean
  • --fragment-stdev
  • --fragment-sample-size
  • --fragment-buffer-size

If a fixed mean and standard deviation are supplied, they are used directly and no learning is performed. Otherwise, confident unambiguous pairs are collected during an initial serial warmup, buffered fragments are replayed after the model is finalized, and processing then continues in parallel. If too few training pairs are available, selection falls back to alignment scores only.

This also adds grouped paired emission so all alternatives and supplementary records for one fragment are serialized together, avoiding interleaving between threads.

Depends on:

Stacked on #5054

Testing

Adds unit and CLI coverage for:

  • primary pair appearing after secondary alternatives
  • pair evidence changing the selected placement
  • deterministic score-only ties
  • duplicate source pairs
  • mate-specific MAPQ / aq handling, including MAPQ 255
  • hard fragment-length limits
  • no-compatible-pair fallback
  • one or both mates unmapped
  • malformed mate links and unsupported supplementary input
  • fragment-model learning and insufficient-training fallback
  • GAM and GAF grouped paired input
  • multithreaded BAM output
  • candidate-local supplementary mate links and SA tags

@Sagorikanag Sagorikanag changed the title surject: support paired diploid placements surject changes: support paired diploid placements Oct 5, 2026
@Sagorikanag
Sagorikanag marked this pull request as ready for review October 5, 2026 07:26
@Sagorikanag
Sagorikanag added this pull request to stack #5053 October 5, 2026 07:26
@Sagorikanag
Sagorikanag force-pushed the surject-stack-05-diploid-paired branch from 665fa0c to 44909bf Compare October 7, 2026 05:18
@Sagorikanag
Sagorikanag force-pushed the surject-stack-05-diploid-paired branch from 44909bf to 226befd Compare October 7, 2026 17:09
@Sagorikanag
Sagorikanag force-pushed the surject-stack-05-diploid-paired branch 2 times, most recently from 66f8f45 to d3f6829 Compare October 8, 2026 16:54
@Sagorikanag
Sagorikanag force-pushed the surject-stack-05-diploid-paired branch from d3f6829 to ce9736d Compare October 21, 2026 12:39
@Sagorikanag
Sagorikanag force-pushed the surject-stack-05-diploid-paired branch from ce9736d to 58302bf Compare October 21, 2026 13:30
@Sagorikanag
Sagorikanag force-pushed the surject-stack-05-diploid-paired branch from 58302bf to 41f4ec0 Compare October 21, 2026 13:53
@Sagorikanag
Sagorikanag force-pushed the surject-stack-05-diploid-paired branch from 41f4ec0 to ef4ae6e Compare November 2, 2026 17:43
@adamnovak adamnovak added the mapping Theme/team for mapping to pangenomes label Nov 2, 2026
@Sagorikanag
Sagorikanag force-pushed the surject-stack-05-diploid-paired branch from 02a8b60 to 0315bca Compare November 4, 2026 18:38
@adamnovak
adamnovak force-pushed the surject-stack-05-diploid-paired branch from 0315bca to f5056af Compare November 9, 2026 17:22
@adamnovak
adamnovak force-pushed the surject-stack-05-diploid-paired branch from f5056af to 0315bca Compare November 9, 2026 17:42

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unfortunately out of time for today, but I think the comments and tests need to be revised for this one, and also I have ideas about how to avoid ending up with dueling representations of fragment length distributions.

Comment on lines +517 to +523
if (fragment_mean.has_value() != fragment_stdev.has_value()
|| (fragment_mean && (!isfinite(*fragment_mean) || *fragment_mean < 0))
|| (fragment_stdev && (!isfinite(*fragment_stdev) || *fragment_stdev <= 0))
|| fragment_sample_size < 2 || fragment_buffer_size == 0) {
logger.error() << "Supply both a finite nonnegative --fragment-mean and positive --fragment-stdev; "
<< "fragment sample size must be at least 2 and buffer size must be positive." << endl;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be more actionable for the user if each leg of this OR was a different, specific error message. If I read this I don't know which option I got wrong and I need to check all of them.

if (diploid_map && (input_format == "GAMP" || force_unpaired)) {
logger.error() << "--diploid-map requires name-grouped GAM/GAF input; "
<< "GAMP and --force-unpaired are not supported." << endl;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be good to have a different error for if it is GAMP, versus the one for if you pass --force-unpaired.

Also, shouldn't --force-unpaired actually be supported? Or at least have an alternative explained here? If I'm a user who has paired-end reads, and I want to surject them as if they were single-ended, and I'm not allowed to pass --force-unpaired, how am I meant to accomplish that?

Comment on lines +681 to +689
FragmentLengthDistribution distribution(fragment_sample_size, 1, 0.95);
bool model_ready = false;
size_t buffered_bytes = 0;
vector<vector<pair<Alignment, Alignment>>> buffered;
auto set_model = [&](double mean, double stdev) {
pairing.fragment_model = Surjector::DiploidFragmentModel{mean, stdev};
if (!max_frag_len) {
pairing.maximum_fragment_length = static_cast<int64_t>(min<double>(
numeric_limits<int32_t>::max(), ceil(mean + 6 * stdev)));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the magic numbers here could be made less magic by using constants shared with other implementations of fragment length distribution learning. I might put them as static constants on FragmentLengthDistribution.

Comment thread src/surjector.hpp
Comment on lines +125 to +130
/// Fixed fragment model, supplied after learning or directly by a caller.
/// The standard deviation must be positive; parameters must be finite.
struct DiploidFragmentModel {
double mean = 0.0;
double stddev = 1.0;
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is kind of overlapping in conceptual space with FragmentLengthDistribution, which, in addition to the learning logic, can also be used to represent a provided fragment length distribution. This gets us weird results, like fragment length distribution learning in diploid surjection not actually using any of the is_finalized() stuff in FragmentLengthDistribution, and having to have a set_model() to copy parameters around.

The Right Way to handle this might be to break up FragmentLengthDistribution so that one piece represents the distribution (and is DiploidFragmentModel but maybe with a constructor and getters instead of mutable fields), and a different piece is responsible for doing the learning.

But if we don't want to try and do that rocket surgery, we could tack a new constructor or factory method onto FragmentLengthDistribution that constructs a "forced" distribution from two parameters, and use that to replace DiploidFragmentModel.

Maybe there's some good reason why we weren't just using FragmentLengthDistribution to represent the distributions already? It's hard for me to tell because the code in that area is awkward and I can't see if there's a real necessary awkwardness under it.

Comment thread src/surjector.hpp
Comment on lines +140 to +143
/// source_pair_index indexes the caller's input vector, not either mate.
/// Each mate vector contains its principal first, then its supplementary pieces.
/// Both vectors are nonempty, including for unmapped mates. Supplementary mate_info
/// and SA links refer only to this alternative. Score excludes supplementary scores.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are field doc comments and shouldn't be in the class doc comment.

Comment thread src/surjector.hpp
Comment on lines +404 to +412
/// Unselected surjections of one source placement, with principal ownership intact.
/// This is shared by unpaired and paired diploid selection; no final flags are assigned.
struct DiploidCandidate {
vector<Alignment> pieces;
vector<tuple<string, int64_t, bool>> positions;
string source_path;
bool preferred = false;
int32_t haplotype_quality = 0;
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More than whether the flags are final, I would quite like to have explanations for the fields. Why do we have e.g. both a source path and strings in the positions of the pieces? What does it mean if one of these is or is not "preferred"?

Also, what's diploid about this? Just that we use it in diploid mode?

Comment thread src/surjector.hpp
int64_t maximum_fragment_length) const;

/// Score an already compatible pair; compatibility is checked separately.
/// Omits the Gaussian normalization constant, which is common to all candidates.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean the caller is going to need to do something to normalize scores later? Or is this just a Fun Implementation Fact about how scores are defined, which doesn't really belong here?

Comment thread src/surjector.hpp
Comment on lines +428 to +430
/// Remove obsolete SA links while retaining unrelated raw tags.
/// SA links are regenerated only after final flags and qualities are assigned.
static void clear_diploid_sa(Alignment& alignment);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't need to be told that this preserves other tags; a method that clears one thing of course must not clear other unrelated things.

Does this only clear "obsolete" SA tags? How does it know if an SA tag is obsolete or not?

I don't think the time when SA information will be re-generated is relevant information for defining the method to clear them.

Comment thread test/t/15_vg_surject.t
is "$(grep -v '^@' paired-gaf.sam | cut -f1-9)" "$(cut -f1-9 paired-selection.sam)" \
"Paired GAM and GAF input produce the same placements and mate fields"

# 1025 groups cross the default 512-group batch boundary and leave a partial batch.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a weird way to explain that this test exercises running multiple batches.

Comment thread test/t/15_vg_surject.t

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid representing whole SAM lines with loads of escapes in them, and doing things with awk that involve nesting a bunch of loops and conditionals. If we have a magic SAM flag number that we need to see, we probably need to explain which flags it is important that the number contains. It would be much better to test single column values than to throw whole lines around.

We need the test code to be simple enough that a person can look at it and quickly convince themselves that the code is looking for the correct thing to be true. This isn't really doing that.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mapping Theme/team for mapping to pangenomes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants