Repository navigation
surject changes: support paired diploid placements - #5055
Sagorikanag wants to merge 2 commits into
Conversation
665fa0c to
44909bf
Compare
44909bf to
226befd
Compare
66f8f45 to
d3f6829
Compare
d3f6829 to
ce9736d
Compare
ce9736d to
58302bf
Compare
58302bf to
41f4ec0
Compare
41f4ec0 to
ef4ae6e
Compare
02a8b60 to
0315bca
Compare
0315bca to
f5056af
Compare
f5056af to
0315bca
Compare
adamnovak
left a comment
There was a problem hiding this comment.
I'm unfortunately out of time for today, but I think the comments and tests need to be revised for this one, and also I have ideas about how to avoid ending up with dueling representations of fragment length distributions.
| if (fragment_mean.has_value() != fragment_stdev.has_value() | ||
| || (fragment_mean && (!isfinite(*fragment_mean) || *fragment_mean < 0)) | ||
| || (fragment_stdev && (!isfinite(*fragment_stdev) || *fragment_stdev <= 0)) | ||
| || fragment_sample_size < 2 || fragment_buffer_size == 0) { | ||
| logger.error() << "Supply both a finite nonnegative --fragment-mean and positive --fragment-stdev; " | ||
| << "fragment sample size must be at least 2 and buffer size must be positive." << endl; | ||
| } |
There was a problem hiding this comment.
This would be more actionable for the user if each leg of this OR was a different, specific error message. If I read this I don't know which option I got wrong and I need to check all of them.
| if (diploid_map && (input_format == "GAMP" || force_unpaired)) { | ||
| logger.error() << "--diploid-map requires name-grouped GAM/GAF input; " | ||
| << "GAMP and --force-unpaired are not supported." << endl; | ||
| } |
There was a problem hiding this comment.
It might be good to have a different error for if it is GAMP, versus the one for if you pass --force-unpaired.
Also, shouldn't --force-unpaired actually be supported? Or at least have an alternative explained here? If I'm a user who has paired-end reads, and I want to surject them as if they were single-ended, and I'm not allowed to pass --force-unpaired, how am I meant to accomplish that?
| FragmentLengthDistribution distribution(fragment_sample_size, 1, 0.95); | ||
| bool model_ready = false; | ||
| size_t buffered_bytes = 0; | ||
| vector<vector<pair<Alignment, Alignment>>> buffered; | ||
| auto set_model = [&](double mean, double stdev) { | ||
| pairing.fragment_model = Surjector::DiploidFragmentModel{mean, stdev}; | ||
| if (!max_frag_len) { | ||
| pairing.maximum_fragment_length = static_cast<int64_t>(min<double>( | ||
| numeric_limits<int32_t>::max(), ceil(mean + 6 * stdev))); |
There was a problem hiding this comment.
Some of the magic numbers here could be made less magic by using constants shared with other implementations of fragment length distribution learning. I might put them as static constants on FragmentLengthDistribution.
| /// Fixed fragment model, supplied after learning or directly by a caller. | ||
| /// The standard deviation must be positive; parameters must be finite. | ||
| struct DiploidFragmentModel { | ||
| double mean = 0.0; | ||
| double stddev = 1.0; | ||
| }; |
There was a problem hiding this comment.
I think this is kind of overlapping in conceptual space with FragmentLengthDistribution, which, in addition to the learning logic, can also be used to represent a provided fragment length distribution. This gets us weird results, like fragment length distribution learning in diploid surjection not actually using any of the is_finalized() stuff in FragmentLengthDistribution, and having to have a set_model() to copy parameters around.
The Right Way to handle this might be to break up FragmentLengthDistribution so that one piece represents the distribution (and is DiploidFragmentModel but maybe with a constructor and getters instead of mutable fields), and a different piece is responsible for doing the learning.
But if we don't want to try and do that rocket surgery, we could tack a new constructor or factory method onto FragmentLengthDistribution that constructs a "forced" distribution from two parameters, and use that to replace DiploidFragmentModel.
Maybe there's some good reason why we weren't just using FragmentLengthDistribution to represent the distributions already? It's hard for me to tell because the code in that area is awkward and I can't see if there's a real necessary awkwardness under it.
| /// source_pair_index indexes the caller's input vector, not either mate. | ||
| /// Each mate vector contains its principal first, then its supplementary pieces. | ||
| /// Both vectors are nonempty, including for unmapped mates. Supplementary mate_info | ||
| /// and SA links refer only to this alternative. Score excludes supplementary scores. |
There was a problem hiding this comment.
These are field doc comments and shouldn't be in the class doc comment.
| /// Unselected surjections of one source placement, with principal ownership intact. | ||
| /// This is shared by unpaired and paired diploid selection; no final flags are assigned. | ||
| struct DiploidCandidate { | ||
| vector<Alignment> pieces; | ||
| vector<tuple<string, int64_t, bool>> positions; | ||
| string source_path; | ||
| bool preferred = false; | ||
| int32_t haplotype_quality = 0; | ||
| }; |
There was a problem hiding this comment.
More than whether the flags are final, I would quite like to have explanations for the fields. Why do we have e.g. both a source path and strings in the positions of the pieces? What does it mean if one of these is or is not "preferred"?
Also, what's diploid about this? Just that we use it in diploid mode?
| int64_t maximum_fragment_length) const; | ||
|
|
||
| /// Score an already compatible pair; compatibility is checked separately. | ||
| /// Omits the Gaussian normalization constant, which is common to all candidates. |
There was a problem hiding this comment.
Does this mean the caller is going to need to do something to normalize scores later? Or is this just a Fun Implementation Fact about how scores are defined, which doesn't really belong here?
| /// Remove obsolete SA links while retaining unrelated raw tags. | ||
| /// SA links are regenerated only after final flags and qualities are assigned. | ||
| static void clear_diploid_sa(Alignment& alignment); |
There was a problem hiding this comment.
I don't need to be told that this preserves other tags; a method that clears one thing of course must not clear other unrelated things.
Does this only clear "obsolete" SA tags? How does it know if an SA tag is obsolete or not?
I don't think the time when SA information will be re-generated is relevant information for defining the method to clear them.
| is "$(grep -v '^@' paired-gaf.sam | cut -f1-9)" "$(cut -f1-9 paired-selection.sam)" \ | ||
| "Paired GAM and GAF input produce the same placements and mate fields" | ||
|
|
||
| # 1025 groups cross the default 512-group batch boundary and leave a partial batch. |
There was a problem hiding this comment.
This is a weird way to explain that this test exercises running multiple batches.
There was a problem hiding this comment.
We should avoid representing whole SAM lines with loads of escapes in them, and doing things with awk that involve nesting a bunch of loops and conditionals. If we have a magic SAM flag number that we need to see, we probably need to explain which flags it is important that the number contains. It would be much better to test single column values than to throw whole lines around.
We need the test code to be simple enough that a person can look at it and quickly convince themselves that the code is looking for the correct thing to be true. This isn't really doing that.
Changelog Entry
vg surject --diploid-mapcan jointly surject grouped paired-end placements, using paired alignment evidence and an optional or learned fragment-length model.Description
This extends diploid surjection to paired-end reads with multiple graph placements per fragment.
For
--diploid-map -i, consecutive interleaved placement pairs for one fragment are processed together. Each alternative remains tied to its original mate pair, including supplementary alignments, so mate fields andSAlinks are rebuilt within the correct candidate.Paired selection:
aqand paired haplotype confidence inhp/hqIf no compatible pair exists, the best alternative from one original source pair is emitted as an improper pair with MAPQ 0 rather than combining mates from different source placements.
Fragment-length behavior can be controlled with:
--fragment-mean--fragment-stdev--fragment-sample-size--fragment-buffer-sizeIf a fixed mean and standard deviation are supplied, they are used directly and no learning is performed. Otherwise, confident unambiguous pairs are collected during an initial serial warmup, buffered fragments are replayed after the model is finalized, and processing then continues in parallel. If too few training pairs are available, selection falls back to alignment scores only.
This also adds grouped paired emission so all alternatives and supplementary records for one fragment are serialized together, avoiding interleaving between threads.
Depends on:
Stacked on #5054
Testing
Adds unit and CLI coverage for:
aqhandling, including MAPQ 255SAtags