Skip to content

surject changes: detect target-path repeats during anchor pruning - #5052

Merged
adamnovak merged 5 commits into
masterfrom
surject-stack-03-target-path-sliding
Nov 9, 2026
Merged

adamnovak merged 5 commits into
masterfrom
surject-stack-03-target-path-sliding

Conversation

@Sagorikanag

Copy link
Copy Markdown
Contributor

Changelog Entry

  • vg surject now checks for nearby repeated anchor sequence on the target path, in addition to the existing read-side repeat check.

Description

This change extends suspicious-anchor pruning so an anchor can be marked for pruning when its reference sequence occurs again nearby on the target path.

The existing read-side repeat search is preserved with its original read-based search radius. The new target-path search uses the anchor's reference span and computes orientation relative to the target path, including reverse-oriented path steps and multi-node anchors.

Tests cover read-only and path-only repeats, slide boundaries, path boundaries, forward/reverse orientation combinations, multi-node anchors, reverse-oriented nodes, and differing read/reference spans caused by insertions.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it's going to work.

The tests are too interdependent for me to be able to look at any of them and tell whether they are asserting the right things or whether they are constructing the situation they claim to be constructing.

Also, I think the implementation shouldn't be based around pulling substring copies out of the path for each slide value; we should pull one string window and do all our string checks inside that, kind of like we do them inside the read already.

Comment thread src/surjector.cpp Outdated
Comment on lines +4763 to +4769
// Preserve the read-side search radius, including for anchors with indels.
const size_t read_span = chunk.first.second - chunk.first.first;
const size_t read_slide_limit = min<size_t>(max_slide, read_span * 2);
const size_t read_start = chunk.first.first - sequence.begin();
const size_t read_remaining = sequence.end() - chunk.first.second;
for (int64_t distance = -static_cast<int64_t>(read_slide_limit);
distance <= static_cast<int64_t>(read_slide_limit); ++distance) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this implementing a search radius in read space, rather than "preserve"ing "the" search in read space? This is itself the search in read space.

And where's the indel handling? The comment suggests we need to be doing something special here to handle indels in anchors, but I can't see that any such special thing is being done, which makes the code look wrong.

Comment thread src/surjector.cpp Outdated
Comment on lines +4795 to +4805
const auto first_step = step_ranges[i].first;
const auto first_handle = graph->get_handle_of_step(first_step);
const auto path_handle = graph->get_path_handle_of_step(first_step);
const auto& first_pos = chunk.second.mapping(0).position();
const bool reverse_on_path =
first_pos.is_reverse() != graph->get_is_reverse(first_handle);

const size_t step_start = graph->get_position_of_step(first_step);
const size_t node_length = graph->get_length(first_handle);
const size_t path_length = graph->get_path_length(path_handle);
const size_t mapping_offset = first_pos.offset();

@adamnovak adamnovak Oct 5, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is starting to read like Javascript, where every project has a linter that will harass you to make anything that never changes const.

But in JS you can just say const to declare a variable, and in C++ it's both more typing to make all the locals cosnt and not, as far as I know, usual practice. See for example https://quuxplusone.github.io/blog/2022/01/23/dont-const-all-the-things/#local-variables-rarely-const (but see dissenting opinions at https://stackoverflow.com/a/38926475).

@Sagorikanag Can you explain why you think marking locals as const when possible is in general better? Is this helping us keep track of what's going on in this giant function? Or are you doing this for no particular reason (in which case I would drop all the consts on locals that aren't important)?

Comment thread src/surjector.cpp Outdated
if (ref_span > path_length - slid_start) {
continue;
}
const string slid_ref_sequence = path_window(slid_start);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to pull almost the same sequence on each successive call, but it's going to build it from scratch every time, by scanning almost the same nodes of the path. I think that's unnecessarily inefficient.

I guess you can say the runtime is already O(path_slide_limit * ref_span) in the worst case, because we need equality tests on all those bases. But when mostly we can stop a lot of those equality tests early, the expected runtime is much faster, whereas here we're definitely spending O(path_slide_limit * ref_span) in even the best case making all those copies of substrings of the path. So I feel like it would be a lot better to traverse the graph path once, and extract the entire relevant window, and then do a bunch of std::equal comparisons within it. We could even maybe find a way to pull out a check-for-slide function and use it in both the read and reference cases, once we've pulled the strings.

(Actually, it would probably be even better to pull the window, and use std::find and std::rfind with appropriate bounds, and thus figure out whether there's an instance of the string other than the center one, because I bet they are faster than repeated std::equal. But we might not need to do that here.)

Comment thread src/unittest/surject.cpp Outdated
Comment on lines +336 to +341
SECTION("Read insertion preserves the read-based search radius") {
region = "ACGATTCGCCCC";
insertion = true;
read_between = "TTACGCTAGA";
max_slide = 12;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the whole set of test cases here is definitely too much of a test construction kit. If there's a good reason to have a test case construction kit, we could do it, but we'd need to explain it a lot better. We have an insertion flag that isn't documented at the top of the function, which we set here, which has the behavior of inserting "GCTA" at offset 2 in the anchor sequence, which we pulled from part of the path sequence at offset 0, which we built by concatenating a sentinel sequence onto the end of a region sequence. If this test later fails at REQUIRE(chunks.size() == (prune ? 1 : 2)); it's going to be a project to reconstruct the actual test case and what's actually supposed to be true about the result that isn't.

Comment thread src/unittest/surject.cpp Outdated
Comment on lines +343 to +347
// A distinct longer anchor prevents the keep-one-anchor fallback from
// hiding removal of the anchor being tested.
const string sentinel = "GATCCGTAGTCACTGACCTAGGTC";
const string path_sequence = sentinel_first
? sentinel + region : region + sentinel;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make more sense to add a flag to control that behavior and turn it off, so we can more easily get at the part we're trying to test?

We could also refactor things so the part that identifies which anchors ought to be kept according to the shift test is its own piece that we can call and test separately. Then those results would later be processed to actually remove the anchors, subject to the constraint that at least one anchor remains.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with your refactoring suggestion. anchor_has_nearby_repeat() now checks an anchor without modifying it, so the tests can exercise the sliding logic directly without a flag.

prune_and_trim_anchors() uses that result to mark anchors for removal, while retaining the existing keep-at-least-one-anchor behavior.

@Sagorikanag
Sagorikanag force-pushed the surject-stack-03-target-path-sliding branch from 9fc5cd8 to a0c9e33 Compare October 7, 2026 05:18
@Sagorikanag
Sagorikanag force-pushed the surject-stack-03-target-path-sliding branch from a0c9e33 to 54be5a8 Compare October 7, 2026 17:09
@Sagorikanag
Sagorikanag force-pushed the surject-stack-03-target-path-sliding branch from c9df095 to 4403580 Compare October 21, 2026 12:39
@Sagorikanag
Sagorikanag force-pushed the surject-stack-03-target-path-sliding branch from 4403580 to e1d68be Compare October 21, 2026 13:29
@adamnovak
adamnovak force-pushed the surject-stack-03-target-path-sliding branch from 3016458 to aad579f Compare October 30, 2026 17:30
@adamnovak adamnovak added the mapping Theme/team for mapping to pangenomes label Nov 2, 2026

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty good, but some of the lambda helper comments seem wrong.

Comment thread src/unittest/surject.cpp Outdated
CHECK(surjector.anchor_has_nearby_repeat(read, chunk, {step_b, step_a}));
}

TEST_CASE("Reverse mappings on reverse path steps use forward path coordinates",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these test case names could hint a bit more that they are about surjection.

Comment thread src/surjector.cpp Outdated
Comment on lines +4680 to +4684
// Limit the search to twice the span without overflowing span * 2.
auto slide_limit = [&](size_t span) {
size_t limit = max_slide;
return span > limit / 2 ? limit : span * 2;
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced this function does what it says it does here. It looks like it limits the search to max_slide, or twice the span, whichever is smaller. The comment just talks about limiting it to twice the span, and also to span * 2, which is the same thing. And it's using integer division in a way that makes me nervous; the code isn't very clear.

Comment thread src/surjector.cpp Outdated
Comment on lines +4685 to +4687
// Compare complete windows, excluding the original occurrence.
auto has_repeat = [&](const string& bases, size_t start,
size_t span, size_t radius) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this function doesn't compare (two individual) windows. It does a whole search.

Comment thread src/surjector.cpp Outdated
Comment on lines +4729 to +4740
assert(mapping_offset <= node_length);

size_t anchor_start;
if (reverse_relative_to_path) {
size_t anchor_end = step_start + node_length - mapping_offset;
assert(ref_span <= anchor_end);
anchor_start = anchor_end - ref_span;
} else {
anchor_start = step_start + mapping_offset;
}
assert(anchor_start <= path_length);
assert(ref_span <= path_length - anchor_start);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I usually like to use crash_unless() instead of assert() when we're working on reads, because when it fails we get more information about the offending read.

Base automatically changed from surject-stack-02-interval-classification to master November 9, 2026 17:22
@adamnovak
adamnovak force-pushed the surject-stack-03-target-path-sliding branch from 3acd85e to f23af19 Compare November 9, 2026 17:22

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me now.

@adamnovak

Copy link
Copy Markdown
Member

Apparently merging the PR below this in the stack has changed the commit this branch points to and kicked off testing again.

@adamnovak
adamnovak force-pushed the surject-stack-03-target-path-sliding branch from f23af19 to 3acd85e Compare November 9, 2026 17:38
@adamnovak

Copy link
Copy Markdown
Member

Commit 3acd85e passed in https://ucsc-ci.com/vgteam/vg/-/pipelines/13355, and I'm pretty sure the Mac tests passed here before, so I am going to merge that commit in manually. I already un-force-pushed all the stacked PR branches. Maybe that gets us out of having to do the rebase?

@adamnovak
adamnovak merged commit 9cf5f55 into master Nov 9, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mapping Theme/team for mapping to pangenomes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants