Skip to content

surject changes: prune anchors in mapper-declared tail regions - #5050

Merged
adamnovak merged 6 commits into
masterfrom
surject-stack-01-tail-pruning
Oct 30, 2026
Merged

adamnovak merged 6 commits into
masterfrom
surject-stack-01-tail-pruning

Conversation

@Sagorikanag

Copy link
Copy Markdown
Contributor

Changelog Entry

To be copied to the [draft changelog](https://github.com/vgteam/vg/wiki/Draft-Changelog) by merger:

  • vg surject adds --prune-tail-region / -j to prune anchors contained entirely within mapper-declared read tail regions.

Description

Adds optional tail-aware anchor pruning to vg surject. When enabled, left_tail_length and right_tail_length annotations from GAM/GAF input are used to remove anchors that fall entirely within the declared tail regions before the existing anchor-pruning logic runs.

The behavior is covered by unit and CLI tests, including forward/reverse read orientation and GAM/GAF input.

First PR in the surjection stack replacing #5026.

@Sagorikanag
Sagorikanag added this pull request to stack #5053 October 2, 2026 03:48
@Sagorikanag
Sagorikanag force-pushed the surject-stack-01-tail-pruning branch from 232f7dd to f9c02e0 Compare October 2, 2026 17:26

@adamnovak adamnovak left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the implementation here is good, but we have over 300 lines of tests for about 80 lines of implementation, and overall the tests seem more complex than the implementation they are meant to test. Some of these tests look like they might not be very useful, including the test to see if the assigned short form of --prune-tail-region is really -j, and the C++ test to make sure that the complete surject function is understanding the left and right tails as read-left and read-right, which as far as I can tell is the only coherent way to understand them. (EDIT: I think the other possible way to understand them would be left and right in the target path. But then I think that's better tested in a way based on the idea of trying different orientations of the read relative to the target path, in a context where we can tell the left and right tail lengths apart, rather than a special test around making sure we don't make that mistake. Unless we did at some point make that mistake?)

I would cut that really long unit test, and replace it with some tests of anchor pruning when the read visits some nodes in their local-reverse orientations if we're really worried it's possible for the implementation to somehow confuse left and right. If we want to test the parsing of the left and right annotations, we probably want to break it out into a testable unit, separate from all of surjection. I'd also make the inputs we're synthesizing in the .t tests into files stored in the repo instead.

Comment thread src/unittest/surject.cpp Outdated
Comment on lines +119 to +128
const string tail_sequence = "ACGTCAGTGCAT";
const string bridge_sequence = "GATCTAGC";
const string core_sequence = "TGCAGATCGTACCTGATGCACTAGGTCAGTAC";

auto misplaced_tail = graph.create_handle(tail_sequence);
auto spacer = graph.create_handle(string(80, 'C'));
auto correct_tail = graph.create_handle(tail_sequence);
auto reference_bridge = graph.create_handle(bridge_sequence);
auto core = graph.create_handle(core_sequence);
auto alternate_bridge = graph.create_handle(bridge_sequence);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test case has an underlying logic of some kind, with 6 moving parts in 3 different types, but the underlying logic isn't explained.

Comment thread src/unittest/surject.cpp Outdated
Comment on lines +115 to +116
TEST_CASE("Surjection uses tail annotations in either read orientation",
"[surject][tail-pruning]") {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why we're bothering with this complex test case.

It sounds like it's trying to test whether we are interpreting the left and right annotations as left and right in read space and not left and right in graph space. But there isn't really such a thing as left and right in graph space, and so there aren't really two read orientations for us to be testing that we support them both.

Couldn't we just have more cases for prune_and_trim_anchors where the read path runs through the nodes in different orientations?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, duh, the other possible interpretation of left and right would be in the space of the target path.

I'm still not sure this is the right way to test that we handle all relative orientations vs. the target path correctly, though.

Comment thread src/surjector.cpp Outdated
// low complexity sequences
// Read mapper-declared tail lengths. These coordinates are
// relative to the stored read sequence and independent of path orientation.
// Missing annotations remain zero, making tail-region pruning a no-op.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The annotation can't remain 0 if it's missing and so doesn't exist to begin with.

I also don't know what the reader would think tail-region pruning would do for a zero-length tail other than nothing. This part isn't actually about the pruning (just the parsing) so we don't need to explain the edge case behavior of the pruning here.

Suggested change
// Missing annotations remain zero, making tail-region pruning a no-op.
// Missing annotations are interpreted as zero.

Comment thread src/surjector.cpp
Comment on lines +4698 to +4701
// Clamp tail lengths to the read length
size_t read_length = sequence.size();
left_tail_length = std::min(left_tail_length, read_length);
right_tail_length = std::min(right_tail_length, read_length);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we encountering tails longer than reads in real life? Do we have a good excuse for them? If not we might want to fail or warn, because if some of our tail annotations are garbage the rest might also be garbage.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We won't encounter a tail longer than the read right now, but surject can create this situation internally: when it splits a read with supplementary alignments into hard-clipped pieces, it currently copies the full read’s tail annotations unchanged. Eg. - A valid 500 bp tail on a 5 kb read can therefore exceed the length of a 300 bp piece.

I’ve kept the existing clamping for now and added a TODO to translate the annotations into each piece’s coordinates. Clamping prevents out-of-range lengths but does not correct the coordinates. Once this is taken care of, I agree that oversized tails should be reported rather than silently accepted.

But since that might need to address the whole supplementary surjection structure, I'm not introducing that in this PR.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, then I think it makes sense to keep this here.

Comment thread test/t/15_vg_surject.t Outdated
Comment on lines +292 to +304
# Reuse the unit-test graph: the input takes 1 -> 6 -> 5, while the
# exact reference placement after tail pruning is 3 -> 4 -> 5.
# Without -u, this fixture already produces the exact placement without
# tail pruning. Enable supplementary output so the baseline retains the
# misplaced tail as a separate alignment, making pruning's effect observable.
tail_dir=$(mktemp -d) || exit 1
tail_seq=ACGTCAGTGCAT
bridge_seq=GATCTAGC
core_seq=TGCAGATCGTACCTGATGCACTAGGTCAGTAC
spacer_seq=$(printf '%080d' 0 | tr 0 C)
printf 'H\tVN:Z:1.0\nS\t1\t%s\nS\t2\t%s\nS\t3\t%s\nS\t4\t%s\nS\t5\t%s\nS\t6\t%s\n' \
"$tail_seq" "$spacer_seq" "$tail_seq" "$bridge_seq" "$core_seq" "$bridge_seq" > "$tail_dir/graph.gfa"
printf 'L\t1\t+\t2\t+\t0M\nL\t2\t+\t3\t+\t0M\nL\t3\t+\t4\t+\t0M\nL\t4\t+\t5\t+\t0M\nL\t1\t+\t6\t+\t0M\nL\t6\t+\t5\t+\t0M\nP\tref\t1+,2+,3+,4+,5+\t*\n' >> "$tail_dir/graph.gfa"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend using real newlines and tabs and heredoc syntax like:

cat >$tail_dir/graph.gfa <<EOF
# GFA lines here
EOF

Then we could actually read the GFA we're generating.

Also we're not so much re-using the complex graph from the unit test as we are translating the code that generated it to Bash. And what's so exact about the reference placement?

Comment thread test/t/15_vg_surject.t Outdated
# Without -u, this fixture already produces the exact placement without
# tail pruning. Enable supplementary output so the baseline retains the
# misplaced tail as a separate alignment, making pruning's effect observable.
tail_dir=$(mktemp -d) || exit 1

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a good reason for these tail-pruning-related tests to work in their own randomly-named temp directory when everything else uses predictably-named files in the current directory as scratch?

It removes the need to name all the scratch files individually in the rm command at the end, but it also makes it slightly harder to copy-paste the test commands to debug a failing test, because you also need to worry about setting up this directory.

Comment thread test/t/15_vg_surject.t Outdated
Comment on lines +332 to +335
vg surject -x "$tail_dir/graph.gfa" -p ref -t 1 -u -s -j "$tail_dir/read.gam" > "$tail_dir/short.sam"
is "$?" 0 "The short tail-pruning flag is accepted"
is "$(grep -v '^@' "$tail_dir/short.sam")" "$(grep -v '^@' "$tail_dir/long.sam")" \
"Short and long tail-pruning flags produce identical alignment records"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't see it being worth it to have a test for the short option that we set being the short option that's actually assigned. We already have that linter script that makes sure the help and the actual option-parsing code agree on what long option goes with what short option; it's nearly syntactically impossible to get this wrong.

If we want to make sure that e.g. wiki documentation is talking about the correct letter for a flag, we should probably handle that with the testable wiki documentation setup.

Suggested change
vg surject -x "$tail_dir/graph.gfa" -p ref -t 1 -u -s -j "$tail_dir/read.gam" > "$tail_dir/short.sam"
is "$?" 0 "The short tail-pruning flag is accepted"
is "$(grep -v '^@' "$tail_dir/short.sam")" "$(grep -v '^@' "$tail_dir/long.sam")" \
"Short and long tail-pruning flags produce identical alignment records"

Comment thread test/t/15_vg_surject.t Outdated
"Short and long tail-pruning flags produce identical alignment records"

vg surject -x "$tail_dir/graph.gfa" -p ref -t 1 -u -s "$tail_dir/plain.gam" > "$tail_dir/plain-baseline.sam"
is "$?" 0 "Unannotated input can be surjected without tail pruning"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is also setup pretending to be a test. We already tested unannotated input with pruning off in basically the whole rest of the file.

Suggested change
is "$?" 0 "Unannotated input can be surjected without tail pruning"

Comment thread test/t/15_vg_surject.t Outdated
"Tail pruning leaves unannotated input unchanged"

vg convert "$tail_dir/graph.gfa" -G "$tail_dir/read.gam" -t 1 > "$tail_dir/read.gaf"
is "$?" 0 "Annotated tail-pruning input converts to GAF"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a test for vg convert, not vg surject.

Suggested change
is "$?" 0 "Annotated tail-pruning input converts to GAF"

Comment thread test/t/15_vg_surject.t Outdated
Comment on lines +292 to +317
# Reuse the unit-test graph: the input takes 1 -> 6 -> 5, while the
# exact reference placement after tail pruning is 3 -> 4 -> 5.
# Without -u, this fixture already produces the exact placement without
# tail pruning. Enable supplementary output so the baseline retains the
# misplaced tail as a separate alignment, making pruning's effect observable.
tail_dir=$(mktemp -d) || exit 1
tail_seq=ACGTCAGTGCAT
bridge_seq=GATCTAGC
core_seq=TGCAGATCGTACCTGATGCACTAGGTCAGTAC
spacer_seq=$(printf '%080d' 0 | tr 0 C)
printf 'H\tVN:Z:1.0\nS\t1\t%s\nS\t2\t%s\nS\t3\t%s\nS\t4\t%s\nS\t5\t%s\nS\t6\t%s\n' \
"$tail_seq" "$spacer_seq" "$tail_seq" "$bridge_seq" "$core_seq" "$bridge_seq" > "$tail_dir/graph.gfa"
printf 'L\t1\t+\t2\t+\t0M\nL\t2\t+\t3\t+\t0M\nL\t3\t+\t4\t+\t0M\nL\t4\t+\t5\t+\t0M\nL\t1\t+\t6\t+\t0M\nL\t6\t+\t5\t+\t0M\nP\tref\t1+,2+,3+,4+,5+\t*\n' >> "$tail_dir/graph.gfa"

jq -n --arg sequence "${tail_seq}${bridge_seq}${core_seq}" '
{name: "tail-pruning", sequence: $sequence,
annotation: {left_tail_length: 12},
path: {mapping: [
{rank: 1, position: {node_id: "1"},
edit: [{from_length: 12, to_length: 12}]},
{rank: 2, position: {node_id: "6"},
edit: [{from_length: 8, to_length: 8}]},
{rank: 3, position: {node_id: "5"},
edit: [{from_length: 32, to_length: 32}]}
]}}' > "$tail_dir/read.json"
vg view -JGa "$tail_dir/read.json" > "$tail_dir/read.gam"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make these static test case GFA and GAM files, instead of trying to generate them at runtime by writing them out from strings in the .t file.

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the code is good here.

I would still revise some of the comments to make more sense, but I feel bad sending more red X reviews.

Comment thread src/surjector.cpp Outdated
// we want to remove anchors that can be error-prone: short anchors in the tails and anchors in
// low complexity sequences
// Read mapper-declared tail lengths. These coordinates are
// relative to the stored read sequence and independent of path orientation.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be so weird for the annotations stored in the read to be relative to some path in the graph and not the read that they are actually on, to the point where I think it is going to confuse the reader to tell them that it isn't the case.

Comment thread src/surjector.cpp
Comment on lines +4698 to +4701
// Clamp tail lengths to the read length
size_t read_length = sequence.size();
left_tail_length = std::min(left_tail_length, read_length);
right_tail_length = std::min(right_tail_length, read_length);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, then I think it makes sense to keep this here.

Comment thread test/t/15_vg_surject.t Outdated
Comment on lines +292 to +298
# The reference follows 1 -> 2 -> 3 -> 4 -> 5; the read follows the shortcut 1 -> 6 -> 5.
# Node 1 (12 bp) and node 3 share the same sequence; nodes 4 and 6 share the same 8-base
# sequence; node 2 is an 80-base run of C's separating them; node 5 is the 32-base body
# of the read. The read's left_tail_length=12 marks node 1 as the tail. Pruning that
# anchor lets the surjector place the whole read through 3 -> 4 -> 5 starting at
# SAM position 93. Without pruning, node 1 (ref pos 1) and node 5 (ref pos 113) are
# too far apart to emit as one alignment, so -u splits the tail into a supplementary record.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is too much detail about something I'm not looking at and have not been convinced I want to understand anyway.

GFA supports # comments, so if we need to name different parts of the graph that might be the way to do it. But we might not need to.

Here we can probably say just that the read here has an anchor in its left tail that is far enough from the rest of the read to force a supplementary alignment if not pruned.

Sagorikanag and others added 2 commits October 20, 2026 14:17
@adamnovak
adamnovak merged commit f1e4cc9 into master Oct 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants