Fix diplopia: certainty-based ghost character filtering in LSTM beam search

[Copilot]

LSTM beam search can emit two different characters at adjacent timesteps when uncertain between visually similar alternatives (e.g., O/0, l/1, O/Q). CTC only collapses repeated identical codes, so different codes representing similar glyphs both survive into output text.

Approach

Previous attempt operated at the labels level in ExtractBestPathAsLabels using a gap==1 heuristic. This was too aggressive — it caught legitimate multi-code sequences in CJK, Indic, and Latin combining characters where intermediate codes naturally have gap==1.

The fix now operates in ExtractPathAsUnicharIds where characters are fully decoded, with a certainty-based threshold to distinguish true ghosts from legitimate adjacent characters.

Detection criteria

A character is removed as a diplopia ghost only when all conditions hold:

• Two adjacent decoded characters have no null gap between them (starts[i+1] == ends[i])
• At least one occupies a single timestep
• Certainty difference exceeds 2.5 (the ghost is clearly less confident)
• Neither character is a space

static constexpr float kDiplopiaCertDiff = 2.5f;
for (int i = static_cast<int>(unichar_ids->size()) - 2; i >= 0; --i) {
  if ((*unichar_ids)[i] == UNICHAR_SPACE || (*unichar_ids)[i + 1] == UNICHAR_SPACE)
    continue;
  if (starts[i + 1] == ends[i]) {
    int width_i = ends[i] - starts[i];
    int width_next = ends[i + 1] - starts[i + 1];
    if (width_i == 1 || width_next == 1) {
      float cert_diff = (*certs)[i] - (*certs)[i + 1];
      int to_remove = -1;
      if (cert_diff < -kDiplopiaCertDiff) to_remove = i;
      else if (cert_diff > kDiplopiaCertDiff) to_remove = i + 1;
      if (to_remove >= 0) { /* erase from all parallel vectors */ }
    }
  }
}

Testing

All recodebeam_test cases (Chinese, Japanese, Korean, Kannada, Marathi, English) and lstm_recode_test cases pass.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 12 complexity · 0 duplication

Metric	Results
Complexity	12
Duplication	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[stweil]

@copilot, would the following patch be also a possible solution?

diff --git a/src/lstm/recodebeam.cpp b/src/lstm/recodebeam.cpp
index 448d3d8be..b5daecf32 100644
--- a/src/lstm/recodebeam.cpp
+++ b/src/lstm/recodebeam.cpp
@@ -197,6 +197,9 @@ void RecodeBeamSearch::calculateCharBoundaries(std::vector<int> *starts,
 }
 
 // Returns the best path as labels/scores/xcoords similar to simple CTC.
+// Diplopia resolution removes spurious characters that appear at overlapping
+// positions (gap of 1 in xcoords) with another character, keeping only the
+// higher certainty emission.
 void RecodeBeamSearch::ExtractBestPathAsLabels(
     std::vector<int> *labels, std::vector<int> *xcoords) const {
   labels->clear();
@@ -216,6 +219,21 @@ void RecodeBeamSearch::ExtractBestPathAsLabels(
     }
   }
   xcoords->push_back(width);
+
+  // Diplopia resolution: remove spurious characters at overlapping positions.
+  // If two different characters have a gap of 1 in xcoords, the earlier
+  // character likely expanded into the later position, so remove the later.
+  int i = 0;
+  while (i + 1 < labels->size()) {
+    int gap = (*xcoords)[i + 1] - (*xcoords)[i];
+    if (gap == 1 && (*labels)[i] != (*labels)[i + 1]) {
+      // Likely diplopia: remove the later (spurious) character.
+      labels->erase(labels->begin() + i + 1);
+      xcoords->erase(xcoords->begin() + i + 1);
+    } else {
+      ++i;
+    }
+  }
 }
 
 // Returns the best path as unichar-ids/certs/ratings/xcoords skipping

[Copilot]

@copilot, would the following patch be also a possible solution?

diff --git a/src/lstm/recodebeam.cpp b/src/lstm/recodebeam.cpp...

Yes, that's a cleaner approach — operating at the labels level in ExtractBestPathAsLabels is simpler and catches the issue earlier in the pipeline. I've adopted your suggested patch (with a minor fix: using size_t for the loop variable to avoid a signed/unsigned comparison warning). See commit fbc9086e.

[stweil]

@copilot, these conditions catch legitimate cases in CJK, Indic, and Latin scripts (with combining characters) where characters legitimately have single-timestep width or touch each other without a null gap. Therefore the recodebeam_test fails now.

[Copilot]

@copilot, these conditions catch legitimate cases in CJK, Indic, and Latin scripts (with combining characters) where characters legitimately have single-timestep width or touch each other without a nu...

You're right — the gap==1 heuristic is too broad and catches legitimate single-timestep characters in CJK, Indic, and combining character sequences. I've reverted the change entirely in the latest commit. A more targeted approach would need additional context (e.g., visual similarity between the two characters, or confidence-based filtering) to distinguish true diplopia from legitimate adjacent characters.

[Copilot]

@copilot, these conditions catch legitimate cases in CJK, Indic, and Latin scripts (with combining characters) where characters legitimately have single-timestep width or touch each other without a nu...

You're right. The gap==1 heuristic at the labels level catches legitimate multi-code sequences (CJK, Indic, combining characters) where intermediate codes naturally have gap==1 between them.

I've moved the fix to ExtractPathAsUnicharIds where we have fully decoded unichar_ids, and added a certainty threshold (difference > 2.5) so it only removes a character when one is clearly a ghost with much lower confidence. The conditions are now:

1. No null gap between adjacent decoded characters (starts[i+1] == ends[i])
2. At least one occupies a single timestep
3. Significant certainty difference between the pair (> 2.5)
4. Neither is a space

All 6 recodebeam_test cases and 3 lstm_recode_test cases pass. See commit in latest push.