[Bee] docling parse generates text as each character separated

[yqliving]

Bug

With DoclingParseV4DocumentBackend, the plain text or Markdown output renders programmatic PDF page as "T i t l e C a s e W o r k e r s ..." instead of "Title Case Workers ...". However, PyPdfiumDocumentBackend is correct.

Steps to reproduce

1. set standard pdf pipeline with DoclingParseV4DocumentBackend
2. do document convert
3. export md or plain text to check the output

Test file: because it is a customer file, please downlow it from https://github.ibm.com/ai-foundation/watson-doc-understanding-issue-tracker/issues/4015#issuecomment-233084821

Docling version

docling-parse from 7.0.0 -> 7.11.0

Python version

3.12

[dosubot]

@yqliving, this is a documented class of bug rooted in the docling-parse C++ library, not the Python layer. When PDFs use per-glyph Tj operators (one character per operator), docling-parse emits each character as a separate text cell. The Python-side sanitize_text() only handles dehyphenation, typographic normalization, and ligature expansion — it has no logic to collapse single-character-spaced runs back into words [1] [2]. The same artifact was reported in issue #3882 for URLs and metadata fields.

Workaround — try enforce_same_font=False: This option was introduced to prevent docling-parse from splitting text cells at font boundaries (e.g. Zwierz ą t → Zwierząt). Since the underlying mechanism — splitting at glyph/font boundaries — overlaps with your case, it may help [3]:

from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.backend_options import PdfBackendOptions

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            backend_options=PdfBackendOptions(enforce_same_font=False)
        )
    }
)

If that doesn't resolve it (the character-spacing artifact may be at the extraction level rather than the font-boundary level), force_full_page_ocr=True will bypass the programmatic text layer entirely by running OCR on rendered page images instead.

The root fix needs to happen in docling-parse itself — the C++ parser needs to detect per-glyph Tj runs and collapse them into proper word cells [4]. A reproducible sample PDF shared with the maintainers would help prioritize this.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[yqliving]

@dosu, the workaround of setting enforce_same_font=False does not work, still see the extra space between each character. So we need a fix in docling-parser.

[dosubot]

@yqliving That makes sense — enforce_same_font=False addresses a different problem (PDFs where base glyphs and diacritics use separate fonts, causing word splits at font boundaries) [1]. Your issue is that docling-parse emits each character as a separate text cell because the PDF uses per-glyph Tj operators, and the Python layer interprets the gaps between cells as spaces [2]. Font enforcement has no bearing on that.

You're right that the fix needs to happen in docling-parse — the C++ parser would need to detect runs of single-character cells with tight inter-glyph spacing and collapse them into word-level cells before returning them to the Python backend. The same class of issue was reported in #3882 for URLs in reference sections, but no upstream patch has landed yet [3].

In the meantime, the only reliable workaround is enabling full-page OCR, which bypasses the programmatic text layer entirely [4]:

from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat

pipeline_options = PdfPipelineOptions()
pipeline_options.do_ocr = True
pipeline_options.force_full_page_ocr = True

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
    }
)

This is slower and table/reading-order fidelity depends on OCR quality, but it should produce correctly joined words since it re-reads from the rendered page image rather than the character stream.

If you can share a redacted/anonymized sample page, filing it directly against docling-parse would help the maintainers prioritize the fix.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[wittjeff]

I dug into this and can reproduce the exact "T i t l e C a s e" artifact with a
synthetic PDF, so here is a constrained diagnosis. The trigger is not per-glyph
Tj operators by themselves — docling-parse reassembles those correctly. The
trigger is a font whose declared widths are zero while every glyph is
positioned explicitly.

Root cause

1. Some generators emit one Tj (plus a Td advance) per character and ship a
   /Widths array of zeros. Rendering is pixel-perfect because positioning
   never consults declared widths — and pdfium reads the real metrics from the
   font program, which is why PyPdfiumDocumentBackend looks correct.
2. docling-parse trusts the declared width unchecked
   (page_font.h, get_width()):
   numb_to_widths[c] is returned even when it is 0, so every character cell
   gets a zero-width bbox at the correct origin. Verified on the repro:
   'T' x0=72.00 x1=72.00 w=0.0000, 'i' x0=79.33 x1=79.33 w=0.0000, …
3. The word/line merger scales its adjacency tolerance by the cell's average
   character width
   (page_item_sanitators/cells.h,
   delta_0 = average_char_width() * space_width_factor_for_merge). Zero-width
   cells give delta_0 = 0, every real inter-glyph gap (~2–7pt) fails
   is_adjacent_to, and nothing merges at word or line level.
4. Docling's assembly then joins the per-character cells with spaces →
   T i t l e C a s e W o r k e r s.

This also explains why enforce_same_font=False had no effect (it gates a
different check in applicable_for_merge), and it is independent of the 7.x
range you tested — I reproduce identically on 6.2.0 and 7.15.0; the logic is
unchanged. Very likely the same mechanism as #3882 (URLs and metadata strings
are exactly the runs a generator positions glyph-by-glyph).

Minimal reproduction

Any PDF with /Widths [ 0 0 0 … ] and per-glyph positioning triggers it:

# builds a one-page PDF: Helvetica, /Widths all zero, one Tj per glyph,
# each followed by a Td of the glyph's true advance ("Title Case Workers")
W = {"T":611,"i":222,"t":278,"l":222,"e":556,"C":722,"a":556,"s":500,
     "W":944,"o":556,"r":333,"k":500," ":278}
ops = ["BT /F1 12 Tf 72 720 Td"]
for ch in "Title Case Workers":
    adv = W[ch] * 12 / 1000.0
    ops.append(f"{adv:.2f} 0 Td" if ch == " " else f"({ch}) Tj {adv:.2f} 0 Td")
ops.append("ET")
content = " ".join(ops)
zero_widths = " ".join("0" for _ in range(95))
font = ("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /FirstChar 32 "
        f"/LastChar 126 /Widths [ {zero_widths} ] >>")
# ...assemble objects + xref as usual; full builder in the linked test file

Result with docling-parse (6.2.0 and 7.15.0 identically):

word cells: ['T','i','t','l','e','C','a','s','e','W','o','r','k','e','r','s']
line cells: ['T','i','t','l','e','C','a','s','e','W','o','r','k','e','r','s']

pdfium on the same file: Ti tl e Case Workers (near-correct even without an
embedded font program; fully correct with one).

Six neighboring constructs do not reproduce it (docling-parse handles all
of these correctly, which is what narrowed the search): plain per-glyph Tj
with honest widths, one BT..ET per glyph, 1pt tracking, TJ arrays with
kerning, Tz 200 scaling, and Tc character spacing.

Proposed fix

In get_width(): treat a declared width of 0 for a glyph that maps to visible
text as missing rather than trusted, falling back to base-font metrics →
non-zero /MissingWidth//DW → default, in that order (a font that lies in
/Widths usually lies in its default width too). Glyphs that legitimately
carry no advance — space glyphs, combining marks, zero-width/format characters
— keep their declared zero, and non-zero declared widths keep precedence
exactly as today.

I have this implemented with four regression tests (zero-widths repro at word
and line level, whole-string Tj variant, zero-width space glyph still acting
as a word boundary, and non-zero declared widths keeping precedence) on a
branch, and I'm happy to open a PR against docling-parse if useful. A sanitized
sample page from the original file would still be valuable to confirm the
customer font matches this profile (zero or degenerate declared widths) —
@yqliving, could you check the /Widths (or CID /W) array of the font in
question?

🤖 Generated with Claude Code