Implement Roslyn-style red-green tree architecture for DOM parser - #1801
Draft
angelozerr wants to merge 17 commits into
Draft
Implement Roslyn-style red-green tree architecture for DOM parser #1801angelozerr wants to merge 17 commits into
angelozerr wants to merge 17 commits into
Conversation
Introduce an immutable green tree layer (width-based, no parent pointers) that the existing red DOM tree wraps with absolute offsets. DOMParser now delegates to GreenTreeBuilder → RedTreeBuilder pipeline. This enables future incremental reparsing by structural sharing of unchanged subtrees. Key components: - Green tree nodes: GreenDocument, GreenElement, GreenText, GreenComment, GreenCDATA, GreenProcessingInstruction, GreenDocumentType, and DTD nodes - GreenTreeBuilder: scanner-driven builder producing immutable green trees - RedTreeBuilder: converts green tree to existing DOMNode hierarchy - Gap-filling: automatic whitespace filler nodes ensure contiguous widths - All 2392 existing tests pass with zero regressions Signed-off-by: azerr <azerr@redhat.com>
angelozerr
force-pushed
the
fix/red-green-tree
branch
2 times, most recently
from
September 4, 2026 19:52
77f89b5 to
633c69a
Compare
Add incremental reparsing that reuses unchanged subtrees when editing a document, avoiding full reparse for single-edit changes. The green tree is preserved on DOMDocument for structural sharing across versions. Key changes: - IncrementalParser: finds reusable prefix/suffix children at document level, reparses only the affected middle region via parseRange() - GreenTreeBuilder.parseRange(): parses a subrange of text, enabling partial reparsing without scanning the full document - ModelTextDocument: model field is now volatile for thread-safe reads, preserves previousModel and edit info for incremental path - DOMDocument stores its GreenDocument for reuse across edits - GreenNode.childrenStartRel() factored from RedTreeBuilder into each green node subclass Tests: 195 tests covering green tree, red tree, incremental parser, structural sharing, ModelTextDocument edit flow, and DOMParser integration. All 2461 existing tests pass with zero regressions. Signed-off-by: azerr <azerr@redhat.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
angelozerr
force-pushed
the
fix/red-green-tree
branch
from
September 5, 2026 05:31
633c69a to
432395f
Compare
The incremental parser now descends into elements when exactly one child contains the edit, finding prefix/suffix children at each nesting level. This is critical for the common XML pattern of a single root element wrapping many children (e.g., <catalog> with 24863 <product> elements). Before: editing one product reparsed the entire <catalog> (20 MB). After: only the modified <product> is reparsed, 24862 others are reused. Key changes: - IncrementalParser: recursive tryIncrementalOnChildren/tryDescentIntoElement replaces flat document-level-only algorithm - GreenElement.withNewChildren(): creates element with new children and adjusted endTagOpenRel/endTagCloseRel offsets - Adjacent GreenText nodes at splice boundaries are coalesced to match full-parse tree structure - 6 new tests: descent into root, deep nesting, many children (100), attribute edit fallback, text coalescing, prolog+doctype+descent Signed-off-by: azerr <azerr@redhat.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers the full architecture: green tree (immutable, width-based nodes), red tree (mutable DOMNode facade), GreenTreeBuilder, RedTreeBuilder, incremental parser algorithm (prefix/suffix detection, recursive descent, text coalescing), LSP integration (ModelTextDocument, volatile fields, edit flow), performance characteristics, and limitations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace previousModel (full DOMDocument with entire red tree) with previousIncrementalData that stores only the extracted GreenDocument. The old red tree is now eligible for GC immediately on cancelModel() instead of being held during the entire incremental parse. For a 21 MB file, this frees ~20 MB of heap earlier. The incrementalDataExtractor function is passed through ModelTextDocuments to ModelTextDocument, keeping the generic API clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
How about benchmarks? |
Contributor
Author
|
@fbricon I will do that, but please try it, the result is impressive (according to my tests with content.xml file from test which is a large file) |
When parsing incrementally, defer DOMNode children creation until accessed via getChildren(), getFirstChild(), etc. Each element stores its GreenNode and absolute offset, expanding only when a feature navigates into it. This avoids creating all 50K+ red nodes upfront when only a small path is accessed (e.g., hover at cursor position). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pack boolean fields into bit flags (widthAndFlags in GreenNode, byte flags in DOMNode) saving ~12MB across ~1.5M green nodes and ~8MB across red nodes - Replace ArrayList with raw arrays in NodeBuilder and GreenElementBuilder, reducing allocation from 578MB to 24MB - Add ArrayLineTracker (flat int[]+byte[]) replacing TreeLineTracker AVL tree, saving ~22MB (5 bytes/line vs 40 bytes/line) - Cache GreenText whitespace nodes for widths 0-127, saving ~14MB - Remove cached delimiter field from DOMCharacterData (computed on demand) - Add trimToSize compaction for XMLNodeList and XMLNamedNodeMap - Reduce default capacity of XMLNodeList from 10 to 2 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use a persistent StringBuilder buffer for incremental text updates instead of creating a new one on every edit. Lazy toString() caching defers String materialization until getText() is actually called. - Persistent textBuffer: eliminates ~2.7 GB of StringBuilder allocations - Lazy cachedText: halves toString() calls (114 → 51 per 30s session) - GC events reduced from 77 to 52 (-32%) - TextDocument.update no longer the #1 allocator Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
I tested the PR against https://samplelib.com/xml/sample-30mb.xml in vscode, it crashed the server with an OOM, I doubled the Xmx to 512MB, I was removing a bunch of closing tags to test diagnostics, it worked fine when working at the end of the document, but crashed the server again when generating diagnostics in the middle of the file (~L 70000). When it reopened (no Xmx change), I went to modify the file in the middle again, it straight up crashed. |
Contributor
Author
|
Thanks for your feedback @fbricon ! My current PR should improve a lot the memory since it manages incremental parser. Now the memory problem comes from the full string text which is stored in memory. When didChange occurs it update the string content with stringbuilder which is fast, but when toString from this string builder is called memory is growing up a lot I am investigating the issue Have you tried to open yiur large xml file with released lemminx? It should be super slow when you open completion and oom should occurs faster, right? |
Replace getText() with getTextSequence() across 44 files so the parser, formatter, completions, and extensions read the StringBuilder directly via CharSequence — zero-copy during rapid typing. Only callers that need String for external APIs (Xerces InputSource, DOMParser.parse(String)) retain getText(). Also replace regex-based #region detection in XMLFoldings with char-by-char scanning. JFR profiling confirms LemMinX allocations dropped out of the top allocators entirely (0 GC events in 30s vs 77 before optimizations). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Intern tag strings in GreenTreeBuilder to deduplicate ~1.7M Strings - Replace XMLNodeList<DOMNode> (ArrayList) with DOMNode[] arrays - Store GreenElement reference in DOMElement instead of caching tag/startTagCloseOffset/endTagOpenOffset/endTagCloseOffset fields - Pre-allocate children array in RedTreeBuilder.addChildren() - Make lazyGreenNode volatile for thread-safe double-checked locking - Reduces DOM memory from ~700MB to ~260MB for 30MB XML files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
buffering When getText() materializes the cached String from the StringBuilder, null out textBuffer to free the duplicate byte[] backing array. Reduces byte[] memory from ~66MB to ~24MB for large XML files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use buildLazy for initial parse (not just incremental), so red tree nodes are materialized on demand instead of all at once - Remove unused contentStartRel field from GreenElement (48→40 bytes) - Clear previousIncrementalData right after incremental parse to allow earlier GC of old green tree (~58MB freed sooner) - Fix DTDAttlistDecl.getInternalChildren() and hasChildNodes() to properly trigger lazy expansion - Fall back to eager build when ignoreWhitespaceContent=false (tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Merge rapid edits into a single dirty range against the original green tree instead of overwriting pendingEdit (which caused coordinate mismatch and DOM corruption after rapid typing) - Reduce validator thread pool from 2 to 1 to limit concurrent tree retention during reparse - Fix GreenElement.childrenStartRel() for unclosed start tags - Invalidate cached model in update() via cancelModel() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
node scope - Replace XMLNamedNodeMap with DOMAttr[] to eliminate 240K wrapper objects (-5.7 MB) - Consolidate DOMAttr fields: reuse inherited start/end/parent instead of duplicate nameStart/nameEnd/ownerElement (80→64 bytes per attr, -8.7 MB) - Move lazyGreenNode/lazyAbsStart from DOMNode to DOMElement/DTDDeclNode so leaf nodes (DOMAttr, DOMText) don't carry unused fields - Use Predicate instead of Function<T,Boolean> in findFirst to avoid autoboxing - Use unsigned right shift for binary search midpoint calculation - Add DOMMemoryOptimizationsTest (21 tests) and MemoryBenchmark utility Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove endTagCloseRel field from GreenElement, derive from width and extraFlag (GreenElement 40→32 bytes, saves ~2 MB for 246K elements) - Add EXTRA_FLAG to GreenNode for endTagHasClose boolean encoding - Optimize IncrementalParser to handle text-only edits without full reparse when the edited region contains no '<' characters - Update DOMParserTest helper to match new GreenElement constructor Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DOMAttr (546K instances) inherited an unused attributeNodes field from DOMNode. Moving it to DOMElement and DOMProcessingInstruction (the only types that use attributes) drops DOMAttr from 64 to 56 bytes per instance. Verified with heap histogram: 546K × 8B = 4.4 MB saved. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migrate ILineTracker.set() and all implementations from String to CharSequence so createLineTracker() uses getTextSequence() instead of getText(), avoiding StringBuilder-to-String conversion. Add CharSequenceReader for SAX InputSource, migrate all hot-path getText() callers (DTDValidator, XSDValidator, DOMUtils, XMLSyntaxErrorCode, DOMDocument.getTrimmedRange) to getTextSequence(). Deprecate getText() on TextDocument and DOMDocument. Add compact children storage for GreenElement (null/single/array) to avoid GreenNode[] allocation for single-child elements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.