Repository navigation
fix(scanner): clear heredoc stack in deserialize - #304
Merged
calebdw merged 7 commits intoSep 24, 2026
Merged
Conversation
Collaborator
|
Thanks! I'll take it from here 👍 |
`deserialize` reset every entry of `heredocs` but kept the array at its previous size, so leftover scanner-state entries were serialized again, 5 bytes each. One stale entry accumulated per heredoc. Once the 1024-byte serialization buffer filled up, `serialize` returned 0, the open heredoc's identifier was lost, and its closer could no longer match. Clear the array after resetting its entries. The reuse branch in the fill loop is then dead and is removed. Cover the failure with generated binding tests: 20 long-tag heredocs (the original overflow) and 250 short-tag ones.
Widen the Node peer range to tree-sitter ^0.25.0 so the package installs alongside 0.25.x. Bump tree-sitter-cli to 0.27 to match CI, refresh Cargo/PyPI versions, and ignore Python bytecode. Closes tree-sitter#301
`use \Foo\{Bar}` is valid PHP and already works for non-group
forms. `_namespace_use_group` required a bare namespace_name, so
the leading separator became an ERROR that swallowed the next
statement.
Closes tree-sitter#303
calebdw
force-pushed
the
fix/heredoc-stack-deserialize
branch
from
September 24, 2026 19:55
c920cca to
54f3194
Compare
PHP allows continuing the expression after a heredoc/nowdoc closer, e.g. SQL.PHP_EOL. The body scanners only stopped for whitespace, ';', ',', or ')', so a following '.' was consumed as string content and the rest of the file was swallowed. Closes the Laravel DatabaseSqliteSchemaStateTest.php example parse failure.
Laravel renamed Foundation/fixtures to Fixtures, so the known bad-syntax fixture was no longer excluded. Also skip PHPUnit's intentional parse-error fixture.
calebdw
force-pushed
the
fix/heredoc-stack-deserialize
branch
from
September 24, 2026 20:07
54f3194 to
9e651d6
Compare
calebdw
approved these changes
Sep 24, 2026
calebdw
force-pushed
the
fix/heredoc-stack-deserialize
branch
from
September 24, 2026 20:26
de485b9 to
b78c60c
Compare
Peer tree-sitter is now ^0.25.0, which cannot install with 0.22.x. That is a breaking change for Node consumers, so this is a minor bump rather than 0.24.3. ABI remains 15.
calebdw
force-pushed
the
fix/heredoc-stack-deserialize
branch
from
September 24, 2026 20:29
b78c60c to
12bf00f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A file with many sequential heredocs stops parsing at some point: the closing identifier is lexed as
string_content, the rest of the file ends up inside the heredoc body and the root becomesERROR.php -laccepts the file.Cause:
deserializeincommon/scanner.hresets every entry ofscanner->heredocsbut never shrinks the array. Each re-lex ofheredoc_startleaves one empty entry behind, andserializewrites all of them, 5 bytes each. Once the 1024-byte serialization buffer would overflow,serializereturns 0 and the open heredoc's identifier is lost.Where it fails depends on the tag length. A stale entry costs 5 bytes, the live one 5 + 4 * N for a tag of N characters (each character is stored on 4 bytes), plus 1 byte for the entry count. So the n-th heredoc of a file fails as soon as its tag reaches
ceil((1018 - 5 * (n - 1)) / 4)characters: a 3-character tag fails from the 203rd heredoc, a 240-character tag from the 13th.The test uses a long tag to keep the corpus short, but the file still grows a lot; I can move it to a separate
heredoc.txtnext tonowdoc.txtif you prefer. It is in its own commit, so you can check it out and see it fail before the fix. With the fix,tree-sitter testpasses for both grammars.