fix(tool): avoid UTF-8 misdetection at encoding sample boundaries - #8322
Open
VirgilMing wants to merge 2 commits into
Open
fix(tool): avoid UTF-8 misdetection at encoding sample boundaries#8322VirgilMing wants to merge 2 commits into
VirgilMing wants to merge 2 commits into
Conversation
…undary The native grep scanner detected encoding from an 8 KiB peek. When it ends mid UTF-8 sequence, the truncated prefix may be accepted as GB18030, so Chinese patterns miss and ASCII matches return mojibake. Select a character-safe detection prefix by preferring strict UTF-8 after removing at most a maximum-width tail, then retaining a prefix that is demonstrably GB18030. This fixes the reported UTF-8 failure without regressing ASCII headers followed by long GB18030 lines.
…GB18030 A no-newline 256 KiB sample ending mid multi-byte sequence was passed to fileenc.Detect untrimmed; the truncated UTF-8 tail is "valid" GB18030, so read_file decoded the whole stream into mojibake. Route the sample through the same detectSample char-boundary trim grep uses, closing the last encoding-detection gap both tools shared.
github-actions
Bot
added
v2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
grepandread_filedetect encodings from a character-safe bounded sample when the read boundary splits a multibyte sequenceIssues
Fixes #8299
Related: #8321 is a viable alternative implementation; its head also passes this PR's two inverse ASCII-header plus long-GB18030 cases, so the overlapping implementations remain separate pending maintainer direction.
Verification
go test -count=1 -run '^(TestGrepUTF8ChinesePeekBoundary|TestGrepUTF8ChineseLongSingleLine|TestGrepGB18030LongLineAfterASCIIHeader|TestReadFileUTF8ChineseLongSingleLine|TestReadFileGB18030LongLineAfterASCIIHeader|TestGrepGB18030|TestGrepGB18030TruncationDoesNotLeakGoroutine|TestReadFileGB18030|TestReadFileStreamsLargeGB18030)$' ./internal/tool/builtingo test -count=1 ./internal/fileutil/encodinggo vet ./internal/tool/builtin ./internal/fileutil/encodingKnown unrelated Windows baseline failures:
go test -count=1 ./internal/tool/builtin:TestBackgroundBashWaitAndOutput,TestBashTerminalFallsBackWhenUnhandled, andTestBashTerminalSkippedWhenEnvFilteringEnabledinvoke POSIXprintfafter falling back to PowerShellgo test -count=1 ./internal/boot:TestGoldenBaselineNoExtensionsis shell-environment-dependent on Windows: it fails when shell auto-resolution falls back to PowerShell and passes unchanged when Bash is available; this change does not modify any tool name, description, or schemainternal/cli,internal/hook, andinternal/shellrun, plusTestStreamSurfacesEarlyConnResetAsInterrupttiming out ininternal/provider/openai; all other packages in that split passedDocumentation impact
Documentation-impact: none - the documented tool contract is unchanged; this corrects encoding detection behind the existing behavior.
Cache impact
Cache-impact: none - tool names, descriptions, schemas, and provider-visible prompt content are unchanged.
Cache-guard: focused encoding regression tests plus a diff review of the tool contract methods.
System-prompt-review: N/A