HIVE-29785: SkippingTextInputFormat: ClassCastException on skip.header.line.count files with lone-CR (\r) line endings - #6680
Open
srbiswal wants to merge 1 commit into
Open
Conversation
…r.line.count files with lone-CR (\r) line endings SkippingTextInputFormat.getCachedStartIndex (header path) and getCachedEndIndex (footer path) called FSDataInputStream.readLine() and then getPos() on the same stream. On a lone '\r' not followed by '\n', DataInputStream.readLine() pushes its look-ahead byte back by replacing the stream's inner input with a non-Seekable PushbackInputStream, so the subsequent getPos() throws ClassCastException during Tez split generation. LF, CRLF and '\r'-at-EOF were unaffected. Replace readLine()+getPos() in both methods with a ByteCountingLineReader that recognizes '\n', '\r\n' and lone '\r' and derives offsets from bytes consumed, so getPos() is never called on a readLine()-mutated stream. Offset semantics are preserved exactly. As a safety net, makeSplitInternal now converts any unexpected RuntimeException from header/footer detection into a clear, file-contextual error instead of a cryptic cast. Adds tests: a lone-CR repro, exact split-offset assertions across LF/CR/CRLF proving no boundary regression, and a footer-path lone-CR case.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For tables with skip.header.line.count/skip.footer.line.count, SkippingTextInputFormat called readLine() then getPos() on the same FSDataInputStream (in getCachedStartIndex and getCachedEndIndex). When the data contains a lone \r (not followed by \n), DataInputStream.readLine() pushes back its look-ahead byte by wrapping the stream in a non-Seekable PushbackInputStream. The following getPos() then throws ClassCastException during Tez split generation. LF, CRLF, and \r-at-EOF were unaffected.
Replace readLine()+getPos() in both methods with a ByteCountingLineReader that handles \n, \r\n, and lone \r, computing offsets from bytes consumed, so getPos() is never called on a mutated stream. Offset behavior is unchanged. As a safety net, makeSplitInternal now turns any unexpected error from header/footer detection into a clear, file-specific message instead of a raw ClassCastException
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added unit tests in
TestSkippingTextInputFormat:testSkipFileSplitsLoneCR— reproduces theClassCastExceptionon a lone-CR file withskip.header.line.count=1(fails on master, passes with this fix).testSkipHeaderSplitOffsetsAcrossLineEndings— writes the same content with LF, lone-CR, and CRLF terminators and asserts exact splitgetStart()/getLength(), proving lone-CR matches LF and there is no boundary regression.testSkipFileSplitsLoneCRHeaderFooter— exercises the footer path (getCachedEndIndex) with lone-CR and skip header/footer, verifying the header and footer rows are skipped.