Repository navigation
Keep default parsers when only some are overridden - #178
Merged
Merged
Conversation
readRowGroup built columnDecoder with the merged parsers first and then spread the raw options on top, so whenever a caller passed a parsers object it overwrote the merge and every default parser was lost. Reading a file with a date, timestamp, UUID, geometry or JSON column then threw "parser is not a function". Moving the merge below the two spreads makes a partial parsers object override only the named parsers. The public parsers option is now Partial<ParquetParsers>, since a partial object was always the intended input but the declared type demanded all nine. The three functions that merge the defaults now keep the merged set in a local, because a reassigned Partial parameter stays Partial for TypeScript. prefetchPageIndexes only needed its JSDoc widened, since it forwards parsers straight to readColumnIndex.
platypii
approved these changes
Sep 16, 2026
platypii
left a comment
Collaborator
There was a problem hiding this comment.
I agree this is a better api. Thanks @MaxFreedomPollard
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #175. A partial
parsersobject throws instead of overriding only the parsers it names.await parquetReadObjects({ file, parsers: { stringFromBytes: () => 'custom' } })ontest/files/duckdb4442.parquetfails withTypeError: parser is not a function, and the same call ontest/files/uuid.parquetfails withparsers.uuidFromBytes is not a function. Any column whose type needs a parser becomes unreadable the moment a custom parser is supplied for anything else.readRowGroupbuildscolumnDecoderat src/rowgroup.js line 28 on master: line 32 merges{ ...DEFAULT_PARSERS, ...options.parsers }and line 33 spreads...optionsover it, so the caller's rawparsersreplaces the merged object and the eight parsers the caller did not name are gone.convert()then readsparsers.timestampFromNanoseconds(src/convert.js line 155) off the partial object forcall_date, which is INT64TIMESTAMPwith unitNANOS, and getsundefined. No existing test passes a partialparsersobject through a read, so nothing tripped it.The fix moves that one line below
...optionsand...chunk.columnMetadata, so the merge is computed last, matching howassembleAsync(src/rowgroup.js line 229),parquetMetadata(src/metadata.js line 91) andreadColumnIndex(src/indexes.js line 17) already merge after their parameters are bound. The type change is not optional:parsers?: ParquetParsersat src/types.d.ts lines 24 and 44 demanded all nine, so without widening itnpx tscfails on the new test with TS2740, and widening it makes those three functions hold the merged set in a local, because a reassignedPartialparameter staysPartial.prefetchPageIndexesforwardsparsersstraight toreadColumnIndex, so only its JSDoc at src/plan.js line 298 widened. On the published types,Partial<ParquetParsers>only breaks a consumer who reads the option back out as a fullParquetParsers;ColumnDecoder.parsersat src/types.d.ts line 533 stays the full set, and the fix is what now guarantees it gets all nine. Nothing else moves:columnDecoderis built in exactly one place andchunk.columnMetadatacarries noparserskey.Verification. The new test
keeps default parsers for types a custom parser does not overridein test/read.test.js readstest/files/duckdb4442.parquetwithparsers: { stringFromBytes: () => 'custom' }and assertscall_typeis'custom'whilecall_dateis stillnew Date('2011-10-06T22:21:49.580Z'), one assertion for the override landing and one for the default surviving. It fails on master withTypeError: parser is not a functionat src/convert.js line 158 and passes with the change.npx vitest rungives 664 passed in 30 files,npm run coverageis green at 92.21% statements, andnpm run lintandnpx tscare clean.