While parsing an optional struct I was surprised to find hyparquet was returning undefined instead of null when the struct was assigned null by duckdb.
Reproduction
Tested hyparquet 1.25.6 and 1.26.2 with the following:
import { parquetWriteBuffer } from 'hyparquet-writer'
import { parquetMetadataAsync, parquetReadObjects } from 'hyparquet'
const schema = [
{ name: 'root', num_children: 2 },
{ name: 'scalar', type: 'INT32', repetition_type: 'OPTIONAL' },
{ name: 'struct', num_children: 1, repetition_type: 'OPTIONAL' },
{ name: 'a', type: 'INT32', repetition_type: 'OPTIONAL' },
]
const arrayBuffer = parquetWriteBuffer({
schema,
columnData: [
{ name: 'scalar', data: [null] },
{ name: 'struct', data: [null] },
],
})
const file = {
byteLength: arrayBuffer.byteLength,
slice: (start, end) => arrayBuffer.slice(start, end ?? arrayBuffer.byteLength),
}
const metadata = await parquetMetadataAsync(file)
const rows = await parquetReadObjects({ file, metadata })
rows[0].scalar // null
rows[0].struct // undefined — inconsistent, and drops silently from JSON.stringify
Root cause
The issue is flattenAtDepth in src/assemble.js which unwraps via arr[i][0] and is undefined for an empty array, unlike assembleLists, which explicitly pushes null for absent scalars.
I can create a PR with the fix and updated tests:
--- src/assemble.js
+++ src/assemble.js
@@ -196,7 +196,7 @@ function flattenAtDepth(arr, depth) {
if (depth) {
flattenAtDepth(arr[i], depth - 1)
} else {
- arr[i] = arr[i][0]
+ arr[i] = arr[i].length ? arr[i][0] : null
}
}
}
While parsing an optional struct I was surprised to find hyparquet was returning
undefinedinstead ofnullwhen the struct was assignednullby duckdb.Reproduction
Tested hyparquet 1.25.6 and 1.26.2 with the following:
Root cause
The issue is
flattenAtDepthinsrc/assemble.jswhich unwraps viaarr[i][0]and isundefinedfor an empty array, unlikeassembleLists, which explicitly pushesnullfor absent scalars.I can create a PR with the fix and updated tests: