Skip to content

Optional struct/list/map fields resolve to undefined instead of null #168

Description

@cas--

While parsing an optional struct I was surprised to find hyparquet was returning undefined instead of null when the struct was assigned null by duckdb.

Reproduction

Tested hyparquet 1.25.6 and 1.26.2 with the following:

import { parquetWriteBuffer } from 'hyparquet-writer'
import { parquetMetadataAsync, parquetReadObjects } from 'hyparquet'

const schema = [
  { name: 'root', num_children: 2 },
  { name: 'scalar', type: 'INT32', repetition_type: 'OPTIONAL' },
  { name: 'struct', num_children: 1, repetition_type: 'OPTIONAL' },
  { name: 'a', type: 'INT32', repetition_type: 'OPTIONAL' },
]

const arrayBuffer = parquetWriteBuffer({
  schema,
  columnData: [
    { name: 'scalar', data: [null] },
    { name: 'struct', data: [null] },
  ],
})

const file = {
  byteLength: arrayBuffer.byteLength,
  slice: (start, end) => arrayBuffer.slice(start, end ?? arrayBuffer.byteLength),
}
const metadata = await parquetMetadataAsync(file)
const rows = await parquetReadObjects({ file, metadata })

rows[0].scalar // null
rows[0].struct // undefined — inconsistent, and drops silently from JSON.stringify

Root cause

The issue is flattenAtDepth in src/assemble.js which unwraps via arr[i][0] and is undefined for an empty array, unlike assembleLists, which explicitly pushes null for absent scalars.

I can create a PR with the fix and updated tests:

--- src/assemble.js
+++ src/assemble.js
@@ -196,7 +196,7 @@ function flattenAtDepth(arr, depth) {
     if (depth) {
       flattenAtDepth(arr[i], depth - 1)
     } else {
-      arr[i] = arr[i][0]
+      arr[i] = arr[i].length ? arr[i][0] : null
     }
   }
 }

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions