onnx-community/OpenMed-PII-SuperClinical: q8 silently returns near-chance logits; fp16 and q4f16 fail to load

[matthewchung74]

Summary

Three of the five ONNX variants published for onnx-community/OpenMed-PII-SuperClinical-Base-184M-v1-ONNX are unusable. Two fail loudly at session creation; one loads fine and silently returns near-chance predictions, which is the reason I'm filing — a consumer of q8 gets a pipeline that appears healthy while detecting essentially nothing.

Variant	File	Size	Result
fp32	model.onnx	704 MB	✅ works — max score 1.00, correct labels
q4	model_q4.onnx	496 MB	✅ works — max score 0.999, correct labels
q8	model_quantized.onnx	232 MB	❌ loads, but predictions are noise
fp16	model_fp16.onnx	369 MB	❌ fails to load (float16/float32 cast mismatch)
q4f16	model_q4f16.onnx	275 MB	❌ fails to load (same class of error)

Environment

• @huggingface/transformers 4.2.0
• onnxruntime-node 1.24.3
• Node v23.6.1, macOS (Darwin arm64)

All three failures reproduced in Node. The q8 behaviour and the q4 fix were additionally confirmed in-browser with the same library version from jsDelivr; fp16 and q4f16 were not retested in a browser.

Reproduction

import { pipeline } from "@huggingface/transformers";

const MODEL = "onnx-community/OpenMed-PII-SuperClinical-Base-184M-v1-ONNX";
const text = "Patient: Jane Q. Testpatient DOB: 03/14/1985 MRN: TESTMRN-424242 seen by Dr. Alice Smith at 123 Canary Lane.";

for (const dtype of ["q8", "q4", "fp16", "q4f16", "fp32"]) {
  try {
    const p = await pipeline("token-classification", MODEL, { dtype });
    const out = await p(text, { ignore_labels: ["O"] });
    console.log(dtype, out.length, "entities, max score",
      out.length ? Math.max(...out.map(o => o.score)).toFixed(3) : "n/a");
  } catch (e) {
    console.log(dtype, "FAILED:", e.message);
  }
}

1. q8 — loads, but output is near-chance and mislabelled

[ { entity: "I-phone_number",               score: 0.042, word: "Jane"  },
  { entity: "B-certificate_license_number", score: 0.033, word: "Alice" } ]

Two spans, both mislabelled, both at roughly chance — Jane as a phone number, Alice as a licence number. No error is raised.

Same input and code with dtype: "q4":

11 entities, max score 0.999
Jane        -> B-first_name    @ 1.00
03/14/1985  -> date_of_birth   @ 0.99
Alice, Smith, Robert, Chen  all detected as names

The near-uniform softmax under q8 suggests quantization damaged the classifier head rather than a tokenizer or config problem.

2. fp16 — fails to load

Load model from .../onnx/model_fp16.onnx failed:
Type Error: Type (tensor(float16)) of output arg (/deberta/embeddings/Cast_output_0)
of node (/deberta/embeddings/Cast) does not match expected type (tensor(float)).

The Cast node at the DeBERTa embeddings emits float16 where the consumer expects float32, so the graph fails type-checking at session creation. The file downloads completely (369 MB), so this isn't a truncated artifact.

3. q4f16 — fails to load

Same failure mode as fp16 (mixed-precision cast mismatch). File is 275 MB and downloads completely.

Why this matters for web deployment

q8 (232 MB) and fp16 (369 MB) are the two sizes most attractive for browser use. With both unusable, the smallest working option is q4 at 496 MB — more than double the broken q8. We shipped q4 for an on-device PII redaction feature and it works well, but the 500 MB first load is a real cost.

The silent q8 failure is the expensive part: our redaction layer appeared to run for some time while catching nothing, and the only symptom was occasional garbled output where a low-confidence span landed mid-word.

Suggested fixes

1. Re-export q8 and verify the classifier head survives quantization. A smoke test asserting max score > 0.5 on a labelled example would catch this class of regression across conversions.
2. Re-export fp16 / q4f16 with consistent precision through the embeddings Cast.
3. Failing either, note the broken variants on the model card so consumers pick q4 or fp32.

Related: #1707 reports a similar session-creation failure for another onnx-community q8 artifact.