Use upb for protobufs

[lfittl]

TODO

• Split out func changes to a separate earlier commit (they don't strictly require upb)
• Additional manual review/rewording

Benchmark

Before:

benchmark/bench_protobuf
[?] iters=20000 queries=10 ops=200000
  encode (parse_protobuf):    3.677 s    18385.1 ns/op       54392 ops/s
  decode (deparse_protobuf):  2.359 s    11796.3 ns/op       84772 ops/s
  protobuf bytes/corpus pass: 3166 (per-op avg 316.6)
  maxrss: 5.8 MB    (sink=87520000)
benchmark/microbench_protobuf
[?] iters=50000 queries=10 ops=500000
  ENCODE nodes_to_protobuf:    8.605 s   17210.3 ns/op       58105 ops/s
  DECODE protobuf_to_nodes:    5.557 s   11114.8 ns/op       89970 ops/s
  bytes/corpus pass: 3166   (sink=2835410448700000)

After:

benchmark/bench_protobuf
[?] iters=20000 queries=10 ops=200000
  encode (parse_protobuf):    1.417 s     7082.7 ns/op      141189 ops/s
  decode (deparse_protobuf):  0.554 s     2767.9 ns/op      361287 ops/s
  protobuf bytes/corpus pass: 3166 (per-op avg 316.6)
  maxrss: 3.8 MB    (sink=87520000)
benchmark/microbench_protobuf
[?] iters=50000 queries=10 ops=500000
  ENCODE nodes_to_protobuf:    2.958 s    5916.2 ns/op      169028 ops/s
  DECODE protobuf_to_nodes:    0.905 s    1809.2 ns/op      552736 ops/s
  bytes/corpus pass: 3166   (sink=2596340496700000)

[benasher44]

Here's a benchmark I put together (with AI help, but then I did an edit pass to try and tone down the typical AI drama). There are 2 main findings:

1. The decode speedup reproduces. 4.1–5.5× over stock 18.0.0 at the C level, consistent
   with the 4.3× in the PR description.
2. This PR rejects parse trees that stock accepts. upb's decoder enforces a default depth
   limit of 100 that protobuf-c did not, and pg_query_ParseResult_parse passes no decode
   options. A query with ~93 or more UNION ALL branches fails with "could not parse protobuf"
   where stock deparses it fine.

Setup

Machine	Apple M4 Max (16 core), macOS 26.6.1 (darwin-arm64), clang -O2
Stock	pganalyze/libpg_query tag 18.0.0
upb	this PR, head 8f3319b
Patched	18.0.0 + two local patches to protobuf-c (described in §5)
Node	v24.11.1
Payloads	45 protobuf parse trees: a 42-statement corpus plus three shapes (168 B, 297 B, 2.6 KB)

All three builds produce identical deparsed SQL on every payload they all accept. Timings are medians of repeated rounds, not single means.

1. Depth limit: a behavioral regression vs stock

UNION ALL nests left-deep, so branch count is a direct depth knob. Encoding the same tree and
handing it to each build:

UNION ALL branches	wire size	stock	upb
82	1.6 KB	ok	ok
92	1.8 KB	ok	ok
94	1.9 KB	ok	FAIL
100	4.0 KB	ok	FAIL
200	8.1 KB	ok	FAIL

Fails in ubp are due to hitting the depth limit.

2. C-level deparse (pg_query_deparse_protobuf)

Median of 9 rounds × 4000 iterations.

payload	stock	patched	upb	patched/stock	upb/patched
168 B	6423 ns	1600 ns	1574 ns	4.01×	1.02×
297 B	12392 ns	2990 ns	2794 ns	4.14×	1.07×
2.6 KB	112064 ns	23103 ns	20404 ns	4.85×	1.13×

upb is 4.1–5.5× faster than stock

3. End to end from a JS parse tree

JS-side protobuf encode + the native call. The three native columns share one encoder, so
differences between them are C-side only. js is
pgsql-deparser 18.3.6, a hand-written
TypeScript reimplementation of deparseRawStmt — included because it's what we're trying to replace for correctness.

Microseconds per call, median of 7 rounds:

case	stock	patched	upb	js	upb/patched	patched/js
SELECT 1	2.0	1.0	1.1	0.8	0.99×	1.36×
simple select	8.4	3.4	3.3	3.2	1.05×	1.07×
CREATE TABLE	7.9	3.9	3.5	4.0	1.10×	0.96×
join	11.5	4.5	4.4	2.7	1.02×	1.66×
CTE	16.4	6.5	5.9	5.1	1.09×	1.27×
100-column projection	136.9	43.3	40.3	69.0	1.07×	0.63×
50-way UNION ALL	86.1	50.1	47.4	40.7	1.06×	1.23×
~340 KB tree	3391	1054	965	1686	1.09×	0.63×

4. Memory

Peak RSS deparsing a 2.5 MB shallow parse tree (1500 independent statements — shallow so
that all three builds actually decode it):

	C-level maxrss	end-to-end RSS growth, 20 deparses
stock	57.2 MB	86 MB
patched	90.7 MB	158 MB
upb	49.6 MB	97 MB
js	—	162 MB

upb is best at the C level but only by ~1.15× over stock, and slightly behind stock end to end (not by much).

5. What "patched" is

Two local patches, included only to answer "how much of upb's win is protobuf-c being fixable?" (aside from other issues that ubp is fixing)

1. palloc patch in Deparse: unpack the protobuf with palloc #360

2. Skip protobuf-c's post-scan field loop when it's a no-op (here and here)

Caveats

• Single machine, darwin-arm64 only. No Linux or musl numbers.
• Deparse only. We encode protobuf in JS, so pg_query_parse_protobuf is untested here; the
  PR's own encode numbers are not something we can confirm.
• The upb build of our Node addon has scan() stubbed — it was our only protobuf-c consumer and
  is unrelated to deparse.

[lfittl]

@benasher44 Thanks for testing - glad to see the positive CPU/memory impact of upb reproduced! (if I read the results correctly)

On the depth limit: That's actually one motivating factor for utilizing upb (to solve a bunch of OSS-fuzz complaints), but I don't see a problem with making that customizable. You had a notion of "suggested fix" for that above, but looks like that was edited out. Can you clarify what would be ideal for you on that end?

[benasher44]

Oh yep sorry. There was an LLM suggested fix, but IIRC it was vague. Yeah it'd be great to expose an API to change the depth limit. So I agree making it customizable would be great.

I'm not 100% sure, but it sounds like protobuf-c had no depth limit? I agree it makes sense to have some depth limit… maybe set the default really high for existing users to avoid breakage

[lfittl]

Oh yep sorry. There was an LLM suggested fix, but IIRC it was vague. Yeah it'd be great to expose an API to change the depth limit. So I agree making it customizable would be great.

Makes sense!

I'm not 100% sure, but it sounds like protobuf-c had no depth limit? I agree it makes sense to have some depth limit… maybe set the default really high for existing users to avoid breakage

Yeah, protobuf-c did not limit the depth at all. I think we could set it to some value between 200 and 500 safely - the main challenge is not running into an out of stack error with the default setting (but I'm fairly certain 200 would be safe vs 100).

[seanlinsley]

I'm not sure what default limit makes the most sense, but we use a recursion limit of 1,000 when deserializing the protobuf in the Rust crate: pganalyze/pg_query.rs#17

[benasher44]

500-1000 would be great, but as long as it could be configured, I think we'd be happy. Excited for this to land!