Performance: reconsider sync.Pool for gzip

[howardjohn]

sync.Pool has a somewhat surprising behavior that it has processor-local copies (https://victoriametrics.com/blog/go-sync-pool/). So on large machines (mine has GOMAXPROCS=32 default), we can end up with 32 copies of items stored in sync.Pool - even without concurrent usage.

Prometheus uses a pool for gzip:

client_golang/prometheus/promhttp/http.go

Line 660 in 9bf26f7

gz := gzipPool.Get().(*gzip.Writer)

. With 32 copies of the gzip writer, we end up with ~10mb heap usage. Which isn't crazy but is a pretty substantial amount for an otherwise very small application doing very little allocations/GC, and serving even a tiny number of metrics.

The reason sync.Pool does this is to improve throughput by avoiding synchronization. However, in a scraping setup I would think that its very uncommon to have much concurrency (maybe you have a couple of scrapers, but I'd imagine the median is 1 scraper)

Below shows my application's memory usage after disabling gzip due to this:

So at least in my specific case, it seems like it might be more optimal to use a sync.Pool-like structure that doesn't have this per-processor copying mechanism.

[srinathshrestha]

hmmmm. interesting , the way i see it is , it's a tuning task , looking for that sweet number of writers for a most typical workloads , cuz the sync.Pool can retain writers across processor-local pools after the trafic has become quite. and you are also right about the fact that for a machine with high GOMAXPROCS this can cause idle compression memory that cloud have been some other scrape's

A burst could still create as many writers as needed, while only a smaller number would be retained afterward. The trade-off is that later bursts may need to create writers again, so reducing idle memory could cost allocations and CPU under repeated high concurrency.

I’d like to reproduce the retained-memory behavior and compare the current implementation with a bounded-reuse prototype. I would measure retained heap after traffic becomes idle, along with allocations and throughput at several concurrency levels. I can also test multiple retention limits.

Does this aligns with the intended plan for the issue? If so, I’m happy to give it a try and share the benchmark results. and may be a some designs .