Skip to content

Skip known query texts from pg_stat_statements - #776

Draft
seanlinsley wants to merge 13 commits into
mainfrom
statements-skip-known
Draft

Skip known query texts from pg_stat_statements#776
seanlinsley wants to merge 13 commits into
mainfrom
statements-skip-known

Conversation

@seanlinsley

@seanlinsley seanlinsley commented Feb 9, 2026

Copy link
Copy Markdown
Member

Building on top of #768, this avoids loading query text from pg_stat_statements for known query IDs as a performance optimization.

Todo:

  • should the protobuf QueryStatistic use fingerprint instead of query index?
  • should <pganalyze-collector> grouping and IgnoreIoTiming be implemented on the server side?
  • should the collector clear its fingerprint cache periodically (e.g. once a day) to ensure query texts are accurately submitted to pganalyze?

SELECT dbid, userid, queryid, %s, query
FROM %s`
FROM %s
WHERE queryid = ANY($1)`

@lfittl lfittl Feb 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, doing this filtering has three main benefits that I can see:

  1. We avoid sending query text over the network (from the database to the collector) that we've already seen
  2. We avoid running normalize on query texts we've already seen
  3. We avoid sending query text to pganalyze that pganalyze already got

Just for clarity, because of how pg_stat_statements works today, this unfortunately won't reduce the effort done to read the query text file and put the query text and statistics into the tuplestore, since the WHERE condition won't be pushed down into pg_stat_statements_internal.

@lfittl

lfittl commented Feb 11, 2026

Copy link
Copy Markdown
Member

should grouping and IgnoreIoTiming be implemented on the server side?

As discussed, for <pganalyze-collector> it may make sense to shift this to the server side by sending the full query text and then instead implementing a UI filter like we have today, but keeping the full text on the backend for analysis. That would then mean you could use the cache like for any regular queries, and it has the added benefit of aiding debugging.

For IgnoreIoTiming, it does seem best to keep this on the collector side (and we can make that conditional on Aurora), and keep a mapping from query ID to a utility statement boolean.


Overall I'm not sure on this yet (and need to spend more time reviewing), specifically:

  1. How we deal with query texts being different between databases on the same server
  2. Other reasons the cache might have bad entries (do hash collisions matter? they probably don't?)
  3. How we deal with network issues when submitting snapshots (how can we be sure that the server side has stored the query text)

Base automatically changed from fingerprint-cache to main February 12, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants