Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions directory.json
Original file line number Diff line number Diff line change
Expand Up @@ -864,10 +864,6 @@
"title": "Collector crashes",
"path": "/docs/install/troubleshooting/collector_crash"
},
"install/troubleshooting/collector_times_out": {
"title": "Collector times out",
"path": "/docs/install/troubleshooting/collector_times_out"
},
"install/troubleshooting/column_stats_helper": {
"title": "Resolving the \"Limited access to table column statistics\" warning",
"path": "/docs/install/troubleshooting/column_stats_helper"
Expand All @@ -888,6 +884,10 @@
"title": "Amazon RDS: Resolving the \"pg_stat_statements must be loaded via shared_preload_libraries\" error",
"path": "/docs/install/troubleshooting/rds_pg_stat_statements_shared_preload_libraries"
},
"install/troubleshooting/reset_stat_statements": {
"title": "Reducing memory usage and query text unavailable",
"path": "/docs/install/troubleshooting/reset_stat_statements"
},
"log-insights": {
"title": "pganalyze Log Insights",
"path": "/docs/log-insights"
Expand Down
2 changes: 1 addition & 1 deletion install/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ backlink_title: 'Installation Guide'
This section provides helpful information about troubleshooting your pganalyze installation:

* [Collector crashing](/docs/install/troubleshooting/collector_crash)
* [Collector times out](/docs/install/troubleshooting/collector_times_out)
* [Reducing memory usage and query text unavailable](/docs/install/troubleshooting/reset_stat_statements)
* [Amazon RDS: Resolving the "pg\_stat\_statements must be loaded via shared\_preload\_libraries" error](/docs/install/troubleshooting/rds_pg_stat_statements_shared_preload_libraries)
* [Resolving the "Limited access to table column statistics" warning](/docs/install/troubleshooting/column_stats_helper)
* [Resolving the "missing extended stats monitoring helper functions" warning](/docs/install/troubleshooting/ext_stats_helper)
Expand Down
4 changes: 4 additions & 0 deletions install/troubleshooting/collector_crash.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ If you find the collector crashing, the most likely cause is running out of memo
a small instance size by default as that is enough for the typical case, but with a very large
schema or a large or diverse query workload, that may not be sufficient.

Before increasing the collector's memory limit, you may want to enable
[automatic pg_stat_statements resets](/docs/install/troubleshooting/reset_stat_statements)
to reduce memory usage from query text.

Collector memory may be limited either through available instance memory, or through systemd (if
your platform uses it).

Expand Down
54 changes: 0 additions & 54 deletions install/troubleshooting/collector_times_out.mdx

This file was deleted.

46 changes: 46 additions & 0 deletions install/troubleshooting/reset_stat_statements.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
title: Reducing memory usage and query text unavailable

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title: Reducing memory usage and query text unavailable
title: Query text unavailable and reducing memory usage

backlink_href: /docs/install/troubleshooting
backlink_title: 'Installation Troubleshooting'
---

If the collector is using too much memory or `<query text unavailable>` is showing up in pganalyze,
you may want to set up an automatic reset of pg_stat_statements.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
you may want to set up an automatic reset of pg_stat_statements.
you may want to set up an automatic reset of `pg_stat_statements`.


## &lt;query text unavailable&gt;

Some versions of Postgres aren't able to accurately normalize queries (e.g. those with variable bind params),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Some versions of Postgres aren't able to accurately normalize queries (e.g. those with variable bind params),
Earlier versions of Postgres aren't able to group certain queries together (e.g. those using `IN` lists of variable lengths),

leading to many duplicate entries that often have very low call counts. When `pg_stat_statements.max` is reached,
Postgres will deallocate (remove) the entries with the lowest call count. In periods of high database activity

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Postgres will deallocate (remove) the entries with the lowest call count. In periods of high database activity
Postgres will deallocate (remove) entries based on their usage, which is based on the call count. In periods of high database activity

this can happen so so quickly that the collector isn't able to capture the query text, causing
`<query text unavailable>` to show up in pganalyze.

This can be addressed by enabling the Maximum Query Stats Reset Frequency setting. We recommend a setting of

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This can be addressed by enabling the Maximum Query Stats Reset Frequency setting. We recommend a setting of
This can be addressed by enabling the "Maximum Query Stats Reset Frequency" setting in pganalyze. We recommend a setting of

"Every 1 hour" unless you're specifically concerned about lock contention on a very busy database. This enables

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Every 1 hour" unless you're specifically concerned about lock contention on a very busy database. This enables
"Every 1 hour" unless you're specifically concerned about lock contention (`LWLock:pg_stat_statements`) on a very busy database. This enables

automatic pg_stat_statments resets to be done once the statement space is 90% full (so a deallocation would occur soon),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
automatic pg_stat_statments resets to be done once the statement space is 90% full (so a deallocation would occur soon),
automatic pg_stat_statments resets once the statement space is 90% full (so a deallocation would occur soon),

but resets will not occur more often than the frequency setting you choose.

## High memory usage

Every 10 minutes, the collector loads the query text from pg_stat_statements into memory in order to normalize and
fingerprint it. This is a very expensive step and can result in the collector crashing from running out of memory.
To address that, when resets are enabled with the Maximum Query Stats Reset Frequency setting, the collector will

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To address that, when resets are enabled with the Maximum Query Stats Reset Frequency setting, the collector will
To address that, when resets are enabled with the "Maximum Query Stats Reset Frequency" setting, the collector will

also perform a reset if the size of the query text grows beyond 250 MB. You can optionally increase that limit

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
also perform a reset if the size of the query text grows beyond 250 MB. You can optionally increase that limit
also perform a reset if the aggregate size of the query text returned from pg_stat_statements grows beyond 250 MB. You can optionally increase that limit

using the Maximum Query Size Before Next Reset setting.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
using the Maximum Query Size Before Next Reset setting.
using the "Maximum Query Size Before Next Reset" setting, which can be advisable when you have very large query texts.


## Setup

Collector version 0.70.0 or later is required.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could link this to the collector release changelog.


After enabling the Maximum Query Stats Reset Frequency setting on the server settings page (and optionally tuning

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd be good to keep a server settings page screenshot here, possibly with the relevant two settings highlighted with a red border or similar to make it very clear what needs to be changed.

Maximum Query Size Before Next Reset), the collector will automatically start performing resets when needed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
After enabling the Maximum Query Stats Reset Frequency setting on the server settings page (and optionally tuning
Maximum Query Size Before Next Reset), the collector will automatically start performing resets when needed.
After enabling the "Maximum Query Stats Reset Frequency" setting on the server settings page (and optionally tuning
"Maximum Query Size Before Next Reset"), the collector will automatically start performing resets when needed.

You should see evidence of that in the collector logs.

If the collector fails to reset with permissions errors, you may need to define this function:

<CodeBlock language="sql">
{`CREATE OR REPLACE FUNCTION pganalyze.reset_stat_statements() RETURNS SETOF void AS
$$
/* pganalyze-collector */ SELECT * FROM public.pg_stat_statements_reset();
$$ LANGUAGE sql VOLATILE SECURITY DEFINER;`}
</CodeBlock>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should put a section after this that talks about reducing pg_stat_statements churn, and specifically references how different Postgres versions have addressed different problems that can occur (and explain when upgrading helps, and to which version). That would also be a good place to talk about pg_stat_statements.max, which I think is still relevant as a fix for this kind of problem.

Binary file removed install/troubleshooting/server_settings.png
Binary file not shown.
4 changes: 2 additions & 2 deletions install/troubleshooting/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
items:
- name: Collector crashes
href: install/troubleshooting/collector_crash
- name: Collector times out
href: install/troubleshooting/collector_times_out
- name: Reducing memory usage and query text unavailable
href: install/troubleshooting/reset_stat_statements
- name: 'Resolving the "Limited access to table column statistics" warning'
href: install/troubleshooting/column_stats_helper
- name: 'Resolving the "missing extended stats monitoring helper functions" warning'
Expand Down