Skip to content

P4Prometheus.adoc

Latest commit

 

History

History
157 lines (104 loc) · 7.17 KB

P4Prometheus.adoc

File metadata and controls

157 lines (104 loc) · 7.17 KB
Raw

Perforce P4Prometheus - Overview

Table of Contents

Support

Preface

P4Prometheus integrates Perforce Helix Core Server (p4d) with Prometheus and related monitoring tools. It collects real-time metrics from p4d logs, p4 monitor, and related sources, then exposes them for Prometheus collection and Grafana dashboards. The metrics can also drive system alerting.

The p4prometheus service continuously parses p4d logs and writes Prometheus textfile-collector metrics. Other project components interrogate p4d and associated logs for supplementary metrics.

1. Overview

P4Prometheus is part of a monitoring solution comprising these components:

  • Prometheus: time-series metrics management.

  • VictoriaMetrics: recommended, Prometheus-compatible long-term metrics storage.

  • Grafana: metrics dashboards and visualization.

  • node_exporter: Linux host metrics and textfile collection.

  • windows_exporter: Windows host metrics.

  • Alertmanager: alert routing, grouping, and de-duplication.

  • Pint: Prometheus alerting rules linter - fork of Cloudflare original.

In addition to these components you can create a monitoring solution tailored to your environment, with automation and alerting including:

  • Alerts being sent to Slack (or Teams/Email/PagerDuty)

    • Such alerts containing links to relevant Grafana dashboards, and a Runbook for incident response, help ensure timely and effective handling of incidents.

    • Options for alerting include notifications of blocked commands, and automated updates when blocks reduce

  • Monitoring of Memory usage, and optional OOM-kill events (Out-Of-Memory killer) on Linux hosts.

Project components:

  • p4prometheus: a binary that parses P4 Server logs.

  • p4metrics: supplementary P4 Server and SDP-compatible metrics.

  • monitor_locks.py and p4monitor_locks.sh: Linux lock monitoring using lslocks and p4 monitor show.

  • p4logtail and p4plogtail: completed-command log tails with JSON output (for use with third party log analysis tools such as Splunk, DataDog, ELK Stack etc.)

P4Prometheus uses go-libp4dlog for p4d log parsing.

The installation and upgrade scripts support standard installation, dedicated data volumes (-d /data), offline assets (--local-tarballs-dir), SDP-safe configuration in /p4/common/site/config/, hostname-specific HMS fleet configuration, and stateful non-destructive upgrades. See the installation guide for details.

Prometheus architecture

2. Support Status

This is a Community Supported Perforce tool.

3. Installation

Use the automated scripts for new installations and upgrades. The canonical guide is P4Prometheus Installation.

It covers monitoring-server, P4 Server, and node-exporter-only installations; upgrades; dedicated data volumes; air-gapped installations; state files; and manual component installation where scripts cannot be used.

4. Grafana Dashboards

When configured, Grafana dashboards provide command, replication, lock, and host-level views.

Commands Summary

Commands Summary

Command Rates and Durations

Commands

Active Commands

Active Commands

Replication Status

Replication Status

Read and Write Locks

Locks

Grafana dashboards can define alerts; Alertmanager processes their notifications. The installation guide includes setup and validation guidance, and Runbook Alert Handling provides a customizable operational runbook.

5. Metrics Available

All principal metrics carry serverid and, for SDP installations, sdpinst labels. The README remains the exhaustive metric-name reference: metrics available.

5.1. p4prometheus Metrics

The primary log parser reports command throughput, active commands, resource use, synchronization activity, command error counts, command durations, and lock wait/held time.

Metric family Examples

Parser health and throughput

p4_prom_log_lines_read, p4_prom_cmds_processed, p4_prom_cmds_pending

Command activity

p4_cmd_running, p4_cmd_counter, p4_cmd_cumulative_seconds

Resource use

p4_prom_cpu_user, p4_prom_cpu_system, p4_prom_memory

Command dimensions

p4_cmd_user_counter, p4_cmd_ip_counter, p4_cmd_replica_counter, p4_cmd_program_counter

Lock timings

p4_total_read_wait_seconds, p4_total_read_held_seconds, p4_total_write_wait_seconds, p4_total_write_held_seconds

5.2. p4metrics Metrics

p4metrics replaces the legacy monitor_metrics.sh metrics while retaining compatible names and labels. It reports active processes, memory, server identity and uptime, replication, journal/log rotation, license state, SDP job results, Swarm status, and optional memory-limit enforcement.

Metric family Examples

Process monitoring

p4_monitor_by_cmd, p4_monitor_by_state, p4_monitor_by_user, p4_monitor_max_cmd_time

Memory and limits

p4_active_memory_by_cmd, p4_active_memory_by_user, p4_memlimit_kill_candidates, p4_memlimit_kills_total

Replication

p4_pull_queue_total, p4_pull_replica_lag, p4_pull_replication_error

Server and license

p4_server_uptime, p4_p4d_build_info, p4_license_expires, p4_licensed_user_count

SDP jobs and logs

p4_sdp_checkpoint_error, p4_sdp_verify_errors, p4_journal_size, p4_log_size

Swarm

p4_swarm_error, p4_swarm_tasks, p4_swarm_workers, p4_swarm_version

5.3. Lock Metrics

Linux lock monitoring requires lslocks. p4monitor_locks.sh invokes monitor_locks.py, which relates operating-system locks to p4 monitor show results. It reports:

  • p4_locks_db_read and p4_locks_db_write

  • p4_locks_cliententity_read and p4_locks_cliententity_write

  • p4_locks_meta_read and p4_locks_meta_write

  • p4_locks_cmds_blocked

The detailed lock-monitoring installation and troubleshooting information is in P4Prometheus Installation.