Skip to content

perf(metrics): update the rpc_endpoint_* series positionally, without a label map per call (MAG-4244) - #609

Merged
AnnaR-prog merged 2 commits into
mainfrom
refactor/MAG-4244-positional-metric-labels
Nov 3, 2026
Merged

AnnaR-prog merged 2 commits into
mainfrom
refactor/MAG-4244-positional-metric-labels

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Jira ticket: MAG-4244

The third of MAG-4244's section 7 duplicates: the MappedLabels* metric wrappers. Sixteen rpc_endpoint_* metrics now update positionally. The metrics exposed are byte-identical to main's, and the per-relay updates now allocate nothing.

Why

The wrappers were written for "a dynamic number of labels based on flags given upon startup". No metric has that: every label set is fixed. Yet each update built a map[string]string of its labels, and the wrapper then built a []string to put them back in order. These run per relay (relays serviced, in flight) and per poll.

The change

  • The sixteen metrics are plain *prometheus.CounterVec / *prometheus.GaugeVec, registered through registerOrReuse like the metrics beside them, and updated with positional WithLabelValues.
  • The URL series get their values from urlSeriesLabelValuesLocked, used both for updates and for DeleteLabelValues when a URL is forgotten.
  • The three wrapper files and the wrapper's own test are deleted.

registerOrReuse is a true drop-in here, checked rather than assumed. All sixteen passed prometheus.DefaultRegisterer, which is the registry registerOrReuse uses, and both reuse an already-registered collector of the same type and panic otherwise.

Measured: allocations per update, steady state

Update main this PR
relay serviced 1 0
in-flight request 1 0
fetch latest success 1 0
tracker request 1 0
latest block 2 1
URL relay serviced 3 1

What could have gone wrong, and what rules it out

The wrapper mapped values by name, so it filled a missing label with "" and dropped an extra one. Positional WithLabelValues does neither, which opens two failure modes:

  1. The wrong number of values makes Prometheus panic. Every one of the 17 update sites supplied exactly its metric's declared keys, so nothing relied on either tolerance. The conversion derived each positional order from the declared labels and refused any key-set mismatch.
  2. A swapped order silently mislabels series the stuck-tip and per-URL alerts read. Ruled out by comparing output, not reasoning. A fixed sequence of updates through every wrapped metric renders 43 series whose names, help text, label sets and values are byte-identical to main's. The sequence includes a URL two providers share and a URL that is then forgotten. Only answered_until's value is redacted, because it derives from time.Now().

A durable guard for the new hazard

Label order now matters where it did not, so it gets a test. TestPositionalLabels_EveryWrappedMetric drives every metric with a different value in every label position and requires each value under its own name, then requires ForgetEndpointURLLatestBlock to remove the URL series.

It passes on main's code too, so it pins labels main already produces. Against six swaps in the new code, it fails every one:

Swap Caught
spec with apiInterface yes
endpoint_id with function, relays serviced yes
endpoint_id with function, in flight yes
endpoint_id with kind, tracker requests yes
url with internal_path, in the URL helper yes
a forget that deletes nothing yes

Two smaller notes

  • A nil-guard comment on AddEndpointTrackerRequest named the deleted wrapper type as its reason. The guard is still needed: prometheus.CounterVec.WithLabelValues dereferences its receiver too. So the comment now names that type instead.
  • Four test files reached into the wrappers. They now construct the plain vectors and read positionally. One of them, cancelled_relay_metrics_test.go, never named the wrapper type: it passed a label map through a helper, so a search for the type alone would have missed it.

Verification

From a detached worktree of the commit.

go build -buildvcs=false ./...                     clean
go vet ./protocol/...                              clean
go test -race ./protocol/metrics/                  ok
golangci-lint run ./protocol/metrics/...  (fresh cache)   0 issues

The fields are unexported, so nothing outside protocol/metrics can touch them, and the manager's public methods keep their signatures.

🤖 Generated with Claude Code

… a label map per call (MAG-4244)

Sixteen rpc_endpoint_* metrics went through MappedLabelsCounterVec and
MappedLabelsGaugeVec, wrappers written for "a dynamic number of labels
based on flags given upon startup". No metric has that: every label set
is fixed. Yet each update built a map[string]string of its labels, and
the wrapper then built a []string to put them back in order. These are
per-relay updates (relays serviced, in flight) and per-poll ones.

They are now plain *prometheus.CounterVec / *prometheus.GaugeVec,
registered through registerOrReuse like the metrics beside them, and
updated with positional WithLabelValues. The three wrapper files and the
wrapper's own test are deleted (MAG-4244 section 7).

registerOrReuse is a true drop-in here, checked rather than assumed: all
sixteen passed prometheus.DefaultRegisterer, which is the registry
registerOrReuse uses, and both reuse an already-registered collector of
the same type and panic otherwise.

Measured, steady-state allocations per update:

  relay serviced, in flight, fetch success, tracker request   1 -> 0
  latest block                                                2 -> 1
  url relay serviced                                          3 -> 1

What could go wrong, and what rules it out:

- A wrong count of positional values makes Prometheus panic, where the
  wrapper filled a missing label with "" and dropped an extra one. Every
  one of the 17 update sites supplied exactly its metric's declared keys,
  so nothing relied on either; the conversion derived each positional
  order from the declared labels and refused any key-set mismatch.
- A swapped order would mislabel series the stuck-tip and per-url alerts
  read, silently. Proved absent: a fixed sequence of updates through
  every wrapped metric, including a url that two providers share and a
  url that is then forgotten, renders 43 series whose names, help text,
  label sets and values are BYTE-IDENTICAL to main's (only
  answered_until's value is redacted, being time.Now()-derived).

Order now matters where it did not, so it gets a durable guard.
TestPositionalLabels_EveryWrappedMetric drives every metric with a
different value in every label position and requires each value under
its own name, then requires ForgetEndpointURLLatestBlock to remove the
url series. It passes on main's code too, so it pins the labels main
already produces. Against six swaps in the new code (spec with
apiInterface; endpoint_id with function, twice; endpoint_id with kind;
url with internal_path in the url helper; a forget that deletes nothing)
it fails every one.

urlSeriesLabelValuesLocked returns the url labels positionally for both
updates and DeleteLabelValues; its comment names the test that guards
its order. The nil guard comment on AddEndpointTrackerRequest named the
deleted wrapper type; it now names prometheus.CounterVec, whose
WithLabelValues dereferences its receiver all the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@avitenzer
avitenzer requested a review from a team as a code owner October 29, 2026 12:11

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b7d6146. I tested it merged with current main cb736b3.

The change is correct.

  • I drove every exported setter of the manager, with a different value in every label position, a url two providers share and a url that is then forgotten. The gathered page, every collector's descriptors and the registration state are byte-identical to c17c2cc's, and, merged, to main's.
  • TestPositionalLabels_EveryWrappedMetric passes on main's code. It fails when I swap endpoint_id and function in AddEndpointRelayServiced, and when urlSeriesLabelValuesLocked returns internal_path before the url.
  • testing.AllocsPerRun reproduces your table: 1 → 0 for relay serviced, in flight, fetch success and tracker request, 2 → 1 for latest block, 3 → 1 for url relay serviced. The url latest-block update also drops from 4 to 1.
  • urlSeriesLabelValuesLocked is only called under m.lock, and DeleteLabelValues removes the same series the old Delete(labels) did.
  • Merged with main: go build, go vet and go test -race ./protocol/metrics/ pass, and golangci-lint reports nothing new.

This conflicts with #604, which rewrites the same setters around map-based helpers. If this lands first, #604 keeps its helpers but positional: incCounter/setGauge plus an addGauge, no endpointSeries/endpointFunctionSeries, and the vectors built with registerOrReuse. I checked that combination: the page stays byte-identical, the allocations stay at this PR's numbers, and the positional test passes. #604's mapped_labels dupl rule then goes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 03f3039: main 924600c plus this PR. The metrics tests pass on this exact tree.

@AnnaR-prog
AnnaR-prog merged commit 2d2709e into main Nov 3, 2026
@AnnaR-prog
AnnaR-prog deleted the refactor/MAG-4244-positional-metric-labels branch November 3, 2026 18:01
AnnaR-prog added a commit that referenced this pull request Nov 3, 2026
Resolves this PR against #608, #609 and the WebSocket stack (#590, #591, #612, #614):
- protocol/metrics: #609's positional vectors in this PR's register-as-built manager; the two
  mapped_labels wrapper files go, and so does their dupl rule in .golangci.yml.
- direct_rpc_connection.go: #608's h.do, called with common.JSONContentType.
- direct_ws_subscription_manager.go: the stack's unsubscribeParams, comparing the API interface
  with spectypes.APIInterfaceTendermintRPC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants