[ENH](wqs): Add queue metrics

[tanujnay112]

Summary

• publish low-cardinality gauges for queue depth, failed entries, aggregate
  failure count, and durable snapshot bytes
• publish counters for PushWork, successful FinishWork, and persistence
  failures
• refresh state gauges on queue mutations and the existing periodic
  persistence cycle
• avoid function and collection IDs as metric attributes

Validation

• cargo test -p worker work_queue::state::tests (9 passed)
• cargo test -p worker work_queue::work_queue_manager::tests (12 passed)
• cargo fmt --all -- --check
• git diff --check

Rollout

Deploy this service change together with
chroma-core/k8s#3206 before
applying the Honeycomb board in
chroma-core/hosted-chroma#8328.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-01T23:57:54.516906Z	95d40a8	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[tanujnay112]

• [ENH](wqs): Add queue metrics #7669 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95d40a86fb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid rescanning the entire queue for every metric update

When the queue contains many entries, every push, finish, defer, or failure-count update now calls metric_values(), which folds over the complete pending_work deque while running on the manager's single component runtime. This adds O(queue depth) synchronous work to each request—even deduplicated pushes that do not change state—and can substantially reduce throughput or delay all other queue messages. Maintain the three aggregates incrementally when state changes, or collect them periodically instead of rescanning per request.

Useful? React with 👍 / 👎.