feat(inference): log gateway quota headers on 429

[rosetta-livekit-bot]

Summary

Port livekit/agents#6736 so inference LLM 429 responses log the gateway's RPM, TPM, and credits quota snapshot before preserving the existing error path.

• extract stamped quota headers with case-insensitive native Headers lookup
• log the model, request ID, and only the quota dimensions provided by the gateway
• add parity coverage for complete, partial, absent, and lowercased headers plus 429 logging with and without quota telemetry

Source diff coverage

• Adapted: livekit-agents/livekit/agents/inference/_utils.py -> agents/src/inference/utils.ts. Python Mapping lookup is adapted to native Headers.get(), preserving case-insensitive lookup and omission of missing/empty values. Structured field names use the target's camelCase logging convention.
• Adapted: livekit-agents/livekit/agents/inference/llm.py -> agents/src/inference/llm.ts. The source's 429 warning is adapted to the existing OpenAI APIError branch and Pino object-first logger while leaving APIStatusError conversion and retryability unchanged.
• Adapted: tests/test_inference_utils.py -> agents/src/inference/utils.test.ts and agents/src/inference/llm.test.ts. All four quota extraction cases and both 429 logging cases added by the source PR are ported, split according to the target's co-located test organization.

No source files or behaviors were omitted. The target already had the required response-header, request-ID, and structured-logging infrastructure.

Testing

• pnpm test agents (1,591 passed, 5 skipped)
• pnpm build
• pnpm lint
• pnpm format:check
• pnpm --filter @livekit/agents typecheck

Source: livekit/agents#6736

---

Ported from livekit/agents#6736

Original PR description

Summary

When an inference.LLM request is rejected with a 429, customers currently get a bare APIStatusError with no indication of which limit they hit or by how much. The agent gateway now stamps quota telemetry headers on LLM completions responses (livekit/agent-gateway, X-LiveKit-Inference-*), including on rate-limit rejections — this PR surfaces that snapshot in the agent logs.

On a 429, inference.LLM now emits a structured warning before raising:

LLM request rate limited by inference gateway
  model=openai/gpt-4o request_id=req_123
  rpm_limit=100 rpm_used=101 tpm_limit=50000 tpm_used=48000
  credits_limit=1000000 credits_used=999999

• X-LiveKit-Inference-RPM-{Limit,Used} — requests/min for the project × model bucket
• X-LiveKit-Inference-TPM-{Limit,Used} — tokens/min
• X-LiveKit-Inference-Credits-{Limit,Used} — cumulative token-credit balance

Only dimensions the gateway actually stamped are logged (a missing header means "not enforced / no data", never zero), so responses from older gateways just log model + request_id. The existing APIStatusError raise path is unchanged.

Changes

• inference/_utils.py: extract_quota_usage() maps the gateway's quota headers to log-friendly fields (rpm_limit, tpm_used, …)
• inference/llm.py: LLMStream._log_rate_limited() logs the quota snapshot on 429 before re-raising
• tests/test_inference_utils.py: unit coverage for header extraction (all dimensions, partial, empty, case-insensitive httpx.Headers) and the 429 log path (with and without quota headers)

Testing

uv run pytest tests/test_inference_utils.py --unit   # 11 passed

ruff format / ruff check clean; make type-check shows only the pre-existing boto3 stub error in the AWS plugin.

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: d385403

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hedra	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🟡 New shared helper for reading quota data is published without documentation

The newly exported helper that pulls quota numbers out of response headers (extractQuotaUsage at agents/src/inference/utils.ts:34) ships with no TypeDoc comment, which the repository's contribution rules require for every new exported function.
Impact: The generated API documentation will have an undocumented entry, so users cannot tell what the helper returns.

Rule source and location

CONTRIBUTING.md states: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented." The new exported function at agents/src/inference/utils.ts:34-41 has only a non-doc // comment above the constant at agents/src/inference/utils.ts:23-24, not a /** */ TypeDoc block on the function itself.

Suggested change

	export function extractQuotaUsage(headers: Headers): Record<string, string> {
	/**
	* Extract the inference gateway quota usage headers (RPM/TPM/credits limits and
	* usage) from a response's headers. Missing or empty dimensions are omitted.
	*
	* @param headers - Response headers stamped by the inference gateway.
	* @returns A map of camelCase quota field names to their stringified values.
	*/
	export function extractQuotaUsage(headers: Headers): Record<string, string> {

---

Was this helpful? React with 👍 or 👎 to provide feedback.