Track Batch API token usage and include it in cost() at half price - #55
Merged
Conversation
Batch responses report usage in each item's body, but it was discarded — batch_chat spend was invisible to usage()/cost(). Accumulate it in a new ChatClient::batch_usage counter, kept separate from `usage` because OpenAI bills the Batch API at a flat 50% of the standard price, and fold it into cost() with that discount applied. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbNrXGx3ddWwHTsBNur2w9
The table now lives in its own crate with no dependencies, and `tysm`
depends on it and re-exports it. `ChatClient::cost()` is unchanged.
The point is that anything needing to price tokens can vendor the table
alone. Android/Soong builds compile from in-tree sources, so depending on
`tysm` there would mean vendoring reqwest, schemars, zstd and dashmap into
the tree in order to multiply tokens by a rate. An empty `[dependencies]`
is the whole feature; keep it that way.
Two things the table could not express before:
- Service tiers per model, rather than three name-keyed tables. Gemini
sells the same tiers OpenAI does and most background traffic is served
at flex, so omitting them billed the common case at twice its rate.
- A long-context card, for the models that re-price a request once its
prompt crosses a threshold (gpt-5.x at 272k, Gemini Pro at 200k).
That second one forces the API to be per-call. The premium keys off how
large one prompt was, so pricing a tally summed over several calls tests
the threshold against the sum of every prompt and bills them all at the
premium. `ChatClient` therefore accumulates dollars per request instead of
pricing accumulated tokens; for models without a long-context card the two
are identical, so this only changes what was already wrong.
`CallTokens` is a named struct because its three fields are all `u32`, and
swapping cached for output would silently mis-bill rather than fail to
compile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELngZ7EJdVAo8WBxs8ZJGP
…5 coverage Anthropic's api ids use dashes (`claude-opus-4-5-20251101`) while the table carried only the dotted spelling, so longest-prefix matching fell through to `claude-opus-4` and billed Opus 4.5 at $15/$75 instead of $5/$25. Both spellings are listed now, with a test over the ids we actually send — the failure is silent and threefold, which is the worst combination. Also corrects the gpt-5 comment, which asserted every 5.x model charges a long-context premium and then recorded the rates for none of them outside the 5.6 family. What we actually know is narrower; say that instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELngZ7EJdVAo8WBxs8ZJGP
Breaking under 0.x rules, where the minor is the breaking position: `ChatClient` gains public fields (`batch_usage`, `spend`), so a downstream struct literal no longer compiles, and `cost()` changes meaning — it now returns dollars accumulated as each request completed, rather than pricing the accumulated token counts. For every model without a long-context card those agree exactly; where they differ, the old answer was the wrong one. Also newly public: the `model_prices` module, and the `model-prices` crate it now re-exports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELngZ7EJdVAo8WBxs8ZJGP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch API responses report
usagein each item's body, butbatch_chat_*discarded it, so batch spend was invisible tousage()/cost()(theusagefield doc even said so).ChatClient::batch_usage: RwLock<ChatUsage>accumulates the reported usage of every billed batch item, kept separate fromusagebecause OpenAI bills the Batch API at a flat 50% of the standard price.cost()now returns live cost + batch cost × 0.5 (batch priced against the standard table, not a service tier).batch_usage()accessor, and a unit test asserting the half-price combination.One caveat noted in a comment: a batch reattached from a previous run (the existing request-hash reuse path) counts its usage again in the new process — the tokens were genuinely billed, just possibly already counted by the earlier process.
The 5 failing
tests::*lib tests fail identically on main (they hit the network); the new test passes.🤖 Generated with Claude Code
https://claude.ai/code/session_01JbNrXGx3ddWwHTsBNur2w9