fix(cost): tier Anthropic cache reads by the whole prompt size - #5730
Open
eeshsaxena wants to merge 1 commit into
Open
fix(cost): tier Anthropic cache reads by the whole prompt size#5730eeshsaxena wants to merge 1 commit into
eeshsaxena wants to merge 1 commit into
Conversation
getThresholdValueFunction returns 0 for cachedInputCost on the anthropic branch, so getPricingTier always resolves to tier 0 and a cache read is priced at the base input rate no matter how large the prompt is. inputCost and outputCost on the same request already use the prompt total. For claude-sonnet-4 a 250K-token prompt with a 240K cache read is billed $0.072 instead of $0.144 - half rate - because the >200K tier doubles the input rate the cache multiplier applies to. vertex, google-ai-studio and xai all include cachedInputCost in their threshold cases, and the Gemini over-threshold test asserts the higher tier for its cache read, so anthropic looks like an oversight rather than intent. Updates the existing over-threshold assertion, which had encoded the old behaviour, and adds a mostly-cached long prompt as a regression test.
|
@eeshsaxena is attempting to deploy a commit to the Helicone Team on Vercel. A member of the Team first needs to authorize it. |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anthropic cache reads are always priced from the base tier, so a long prompt is billed at half rate.
Problem
getThresholdValueFunctionreturns0forcachedInputCoston theanthropicbranch (it falls through todefault), sogetPricingTieralways resolves to tier 0 — even thoughinputCostandoutputCoston the same request use the prompt total.For
claude-sonnet-4-20250514, tier 200000 doubles the input rate (0.000003->0.000006) and inherits thecachedInput: 0.1multiplier. A 250K prompt with a 240K cache read:$0.072$0.144inputCoston that same request is already tiered correctly, so only the cache read is wrong.Why this looks like an oversight
vertex,google-ai-studioandxaiall listcachedInputCostin their threshold cases; onlyanthropicomits it. The existing Gemini over-threshold test asserts its cache read at the higher tier (25000 * 0.000004 * 0.1, commented "Total prompt = input + cachedInput = 325000 (over threshold)"), while the Anthropic one asserted the base rate for the same situation. Anthropic's published pricing also tiers cache reads above 200K.Change
Add
cachedInputCostto the anthropic case so it uses the same prompt total asinputCost/outputCost.I also updated the existing over-threshold assertion, since it had encoded the old behaviour, and added a mostly-cached long prompt as a regression test. I verified both fail against the current code and pass with the change.
Notes
cacheWrite5m/1h) still price frombasePricingunconditionally for every provider, so they ignore tiers too. That looked like a separate question so I left it alone and noted it in the test.anthropicblock.__tests__/costpasses (188). The 3registrySnapshotsfailures reproduce on a clean checkout in my environment and are unrelated.