NHTSA answers 403, not 429, so the crash-test walk goes slowly on purpose - #8
Merged
Merged
Conversation
…p waiting on a map server Two things the live deployment showed that the local one could not. vPIC decodes a VIN to "Outback". The EPA lists "Outback AWD" and "Outback AWD Turbo" for the same year, so an exact match found nothing and a 2019 Outback came back with no mpg, no engine and no fuel type. When the exact name misses, ask the EPA what it calls that make's models that year and take the closest: ours-plus-a-qualifier first, then ours-minus-one, then the loose match, then nothing rather than the wrong car. The answer says which name it matched on. And a cold Overpass mirror turned a three-second profile into a forty-nine second one. Inside a profile the map now gets nine seconds and two mirrors, because the other five sections are already answered and nobody should wait on repair-shop names to be told their airbag is recalled. Overpass is told the same budget as its own server-side timeout, so it stops working on an answer no one is waiting for. /mechanics on its own is still patient, and either way the tile is cached for a week: one caller waits, nobody after them does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012BzJdwiBGU9hmMGE5fBxjA
…pose Ratings cost three nested calls per vehicle — the make's models, that model's variants, then each variant's full record — which makes this by far the hungriest walk here. Raising the per-run budget to fill the catalogue faster pushed it over whatever line NHTSA draws, and the answer came back 403. A 403 is not 429, so the client's retry-after backoff never saw it and the source recorded a failure instead of waiting. So this one now goes deliberately slowly: sixty lookups a run with a short pause between calls, both configurable. Nothing about a crash test from a past model year needs to arrive quickly. And a throttle on the very first call of a run — listing the year's makes — now ends the run quietly with the cursor untouched, so the next one resumes rather than recording the whole source broken. The registry test caught that the new options were not declared as config fields, which is exactly what it is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012BzJdwiBGU9hmMGE5fBxjA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by watching the live deployment after raising the per-run lookup budget to fill the catalogue faster.
What happened
Crash-test ratings cost three nested calls per vehicle — the make's models, that model's variants, then each variant's full record — which makes this by far the hungriest walk in the collection. At the raised budget it went over whatever line NHTSA draws, and the answer came back:
A 403 is not a 429, so the shared client's
retry-afterbackoff never saw it, and the source recorded a hard failure — which pushesnext_run_atout by a full cadence.Confirmed transient rather than an IP block: the same three endpoints all return 200 from inside the Railway container, and every user-agent returns 200 from a dev box. It was rate.
The fix
This walk now goes deliberately slowly: 60 lookups a run with a 150ms pause between calls, both configurable per source. Nothing about a crash test from a past model year needs to arrive quickly, and the other two NHTSA walks are comfortable at the raised budget because they make one call per vehicle, not three.
A throttle on the first call of a run — listing the year's makes — now ends the run quietly with the cursor untouched, so the next run resumes instead of recording the whole source broken.
Also
The existing registry test caught that
perRunandpaceMswere passed to the default source without being declared as config fields — exactly what that test is for. Both are now declared, so they show up on the add-source page like every other option.🤖 Generated with Claude Code
https://claude.ai/code/session_012BzJdwiBGU9hmMGE5fBxjA