Amazon 979-prefix ISBN-13 lookup via Creators API search_items

[openlibrary-bot]

Summary

Books with a 979-prefix ISBN-13 can never be looked up on Amazon. They are silently rejected by the affiliate server before any Amazon call is made, so we get no price, no affiliate link, and no Amazon import metadata for them — even though Amazon stocks these books.

The fix is to resolve them via the Creators API search_items operation, which AmazonCreatorsAPI does not currently expose.

Why 979 ISBNs are structurally excluded

Amazon's get_items is an exact ID lookup: it accepts an ISBN-10 or a real ASIN, nothing else. A 979-prefix ISBN-13 has no ISBN-10 equivalent (the 979 range was allocated precisely because the 978 space ran out, and there is no back-conversion). Amazon assigns such books an arbitrary B* ASIN with no algorithmic relationship to the ISBN, so no amount of conversion will find it. Only a keyword search can.

This is the same root fact already documented in amazon_affiliate_url() (from #6572, per @hornc):

https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/vendors.py#L311-L336

That fix handled the outbound link case by falling back to an Amazon search URL. The metadata lookup case was never addressed.

Where it dies today

Submit.GET in scripts/affiliate_server.py derives its Amazon key and finds nothing usable:

b_asin, isbn_10, isbn_13 = normalize_identifier(identifier)
key = isbn_10 or b_asin

For 9791234567896 → normalize_identifier returns (None, None, "9791234567896"), so key is None. Two exits follow, neither of which touches Amazon:

• high_priority + stage_import → straight to Google Books (stage_from_google_books)
• otherwise → {"error": "rejected_isbn"}

openlibrary/core/vendors.py::_get_amazon_metadata gates on the 978 prefix explicitly for the same reason:

if len(id_) == 13 and id_.startswith("978"):
    isbn = isbn_13_to_isbn_10(id_)

So a 979 ISBN is never converted, never queued, and never fetched. The Amazon branch is dead code for this entire ISBN range.

Why this is newly fixable

AmazonCreatorsApi exposes search_items(keywords=..., search_index=..., item_count=...), returning a SearchResult whose .items are the same Item type get_items returns — so the existing AmazonCreatorsAPI.serialize() works on search results unchanged.

Our wrapper AmazonCreatorsAPI has no search method at all. Legacy AmazonAPI.search() exists but is explicitly CLI-only ("Adding method to test amz searches from the CLI, unused otherwise") and is being removed along with PA-API in #13315.

Two details of the existing code mean the cache/import layers need no changes:

1. AmazonCreatorsAPI.serialize() already sources isbn_13 from external_ids.eans rather than deriving it from the ASIN, so a B*-ASIN product still serializes with its true 979 ISBN-13.
2. make_cache_key() prefers isbn_13 first, so the product caches under amazon_product_9791234567896 — exactly the key Submit.GET already reads back.

Proposed change

1. Add AmazonCreatorsAPI.search_items() — mirroring get_products()'s throttle discipline and its swallow-and-log-on-error contract.
2. Resolve unmatched ISBN-13s through it only when there is no ISBN-10 and no B* ASIN — i.e. exactly the path that returns rejected_isbn today. Strictly additive: nothing that currently works can regress.
3. Verify before accepting. A keyword search is not an exact-match lookup, so the returned item's external_ids.eans must be confirmed to contain the requested ISBN-13 before the product is cached or staged. Without this we would import the wrong book's metadata under the right ISBN. This is the most important safety property of the change.

Constraints (#13277, #13296)

Amazon affiliate lookups caused a ~12h site-wide outage, so latency on this path is the primary risk and is treated as a hard design constraint:

• The resolution must happen in the background amazon_lookup thread (process_amazon_batch), never inline in Submit.GET. No new blocking Amazon call may be added to a web-worker request path.
• get_items batches up to 10 identifiers per call; search_items is one ISBN per call. A burst of 979 ISBNs therefore costs one API call each and could starve the batched path. The existing API_MAX_ITEMS_PER_CALL / API_MAX_WAIT_SECONDS budget must be respected, and the 979 partition bounded rather than unbounded.

Not yet verified

Live Amazon behavior is unconfirmed — I have no Creators API credentials in this environment. Specifically unverified:

• that search_items(keywords="<979 isbn>", search_index="Books") reliably returns the correct edition
• whether Amazon indexes the raw ISBN-13 string at all for these items
• the real hit rate

If searching by ISBN-13 turns out not to be indexed reliably, the correct outcome is to close this rather than ship a low-hit-rate extra API call on an outage-sensitive path. A prod spot-check should gate implementation.

Success criteria

• AmazonCreatorsAPI.search_items() exists, is throttled, and fails soft like get_products()
• A 979 ISBN-13 resolves to the correct Amazon product, verified via eans before use
• A search result whose eans does not contain the requested ISBN is rejected, not cached
• No new Amazon call on any synchronous request path
• No behavior change for 978 ISBNs or B* ASINs

[openlibrary-bot]

Validation plan

Approach

Resolve 979-prefix ISBN-13s to their arbitrary Amazon B* ASIN via Creators API search_items, entirely inside the existing background amazon_lookup thread, and accept a result only when the returned item's external_ids.eans contains the requested ISBN.

Strictly additive: this path currently returns rejected_isbn or falls through to Google Books, so no behavior that works today can regress.

Files to modify

File	Change
openlibrary/core/vendors.py	Add AmazonCreatorsAPI.search_items() + an ISBN→product resolver that verifies via eans
scripts/affiliate_server.py	Submit.GET: allow a 979 ISBN-13 as a queue key. process_amazon_batch: partition direct-lookup vs. search-resolved identifiers
openlibrary/tests/core/test_vendors.py	Unit tests for the new methods
scripts/tests/test_affiliate_server.py	Tests for partitioning and the no-import mapping

Two non-obvious details this design has to get right

1. Verify the search result, or import the wrong book. Keyword search is not exact-match lookup. Amazon may return a different edition, a boxed set, or an unrelated item. The returned item's external_ids.eans must contain the requested ISBN-13 or the product is discarded — never cached, never staged. This is the single most important safety property of the change.

2. no_import_identifiers breaks silently without a back-mapping. process_amazon_batch currently filters staging with:

no_import_identifiers = {i.identifier for i in asins if not i.stage_import}
books = [... for product in products if product.get("source_records")[0].split(":")[1] not in no_import_identifiers]

For a search-resolved product, source_records[0] is amazon:{B-ASIN} while the queued identifier was the 979 ISBN-13. The two never compare equal, so a stage_import=false request for a 979 ISBN would be staged for import anyway — a real regression introduced by this change if unhandled. Search-resolved products must be mapped back to the identifier that was queued for them before that filter runs.

Tests (input → expected output)

Test	Input	Expected
search_items happy path	keywords=9791234567896	returns serialized products; one search_items call
search_items failure is soft	library raises ItemsNotFoundError	returns None, logged, no exception escapes
search_items throttled	two consecutive calls	second waits 1/throttling, same contract as get_products
ISBN verified	search returns item whose eans contains the ISBN	product returned
ISBN mismatch rejected	search returns a different book	returns None; nothing cached, nothing staged
Submit.GET queues 979	9791234567896	identifier enters queue (today: rejected_isbn)
978 unchanged	9780312368615	still converted to ISBN-10, no search call
B* ASIN unchanged	B00ABCDEFG	direct get_items, no search call
Partitioning	mixed batch of ISBN-10s + one 979	one batched get_items + one search_items
stage_import=false honored	979 with stage_import=false	resolved product not staged

Success criteria

• All tests above pass in Docker; no new pre-commit or CI failures
• No new Amazon call on any synchronous request path — resolution happens only in the amazon_lookup background thread
• 978 ISBNs and B* ASINs take byte-identical code paths to today
• A mismatched search result is provably rejected

Out of scope

• Changing get_items batching or the existing PA-API→Creators migration (feat(amazon): require Creators API, remove PA-API fallback from affiliate server #13315)
• The FastAPI event-loop blocking fix (Amazon affiliate fetch blocks the FastAPI event loop (P1 amplifiers from the 13277 outage) #13296)
• Google Books fallback ordering
• Backfilling existing 979 editions

Risks

Risk	Mitigation
Premise may be false — Amazon may not index 979 ISBN-13 strings at all	Gating on a prod spot-check before implementation (see below)
Wrong-book import	eans verification; reject on mismatch
Throughput: search_items is 1 ISBN/call vs. 10/call for get_items; a 979 burst could starve the batch queue	Bound the search partition per batch; respect existing API_MAX_ITEMS_PER_CALL / API_MAX_WAIT_SECONDS budget
Latency on an outage-sensitive path (#13277)	All work stays in the background thread; Submit.GET keeps its existing cache-poll behavior unchanged

---

⚠️ Requesting a staff decision before implementing

The core premise of this issue is unverified, and I cannot verify it myself. I have no Creators API credentials and no access to ol-home0 from this environment (SSH times out), so I cannot run the spot-check that PR #13315 used.

The unknown is empirical, not architectural: does search_items(keywords="<979 isbn>", search_index="Books") actually return the correct edition, and at what hit rate?

• If yes → the plan above is ready to implement as written.
• If no → the correct outcome is to close this issue, not to ship it. A low-hit-rate, un-batched extra API call on the exact code path that caused the ~12h outage in Post-mortem: ~12h site-wide outage — Amazon affiliate link generation saturated all FastAPI workers #13277 is a bad trade, and the fail-safe eans check would just make it a reliably wasted call.

Because the design fails safe either way, this is purely a question of whether the feature earns its cost — which needs a real hit rate, not a guess.

What would unblock this: anyone with ol-home0 access running the module's documented procedure against a handful of known 979 ISBNs:

import web
from scripts.affiliate_server import load_config
load_config('/olsystem/etc/openlibrary.yml')
web.amazon_api.api.search_items(keywords="9791234567896", search_index="Books")

I will write the test stubs while this is gated, and implement immediately on a green light.

[openlibrary-bot]

Concrete test case for the spot-check, and partial evidence the premise holds

Our own test suite already contains a real 979 ISBN with its real, verified Amazon ASIN — from the #6572 work:

• ISBN-13 9798776159572 (Pickleball Soap Opera) → ASIN B09MJ3TKX3

(openlibrary/tests/core/test_vendors.py, test_amazon_affiliate_url_979_isbn_falls_back_to_search_url and test_amazon_affiliate_url_explicit_asin_overrides_isbn_conversion.)

Two things follow:

1. It confirms the mapping this issue depends on genuinely exists — a 979 ISBN-13 does correspond to a real B* ASIN that Amazon serves. get_items simply has no way to reach it.
2. The Links to Amazon not working for some books when they should #6572 fix deliberately falls back to the Amazon storefront search URL (/s?k=<isbn>&i=stripbooks) for exactly these ISBNs. That fallback is only useful if Amazon's search index resolves a raw 979 ISBN-13 string to the right product — which is the same index search_items queries. So there is reasonable prior evidence the premise holds.

This is suggestive, not confirmation: it does not establish that the API surface behaves like the storefront, and it says nothing about hit rate across the 979 range. A live check is still needed. But it gives the spot-check a known-good input with a known-correct answer:

web.amazon_api.api.search_items(keywords="9798776159572", search_index="Books")
# expected: an item with asin == "B09MJ3TKX3",
#           whose item_info.external_ids.eans contains "9798776159572"

If that returns B09MJ3TKX3, the premise is confirmed end-to-end and the plan above is ready to implement as written.