[yixin0829]

Why

lp-backend-prod was killed for exceeding its 512 MiB memory limit 6 times in the last 30 days. Each kill dropped the request in flight, and three of them showed users a 503.

The cause is a memory leak in responses.parse(text_format=...) in the locked openai 2.21.0:

• uvicorn runs each request in a fresh contextvars.Context.
• pydantic keeps its generic-class cache in a ContextVar, so that cache is empty on every request.
• So each responses.parse call builds new ParsedResponse[...] classes, and the SDK caches a TypeAdapter for each one in an unbounded lru_cache.

In production, about 1–1.4 MiB stays behind after every uncached generation. Every instance that was killed had served about 320–345 uncached generations, no matter how long it had been running. Upstream tracks this as openai-python#3084. It is fixed in openai 3.16.2, but that is a major-version upgrade (it switches the HTTP client to httpx2), so this PR works around the leak instead.

What changes

• Generation call: it now uses responses.create and sends the same strict JSON schema format. LEARNING_PATH_TEXT_FORMAT is built once at import with the SDK's own type_to_text_format_param, the helper responses.parse uses.
• Parsing: the new parse_graph_output copies what responses.parse did. It validates every output_text part with LearningPathGraphOutput.model_validate_json and returns the first one, or None when there is none. It runs inside _request_completion's existing try, so parse errors map to the same errors as before.
• Tests:
  • Unit tests now mock responses.create.
  • New tests for parse_graph_output cover several parts, one invalid part, and refusal and reasoning items.
  • New regression test test_generation_does_not_grow_sdk_type_cache: it runs 20 generations through the real SDK, each in a fresh contextvars.Context as uvicorn does, and asserts the SDK's TypeAdapter cache does not grow. On main it fails with assert 47 == 7.

Moderation, the prompt, the model and reasoning settings, caching and timing are unchanged. The request OpenAI receives is the same, so the timing history version does not need a bump.

Verification

Check	Result
uv run pytest	93 passed. Also verified on Python 3.12, the Docker and CI version.
ruff format --check, ruff check	clean
Regression test on main / on this branch	fails (47 == 7) / passes 10 out of 10 runs, in any test order
/v1/responses request body and headers, old vs new	byte-identical for gpt-6-luna and a model override
Old vs new outcome for 38 synthetic response shapes plus HTTP 400/401/429/500 and non-JSON	same status and result for all of them. The only difference is the wording of the 500 detail for refusal or empty output (response.output_parsed → parsed output).

Memory under the real server. Setup: single-process uvicorn, Python 3.12, and a local fake OpenAI server (no real API calls). It ran 400 sequential uncached generations, then 5 bursts of 40 concurrent generations.

Requests	main working set	This branch working set
50	241 MiB	110 MiB
400	525 MiB	111 MiB
600 (after bursts)	635 MiB	123 MiB

On this branch the SDK's TypeAdapter cache stays at 13 entries; on main it grows by 2 per request. A separate 3,000-request soak on this branch stayed level at about 111 MiB.

Not tested here: a generation against the real OpenAI API. The request is byte-identical, so the first prod generations after the deploy will confirm it.

Deploy note

This changes server/**, so merging deploys lp-backend-prod through Cloud Build. After the deploy, check memory/utilizations. It should stay flat past about 350 uncached generations, where it used to reach the limit.

Merge order with #36: the two PRs merge without conflicts, but #36's new test test_moderation_and_generation_get_cleaned_topic_with_punctuation still mocks responses.parse and fails on top of this PR. Whichever PR merges second needs to rebase first. On #36, its two new tests should mock responses.create with output=[message], as the tests here do.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
lp-generator-agro	Ready	Preview	Oct 30, 2026 11:42pm UTC