http_client_proxies is silently dropped by every provider except the three Azure paths (docs promise All) #7492

Activity

1. github-actions

   added

   sdk-pythonPython SDK specificPython SDK specific

   on Sep 29, 2026
2. chenhz01 commented on Sep 30, 2026

   

   chenhz01

   on Sep 30, 2026

   More actions

   Hi @passionworkeer - this extends #6651 (OpenAI paths, closed as not_planned) to the full provider matrix, and the symptom is sneaky: build_http_client() builds the proxied httpx.Client for every provider, but most constructors don't forward config.http_client, so 17 providers go direct with no warning - prompts and keys included.

   Here is a probe that proves whether a provider's client is actually proxied:

   # Verify a provider actually routes through your proxy instead of going direct.
   # Build the client the way mem0 does, then assert the transport is proxied.
   # Plug in your own httpx.Client (e.g. from build_http_client(config)).
   import httpx
   
   def proxy_applied(client) -> bool:
       transport = getattr(client, "_transport", None)
       pool = getattr(transport, "_pool", transport)
       return type(pool).__name__ == "HTTPProxy"
   
   if __name__ == "__main__":
       direct = httpx.Client()
       proxied = httpx.Client(proxy="http://127.0.0.1:8080")
       print("direct: ", proxy_applied(direct))    # False -> traffic goes direct
       print("proxied:", proxy_applied(proxied))   # True  -> proxy is wired through
       # False here means the provider's constructor dropped config.http_client.

   proxy_applied() inspects the transport's pool type; False means the proxy was silently dropped. If useful I can script this across all 20 providers to produce the exact pass/fail table.
3. bluetomlee commented on Oct 1, 2026

   

   bluetomlee

   on Oct 1, 2026

   More actions

   Confirmed the impact on current main (94c3fe9): build_http_client() builds the proxied httpx.Client into config.http_client for every provider, but only the three Azure constructors forward it to their SDK client — on the other 17 the proxy is silently dropped while the config still looks wired.

   One question before anyone invests in a PR, given #6651 was closed as not_planned: is wiring the remaining providers something you want to take on now?

   If yes, I'd suggest keeping the fix in one place rather than touching 17 constructors — a shared wiring helper each provider constructor calls (passing config.http_client to the SDK client where the SDK accepts one), and an explicit warn-on-drop for the few SDKs that can't take an httpx client, so the failure mode stops being silent. Regression test would be parametrized over the provider matrix and assert on the transport pool type (httpcore.HTTPProxy), same inspection method the report used.

   Happy to take this on if it's agreed.
4. Varshith-Kali commented on Oct 8, 2026

   

   Varshith-Kali

   on Oct 8, 2026

   More actions

   Confirming the report on current main: build_http_client() in mem0/configs/llms/base.py builds the proxied httpx.Client into config.http_client for every provider, but only the three Azure constructors (azure_openai.py, and the two other Azure paths) forward it to their SDK clients. The other ~17 provider constructors (OpenAI, LiteLLM, Gemini, etc.) construct their SDK clients without http_client=..., so the proxy config — and anything relying on it for egress control — is silently dropped.

   Before writing code, I wanted to check the preferred approach, since #6651 (the OpenAI-only subset) was closed as not planned:

   1. Wire-through: each provider constructor passes config.http_client to its SDK client where the SDK accepts an httpx client (most OpenAI-compatible SDKs do), via a small shared helper to avoid 17 copy-pasted edits.
   2. Loud fallback: for the few SDKs that can't take an httpx client, log an explicit warning at construction time so the dropped proxy stops being silent.

   Is the full-matrix version of this in scope, and would approach (1)+(2) be welcome? Happy to implement it that way if so.
5. chenhz01 commented on Oct 8, 2026

   

   chenhz01

   on Oct 8, 2026

   More actions

   Thanks @Varshith-Kali for the independent confirmation — and with @bluetomlee asking essentially the same "should we fix this?" back on Oct 1, this thread now has two confirmations and two nearly identical questions waiting on the same answer. The bug's whole trick is failing silently; the fix decision shouldn't have to die silently too. 🙂

   On approach: (1) and (2) are complementary, not either/or. (1) is the actual fix — a shared wiring helper means one fix point instead of ~17 copy-pasted edits, which is almost exactly what @bluetomlee proposed on Oct 1. Two independent reads landing on the same shape is usually a good sign. (2) is the safety net for the few SDKs that can't accept an httpx client — given the failure mode here is silence, the loud warning is what prevents the next variant of this bug.

   One honest caveat on "in scope": that call belongs to the mem0 maintainers, not me — I'm a reporter in this thread (my Sep 30 comment is a transport-level probe confirming the drop). But given #6651 was closed as not_planned, a PR is probably the cheapest way to force a maintainer decision: if it merges, the full provider matrix is fixed; if it doesn't, it still serves as the reference implementation for whoever picks this up next.

   If you do open the PR, two concrete offers:

   1. Build the acceptance test on the transport probe from my Sep 30 comment — assert the constructed client's transport._pool is httpcore.HTTPProxy, not just that config.http_client looks wired. Config-level assertions would have missed this exact bug, and that's the entire lesson of this issue.
   2. I'll run the probe against your branch before you open the PR (dead-proxy port and all), and review it once it's up. Consider this a green light for the approach from the thread's side — the maintainer sign-off is the only gate left, and a working PR is the strongest possible argument for it.
6. Varshith-Kali commented on Oct 9, 2026

   

   Varshith-Kali

   on Oct 9, 2026

   More actions

   @chenhz01 the fix is built and the branch is up if you'd like to run your transport probe against it before anything is opened: fix/http-client-proxies-all-providers on my fork.

   What it does: a shared wiring helper routes config.http_client into every LLM and embedding provider's SDK client where the SDK accepts an httpx client (15 providers), plus an explicit warning at construction time for the SDKs that can't take one — so the dropped proxy stops being silent. 35 regression tests, including transport-level assertions (transport._pool is httpcore.HTTPProxy) per your suggestion rather than config-level ones.

   One process note for the maintainers: I'm holding the PR itself until this issue carries the accepted label (per the repo's PR gate) — happy to open it as soon as that's in place.
7. chenhz01 commented on Oct 15, 2026

   

   chenhz01

   on Oct 15, 2026

   More actions

   @Varshith-Kali ran your branch through the transport probe as offered — baseline vs fix/http-client-proxies-all-providers (81ca368), same probe, two runs, side by side.

   Environment

   Windows, Python 3.13.14, httpx 0.28.1 / httpcore 1.0.9. Baseline = PyPI mem0ai 2.2.1 (which is upstream release commit 94c3fe9f, the exact commit the original report was verified on). Your branch verified via the compare API: 1 commit ahead of main, 0 behind, and none of the 22 source files your commit touches changed between 94c3fe9f and main — so the branch was exercised by overlaying your files onto a stock 2.2.1 install (import path confirmed per run). Ambient HTTP_PROXY/HTTPS_PROXY were stripped from the probe environment — see the second method note below for why that matters.

   Result matrix

   Every one of the 17 paths the original report marked as dropped now honors the proxy on your branch — in both string and dict config forms (34/34 cases inspectable, 32 proxied + 2 environment-blocked):

   • 12 LLM paths (openai, openai_structured, anthropic, deepseek, groq, together, xai, minimax, vllm, lmstudio, ollama, gemini) × both forms: dropped → proxied
   • 5 embedding paths minus ollama (openai, together, lmstudio, gemini) × both forms: dropped → proxied
   • embeddings/ollama × both forms: constructor performs a live call to the local Ollama host, which does not exist on this machine, so construction fails before the transport can be inspected — identical behavior on baseline and branch, so it is environment-blocked rather than a finding either way.
   • The 3 Azure paths: proxied on baseline and on your branch — no regression.
   • sarvam, litellm: still no httpx client surface (as in the original report's N/A rows).
   • aws_bedrock (both LLM and embeddings): still classified N/A by transport inspection (botocore has no httpx client) — but your build_botocore_config wiring produced a nice piece of behavioral evidence: constructing AWSBedrockLLM on your branch logs Failed to connect to proxy URL: "http://127.0.0.1:9" (the probe's dead-proxy URL), while baseline reaches the real AWS endpoint and fails on the dummy credentials instead. So on your branch the original N/A boto3 paths are behaviorally proxied too — worth a line in the PR description.
   • embeddings/huggingface: not run — sentence_transformers is not installed in this environment; recording as an honest gap rather than a pass.

   Your own tests/test_http_client_proxies.py from the branch: 39/39 passed on this environment. (It does not collect on stock 2.2.1 since it imports your new mem0.utils.http helper, so the baseline side of the comparison comes from the probe instead.)

   Two probe-method notes you may want in the PR tests

   1. httpx 0.28 changed where the proxy lives. The original report's detection — transport._pool being httpcore.HTTPProxy — is blind on httpx 0.28.1: httpx.Client(proxy=...) no longer builds an HTTPProxy pool on the main transport; the proxied transports now sit in client._mounts (URLPattern → HTTPTransport whose _pool is httpcore.HTTPProxy). A probe that only checks the legacy location reports every path as "dropped" even when the proxy is honored. The version-adaptive check that produced the matrix above uses three evidence channels: legacy transport._pool, any _mounts transport pool, and object-identity with config.http_client (the strongest signal where the SDK accepts an injected client — that is how the openai/deepseek/groq/together/xai/minimax/vllm/lmstudio/gemini paths verified). Your 39 tests evidently already handle this correctly on 0.28 — worth pinning explicitly in case CI pins a different httpx.

   2. Strip ambient proxy env vars or every path looks proxied. The first probe run in this environment carried HTTP_PROXY/HTTPS_PROXY in the process env, and with trust_env=True (the SDK default) every SDK client — including all 17 baseline paths — showed proxied mounts. That is the env-layer twin of the config-layer false pass the original report warned about: ambient env proxies make the whole matrix pass regardless of mem0's wiring. The clean run strips HTTP_PROXY/HTTPS_PROXY/ALL_PROXY before constructing anything; the dead proxy is passed explicitly via http_client_proxies only. After stripping, baseline cleanly reproduces 32/32 dropped and the branch cleanly reproduces 32/32 proxied.

   One discrepancy recorded for completeness: the xai construction crash from the original report (AttributeError: 'Client' object has no attribute 'url', still open as #6230) did not reproduce on the PyPI 2.2.1 wheel used here — XAILLM constructs cleanly in both forms (and is proxied on your branch). Wheel-vs-main drift or an environment difference; noting it rather than quietly dropping it.

   No ask attached to this — the accepted label decision is the maintainers' and you are already holding the PR per the repo gate. The probe script (single file, stdlib + the provider SDKs, per-path evidence records included) is available if it is useful for the PR's test suite.