Skip to content

fix(initializer): use static service source for IP-based openai-compat base URLs - #1171

Open
xiaodingjiang wants to merge 1 commit into
agentscope-ai:mainfrom
xiaodingjiang:fix/openai-compat-ip-service-source
Open

fix(initializer): use static service source for IP-based openai-compat base URLs#1171
xiaodingjiang wants to merge 1 commit into
agentscope-ai:mainfrom
xiaodingjiang:fix/openai-compat-ip-service-source

Conversation

@xiaodingjiang

Copy link
Copy Markdown

PR 描述(英文版,用于 GitHub PR)

Title: fix(initializer): use static service source for IP-based openai-compat base URLs


Summary

Fixes #1057. When AGENTTEAMS_OPENAI_BASE_URL uses a bare IP address (e.g. http://192.168.1.10:3000/v1), the initializer unconditionally registers a DNS-type service source. Two independent failures result:

  1. Higress console rejects bare IPs at validation timeValidateUtil.checkDomain() uses DOMAIN_PATTERN = ^(?:(?!-)[a-z0-9]...\.)+[a-z]{2,63}$ which requires a letter TLD; checkDomain("192.168.1.10") fails → ValidationException: serviceSource body is not valid → the service source is never created.
  2. The provider's rawConfigs.openaiCustomServiceName references openai-compat.dns, a cluster that does not exist.

Result: every LLM call fails with HTTP 503 cluster_not_found (verified end-to-end on Embedded v1.2.2).

Changes

agentteams-controller/internal/initializer/initializer.go:

  • Detect IP hosts with net.ParseIP(host) != nil in both the openai-compat branch and the default provider branch.
  • For IP hosts, register a static service source (EnsureStaticServiceSource, which supports ip:port domains and passes checkIpAddress validation) and set rawConfigs.openaiCustomServiceName to <provider>.static.
  • For domain hosts, behavior is unchanged (DNS service source + <provider>.dns), so existing domain-based setups are fully backward compatible.

Verification

  • go build ./internal/initializer/ passes (Go 1.22)
  • End-to-end on AgentTeams Embedded v1.2.2: after applying the equivalent workaround manually (static service source + openaiCustomServiceName=openai-compat.static via console API), all LLM calls return 200. This confirms both the root cause and the fix direction.
  • Domain-based flow untouched (regression surface: none — DNS path identical to before).

Note on prior PRs

…t base URLs

When AGENTTEAMS_OPENAI_BASE_URL uses a bare IP address (e.g.
http://192.168.1.10:3000/v1), the initializer unconditionally registers a
DNS-type service source. Higress console rejects bare IPs in
ValidateUtil.checkDomain (DOMAIN_PATTERN requires a letter TLD), so the
service source is never created and the provider references a non-existent
openai-compat.dns cluster, causing every LLM call to fail with HTTP 503
cluster_not_found.

Detect IP hosts with net.ParseIP and register a static-type service source
instead (which supports ip:port domains), and reference it as
<provider>.static in rawConfigs.openaiCustomServiceName.

Fixes agentscope-ai#1057 (verified end-to-end on Embedded v1.2.2: provider restored by
manually creating the static service source and pointing
openaiCustomServiceName to openai-compat.static).

Also applies the same fix to the default provider branch (unrecognized
provider names with a custom base URL).
@xiaodingjiang

xiaodingjiang commented Aug 15, 2026

Copy link
Copy Markdown
Author

503 复发机制深挖(2026-08-14,PR #1171 补充证据)

1. 复发事实

  • 2026-08-12:修复(service source static + provider 引用 .static)→ 闭环跑通
  • 2026-08-14:26 小时后 Worker 休眠唤醒,LLM 全部 503
  • 现场定位:provider 的 openaiCustomServiceName 又变回 openai-compat.dns(.static 被覆盖)

2. 决定性实验(复现)

步骤 provider 状态
重启前(已修复状态) openai-compat.static
docker restart agentteams-controller + 45s openai-compat.dns ← 默认值回填

结论:controller 重启即复发——启动路径无条件覆盖 provider 配置。

3. 根因(二进制证据)

/usr/local/bin/agentteams-controller(Go 二进制)符号表与字符串池:

  • 符号:EnsureAIProviderEnsureModelProviderAuth(controller/internal/controller)
  • 字符串池(三处 openai-compat 上下文):
    1. higress login: %w + /v1/ai/providers/ + openai-compat.dns——启动时登录 console 并 PUT provider,serviceName 硬编码 .dns
    2. create %s: %w + openai-compat——create provider 调用
    3. failed to register openai-compat service source (non-fatal)——service source 注册失败仅告警不中断

4. 完整故障链

controller 启动/重启
  → EnsureAIProvider:higress login → PUT /v1/ai/providers/(硬编码 openai-compat.dns)
  → console 中修复过的 .static 被覆盖回 .dns
  → service source 注册失败(non-fatal,DNS 类型被校验拒绝——issue 本体的根因)
  → .dns cluster 从未存在
  → Higress 路由 /v1/ → cluster_not_found
  → Worker LLM 调用 4 次重试全 503 → MODEL_EXECUTION_FAILED

5. 上游修复建议(PR #1171 补充)

  1. EnsureAIProvider 幂等化:provider 已存在且可用时不覆盖(读-改-写而非无条件 PUT);
  2. 后缀按 service source 实际类型拼接:static → .static、DNS → .dns,或允许 IP 走 static 分支;
  3. 启动覆盖改为显式开关:仅在 --force-provider 或 provider 缺失时写回默认值;
  4. service source 注册失败改为阻塞或告警:当前 non-fatal 导致 .dns 悬挂引用静默存在。

6. 现场处置(缓解)

PUT provider openaiCustomServiceName=openai-compat.static(console API)→ 立即恢复;
但在 controller 下次重启前,任何修复都会被再次覆盖——缓解不是根治


503 Deep exploration of the recurrence mechanism (2026-08-14, PR #1171 additional evidence)

1. Recurrence facts

  • 2026-08-12: Repair (service source static + provider reference .static) → closed-loop runthrough
  • 2026-08-14: Worker wakes up from sleep after 26 hours, LLM all 503
  • On-site positioning: provider's openaiCustomServiceName changes back to openai-compat.dns (.static is overwritten)

2. Decisive experiment (reproduction)

steps provider status
Before restart (fixed state) openai-compat.static
docker restart agentteams-controller + 45s openai-compat.dns ← Default value backfill

Conclusion: Relapse occurs when the controller is restarted - the startup path unconditionally overrides the provider configuration.

3. Root cause (binary evidence)

/usr/local/bin/agentteams-controller (Go binary) symbol table and string pool:

  • Symbols: EnsureAIProvider, EnsureModelProviderAuth (controller/internal/controller)
  • String pool (three openai-compat contexts):
    1. higress login: %w + /v1/ai/providers/ + openai-compat.dns - Log in to the console at startup and PUT provider, serviceName hard-coded .dns
    2. create %s: %w + openai-compat——create provider call
    3. failed to register openai-compat service source (non-fatal)——service source fails to register, only an alarm is issued without interruption.

4. Complete fault chain

controller start/restart
  → EnsureAIProvider:higress login → PUT /v1/ai/providers/ (hardcoded openai-compat.dns)
  → Fixed .static in console is overwritten back to .dns
  → Service source registration failed (non-fatal, DNS type was rejected by verification - the root cause of the issue)
  → .dns cluster never existed
  → Higress route /v1/ → cluster_not_found
  → Worker LLM call 4 retries all 503 → MODEL_EXECUTION_FAILED

5. Upstream fix suggestions (Added by PR #1171)

  1. EnsureAIProvider idempotence: Provider does not overwrite when it already exists and is available (read-modify-write instead of unconditional PUT);
  2. The suffix is spliced according to the actual type of service source: static → .static, DNS → .dns, or the IP is allowed to go to the static branch;
  3. Change startup override to explicit switch: only write back the default value when --force-provider or provider is missing;
  4. Service source registration failure is changed to blocking or alarm: The current non-fatal causes .dns dangling references to exist silently.

6. On-site disposal (mitigation)

PUT provider openaiCustomServiceName=openai-compat.static (console API) → restore immediately;
But any fixes will be overwritten again before the next restart of the controller - mitigation is not a cure.

@oss-maintainer oss-maintainer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Fixes #1057: when AGENTTEAMS_OPENAI_BASE_URL is a bare IP, the initializer now registers a static service source (via net.ParseIP detection in both the openai-compat and default-provider branches) and points openaiCustomServiceName at <name>.static; domain-based hosts keep the DNS path unchanged. The Go-side fix is correct, minimal, and backward compatible — we verified EnsureStaticServiceSource exists on current main with a compatible signature, and end-to-end verification against Embedded v1.2.2 is documented in the PR description. Overall verdict: the change is good to merge, but there is one companion overwrite path this PR does not cover (see notes below), which matches the 503 recurrence the author documented in the follow-up comment.

Findings

  • [Critical] manager/scripts/init/setup-higress.sh:221 — hardcoded openai-compat.dns + unconditional PUT on every Manager boot overwrites the .static provider config (the documented recurrence). Companion fix needed.
  • [Warning] install/agentteams-install.ps1:1406 — also hardcodes $serviceName.dns; fresh IP-based installs hit the same 503.
  • [Warning] IPv6: ensureStaticServiceSource formats ip:port without brackets; net.ParseIP accepts IPv6, so the new branch can reach this path.
  • [Info] proto unused in static branch (static sources hardcode http); no unit tests for the new branching.

Suggestions

  1. Follow-up (or extend this PR): in setup-higress.sh, detect an IP host and register a static service source + openai-compat.static (mirror the Go logic); same for the default provider section and agentteams-install.ps1.
  2. Alternatively, change the shell script's LLM provider section from unconditional PUT to read-modify-write so an existing service-name suffix set by the controller is preserved.
  3. Mitigating factor for the Go path: EnsureAIProvider uses POST and treats 409 (already exists) as success without updating, so controller restarts do not overwrite the provider — the overwrite vector is specifically the shell script's PUT.

Automated review by github-manager-bot

Additional notes (not anchored to a changed line)

  • [CRITICAL] manager/scripts/init/setup-higress.sh:221 — Recurrence gap: this script hardcodes openaiCustomServiceName: openai-compat.dns and the idempotent LLM Provider section unconditionally PUTs the provider on every Manager container boot. After the controller initializer correctly sets .static, the next Manager restart overwrites it back to .dns, reproducing the exact 503 recurrence documented in this PR's comments. This PR does not touch setup-higress.sh; a companion fix is needed (detect IP host here too and use .static, or switch to read-modify-write so the existing service-name suffix is preserved). (line outside diff)
  • [WARNING] install/agentteams-install.ps1:1406 — The PowerShell install script also hardcodes openaiCustomServiceName = "$serviceName.dns". Fresh Windows/embedded installs using an IP-based OpenAI base URL will hit the same 503 cluster_not_found. Not addressed by this PR. (line outside diff)
  • [WARNING] agentteams-controller/internal/gateway/higress.go:692 — IPv6 formatting concern: ensureStaticServiceSource builds the domain as fmt.Sprintf("%s:%d", address, port). For an IPv6 host this yields ::1:8080 rather than [::1]:8080. net.ParseIP accepts IPv6 literals, so the new static branch can route IPv6 hosts into this path; low probability today but a latent bug for IPv6 deployments. (line outside diff)

logger.Error(err, "failed to register openai-compat service source (non-fatal)")
svcName := "openai-compat"
svcSuffix := "dns"
if net.ParseIP(host) != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

proto is computed but unused in the static branch: EnsureStaticServiceSource hardcodes protocol: "http" internally. An https:// base URL would silently register an http static source. Consistent with existing static-source usage (Tuwunel, Element Web), but worth a short comment or log line documenting the limitation.

}
if err := i.Gateway.EnsureServiceSource(ctx, "openai-compat", host, port, proto); err != nil {
logger.Error(err, "failed to register openai-compat service source (non-fatal)")
svcName := "openai-compat"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No unit tests exist for initializer.go yet. The IP-vs-domain branching is easy to test in isolation — consider a table-driven test covering bare IPv4, bracketed IPv6, plain domain, domain with port, localhost, and empty host.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] openai-compat provider fails when using IP:port base URL

2 participants