Fix startup xDS snapshot using legacy cluster names after controller restart - #3236
Open
MinuraKariyawasam wants to merge 1 commit into
Open
Conversation
…restart The initial Envoy xDS snapshot was generated before the transformer registry was wired into the translator. With t.transformers still nil, the translator silently fell back to the legacy translation path, which names clusters "cluster_<scheme>_<host>", while the policy engine's routes (built later via the transformer path) reference "upstream_<name>_<host>_<port>". After any controller restart, every request to a previously deployed API failed with 503 cluster_not_found until the API was manually redeployed, since control-plane reconciliation skips APIs whose deployed_at is unchanged. Fix: build the transformer registry and call translator.SetTransformers before the initial snapshot is generated, so the very first snapshot already uses transformer-path naming. This also avoids the transient mismatch window a regenerate-after-wiring approach would leave for already-running gateway runtimes that reconnect as soon as the xDS server starts. The event-gateway controller has the same initialization order, so the same reorder is applied there. Also log a warning when the translator falls back to the legacy path for a non-WebSubApi kind, so this class of regression is visible in logs instead of failing silently. Resolves wso2#3197
MinuraKariyawasam
requested review from
AnuGayan,
Arshardh,
CrowleyRajapakse,
HeshanSudarshana,
HiranyaKavishani,
Induwara04,
Krishanx92,
PasanT9,
Piumal1999,
RakhithaRR,
Tharsanan1,
Thushani-Jayasekera,
VirajSalaka,
ashera96,
chamilaadhi,
dushaniw,
hisanhunais,
lasanthaS,
malinthaprasan,
pubudu538,
renuka-fernando,
senthuran16,
tgtshanika,
tharikaGitHub,
tharindu1st and
thivindu
as code owners
August 17, 2026 08:14
|
|
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe controller now creates one transformer registry before the initial xDS snapshot, shares it with Envoy translation and policy management, and warns when non- ChangesTransformer startup and translation
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to The PR makes a localized startup-ordering fix so existing APIs retain consistent routing after controller restarts, with added warning logs for unexpected legacy translation. No actionable merge-blocking risk remains after normal checks and review. Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Resolves #3197
Whenever the gateway controller restarts (a helm upgrade, a pod restart, anything), every API that was already deployed starts returning 503 with
cluster_not_found. Auth still works (you get 401 without a token), so it looks like the backend is down, but it isn't. The only way to recover is to redeploy every API from the Publisher, and it breaks again on the next restart.The root cause is an ordering problem in
cmd/controller/main.go. The first Envoy xDS snapshot is built before the transformers are wired into the translator. At that pointt.transformersis still nil, so the translator quietly falls back to the legacy path, which names clusterscluster_<scheme>_<host>. The policy engine builds its routes later through the transformer path, which names the same clustersupstream_<name>_<host>_<port>. The two sides never agree on a name, so Envoy can't find any cluster the policy engine routes to. Nothing ever rebuilds the snapshot either, because control plane reconciliation skips APIs whosedeployed_athasn't changed. So the broken state stays until someone redeploys by hand.Goals
Approach
Fix the order instead of building the snapshot twice. This PR moves the transformer registry construction (
policyVersionResolver,RestAPITransformer,LLMTransformer,transform.NewRegistry) and thetranslator.SetTransformers(...)call up, so they run before the initialsnapshotManager.UpdateSnapshot(ctx, ""). The constructors just assemble structs, and everything they need (configStore,db,cfg,policyDefinitions) already exists at that point.policyManager.SetTransformers(transformerRegistry)stays where the policy manager is created and reuses the same registry.The issue thread suggested a different fix: keep the order as is, and add a second
UpdateSnapshotcall after the transformers are wired. I went with the reorder instead, for one reason. In a real deployment the gateway runtimes keep serving traffic while the controller restarts, and they reconnect the moment the xDS server starts (which happens before the transformers were wired). With the second-snapshot approach, Envoy would still get the broken legacy snapshot first, live traffic would 503 until the second snapshot arrives, and the routes would churn twice on every controller boot. With the reorder, the first snapshot is already correct, so there is no broken window and no extra rebuild.The event-gateway controller (
event-gateway/gateway-controller/cmd/controller/main.go) has the same startup order and shares the same translator package, so the same reorder is applied there too.On top of that, the silent fall-through in
TranslateConfigsnow logs a warning when no transformer is registered for a kind (exceptWebSubApi, which intentionally uses the legacy path).LlmProviderTemplateobjects live in a separate store map and never reachTranslateConfigs, so the warning can't fire for them.Nothing else changes:
Transformfails for one config, it still falls back to the legacy path with an error log, same as before.pkg/xdsorpkg/transform, so the change is contained to the two controller binaries.Note:
event-gateway/gateway-controller/cmd/controllercurrently fails to compile onmainfor an unrelated reason (itsadminserver.NewServercall was not updated when that function gained a middleware parameter). That breakage exists before this PR and is not touched here. This PR's change to that file was typechecked separately (with that one call patched locally) andgo vetpasses.User stories
As an operator running the Platform Gateway against an APIM control plane, I can restart or upgrade the gateway controller without every deployed API breaking with 503
cluster_not_found, and without redeploying each API from the Publisher afterwards.Automation tests
gateway-controllermodule suite passes (30 packages, includingpkg/xds,pkg/transform,cmd/controller). Theevent-gateway/gateway-controllermodule has no unit tests outside the pre-brokencmd/controllerpackage (see the note above).kubectl rollout restart(used to give 503cluster_not_found, now gives 200), and check Envoy's/clusters, which shows theupstream_<name>_<host>_<port>names right after the restart.Security checks
go vetis clean)Samples
N/A
Related PRs
None
Test environment
Go (toolchain go1.26.5), macOS (Apple Silicon). Reproduced and verified on: Gateway 1.2.0 Helm chart on k3s v1.35.0, WSO2 APIM 4.7.0 control plane, PostgreSQL 16.