[Bug]: Gateway controller restart silently breaks all API traffic with 503 cluster_not_found until every API is redeployed

[MinuraKariyawasam]

Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/Other (Anything else which does not match above categories)

Description

We have a gateway (1.2.0) connected to an on-prem APIM 4.7.0 control plane, deploying APIs top-down from the Publisher. Everything works - APIs deploy, JWT auth works, calls return 200.

But whenever the gateway controller restarts (a helm upgrade, a pod restart, anything), every API starts returning 503. Auth still works - calls without a token still get 401 - so it looks like the backend is down, but it isn't. Envoy's access log shows the real reason:

{"respCd":503, "respCdDtl":"cluster_not_found", "respFlg":"NC"}

The only way to recover is to redeploy each API from the Publisher. Then it works again - until the next restart.

How to slove this? is there anything that we're missing on our side?

Minura.

Steps to Reproduce

Steps to Reproduce

1. Connect a gateway (1.2.0) to an APIM 4.7.0 control plane (Platform Gateway
   mode, top-down).

2. In the Publisher, create a simple REST API - we used
   https://jsonplaceholder.typicode.com/todos as the endpoint, context
   /todos, version 1.0.0. Deploy and publish it.

3. Call it — works:

   curl -k https://<gateway-host>:8443/todos/1.0.0 -H "Authorization: Bearer <token>"
   # HTTP 200

4. Restart the controller:

   kubectl -n <ns> rollout restart deploy/<release>-controller

5. Call it again, same request:

   # HTTP 503 (empty body)
   # without a token you still get 401, so auth is fine

6. To see the mismatch, compare Envoy's cluster names before and after the
   restart (router admin port 9901):

   curl -s http://localhost:9901/clusters | cut -d: -f1 | sort -u
   # before restart: upstream_main_jsonplaceholder.typicode.com_443
   # after restart:  cluster_https_jsonplaceholder_typicode_com

7. Redeploy the API from the Publisher → 200 again. Restart again → 503 again.

---

Environment Details (with versions)

Environment Details (with versions)

Gateway        : 1.2.0 (Helm chart, ghcr.io images)
Control plane  : WSO2 APIM 4.7.0, gateway_type = "APIPlatform"
Kubernetes     : k3s v1.35.0 (local, Colima on macOS / Apple Silicon)
Storage        : PostgreSQL 16
Policy         : JWT Auth v1.3 attached at API level, JWKS from APIM

[MinuraKariyawasam]

After a restart, the Envoy cluster for our upstream is named cluster_https_jsonplaceholder_typicode_com, but the policy engine routes requests using the x-target-upstream header set to upstream_main_jsonplaceholder.typicode.com_443. Those names don't match, so Envoy can't find the cluster. After a fresh deploy from the Publisher, the cluster is named upstream_main_... and everything matches.

When checking gateway-controller/cmd/controller/main.go, we found these:

• The initial xDS snapshot is built at line 382, but the transformers are only wired into the Envoy translator at line 444. So the startup snapshot seems to be built by the legacy translation path (which names clusters differently), while the policy engine always uses the new path.
• The fall-through in pkg/xds/translator.go:722 doesn't log anything, so there's no hint in the logs when the legacy path is taken.
• It also never fixes itself - the control plane sync skips APIs that haven't changed, so the broken snapshot stays until a manual redeploy.

[ShalkiWenushika]

Hi Minura,
I tested this with an APIM 4.7.0 pack and Gateway 1.2.0 pack. I restarted the Gateway Controller and invoked an API that was deployed before the restart, but the invocations are still returning 200. I couldn’t reproduce the issue.

Anyway, restarting the Gateway Controller should not affect the Gateway Runtime. Could you please share the Gateway Controller and Gateway Runtime logs if the issue persists?

[ShalkiWenushika]

As I stated earlier, I was not able to reproduce the issue by simply following the steps below:

1. Run APIM 4.7.0 locally.
2. Run the 1.2.0 Platform Gateway via Helm.
3. Deploy an API.
4. Invoke the API → 200 response.
5. Roll out restart the Gateway Controller.
6. Invoke the API again → it still returned 200 for me.

However, after further debugging at the code level, the identified root cause may be possible and is worth verifying.

Following are the findings, and I’m transferring this to the Product team for further verification.

Root Cause                                                                                                                                                                  
                                                                                                                                                                              
  This is an order-of-initialization bug in main.go. Two independent subsystems — the Envoy xDS snapshot and the policy engine snapshot — end up with structurally different  
  cluster names because the Envoy snapshot is built before the transformer registry is wired in. They never agree on what a cluster is called, so Envoy returns             
  cluster_not_found on every request after a restart.

  The sequence in main.go:

  Line 382:  snapshotManager.UpdateSnapshot(ctx, "")   ← Envoy snapshot built here
                t.transformers is nil → falls to legacy path
                legacy path: sanitizeClusterName() → "cluster_https_jsonplaceholder_typicode_com"

  Line 444:  translator.SetTransformers({...})          ← transformers wired HERE (too late)

  Lines 456–470: loadRuntimeConfigsFromExistingAPIConfigurations(...)
                uses transform/restapi.go:513
                clusterKey = fmt.Sprintf("upstream_%s_%s_%d", upstreamName, host, port)
                → "upstream_main_jsonplaceholder.typicode.com_443"

  Line 475:  policySnapshotManager.UpdateSnapshot(ctx)  ← policy snapshot built with new names
                policy engine routes reference: "upstream_main_jsonplaceholder.typicode.com_443"

  No second Envoy snapshot is ever generated. Envoy keeps the stale legacy-path snapshot indefinitely.

  The two naming functions:

  Legacy path (translator.go:2597–2601) — used for the startup Envoy snapshot:
  func (t *Translator) sanitizeClusterName(hostname, scheme string) string {
      name := strings.ReplaceAll(hostname, ".", "_")
      name = strings.ReplaceAll(name, ":", "_")
      return "cluster_" + scheme + "_" + name
      // → "cluster_https_jsonplaceholder_typicode_com"
  }

  New transformer path (transform/restapi.go:513) — used for the policy snapshot and all post-restart deploys:
  clusterKey := fmt.Sprintf("upstream_%s_%s_%d", upstreamName, parsedURL.Hostname(), port)
  // → "upstream_main_jsonplaceholder.typicode.com_443"

  Why the translator falls through silently

  In translator.go:722:
  if transformer, ok := t.transformers[cfg.Kind]; ok {
      // new path
  } else {
      // falls through to legacy — no log at all when transformers map is nil
  }

  When t.transformers is nil (as it is at line 382), nil["RestApi"] returns ok = false in Go — no panic, no log, silently takes the legacy path.

  Why a redeploy from Publisher fixes it

  After startup, transformers are already wired. A Publisher deploy triggers a fresh UpdateSnapshot call that goes through the new transformer path — both Envoy and the
  policy engine now agree on upstream_main_.... The fix persists until the next restart wipes the Envoy snapshot back to legacy naming.

  Why the control plane reconciliation doesn't auto-heal it

  The reconciler compares deployed_at timestamps. APIs already in the DB whose deployed_at hasn't changed are skipped — no UpdateSnapshot is triggered for them, so the broken
   Envoy snapshot is never corrected.

---
  The Fix

  The Envoy xDS snapshot needs to be regenerated after transformers are wired and runtime configs are loaded. There's a missing second snapshotManager.UpdateSnapshot call.
  The correct order is:

  1. Wire transformers            (line 444 — already correct)
  2. Load runtime configs         (lines 456–470 — already correct)
  3. Rebuild Envoy snapshot       (MISSING — needs to be added here)
  4. Build policy snapshot        (line 475 — already correct)

  In main.go, between lines 470 and 472, a second snapshot update call is needed:

  log.Info("Loaded runtime configs from API configurations",
      slog.Int("total_apis", len(loadedAPIs)),
      slog.Int("configs_loaded", loadedCount))

  // ← ADD: regenerate Envoy snapshot now that transformers are wired and
  // runtime configs are loaded, so cluster names match what the policy
  // engine's routes reference ("upstream_main_..." not "cluster_https_...").
  log.Info("Regenerating xDS snapshot with transformer-path cluster names")
  ctx, cancel = context.WithTimeout(context.Background(), 10*time.Second)
  if err := snapshotManager.UpdateSnapshot(ctx, ""); err != nil {
      log.Warn("Failed to regenerate xDS snapshot after transformer init", slog.Any("error", err))
  }
  cancel()

  // Generate initial policy snapshot
  log.Info("Generating initial policy xDS snapshot")

  Additionally, the silent fall-through at translator.go:722 should emit a warning log so this class of regression is visible in future:

  if transformer, ok := t.transformers[cfg.Kind]; ok {
      // ...
  } else if cfg.Kind != "WebSubApi" {
      // Log when falling back unexpectedly — transformers should be set for all non-WebSub kinds
      log.Warn("No transformer registered for API kind, using legacy translation path",
          slog.String("kind", cfg.Kind),
          slog.String("id", cfg.UUID))
  }

[ShalkiWenushika]

I was able to reproduce the issue today. The issue appears to be intermittent. As @MinuraKariyawasam mentioned, when the request returns a 503, the cluster name is like cluster_https_test_requestcatcher_com, whereas when it returns 200, the cluster name is like upstream_main_test.requestcatcher.com_443.

Redeploying a revision to the Gateway resolves the issue.

@renuka-fernando Can we expect an ETA for a fix, please?

[MinuraKariyawasam]

Thanks for reproducing and confirming the root cause.

I went and opened a PR with the fix: #3236

Two notes on it.

First, on the "intermittent" behavior. The bug itself is deterministic. It only looks intermittent because any deploy event after the restart triggers a fresh snapshot through the transformer path, which quietly repairs the naming. If nothing is deployed after the restart, the 503 happens every single time. That's probably why the first reproduction attempts returned 200.

Second, the PR fixes the ordering instead of adding a second UpdateSnapshot call. The reason: the gateway runtimes keep serving traffic while the controller restarts, and they reconnect the moment the xDS server starts (which happens before the transformers are wired). With a second-snapshot approach, Envoy would still receive the broken legacy snapshot first, so live traffic would 503 in the window until the second snapshot arrives, and the routes would churn twice on every controller boot. Moving the transformer wiring to before the initial snapshot means the first snapshot is already correct, so there is no broken window and no extra rebuild. The warning log you suggested for the silent fall-through in translator.go is included as well.

The event-gateway controller has the same initialization order and shares the same translator package, so the same reorder is applied there in the PR.

@ShalkiWenushika could you please open an internal issue pointing to this PR?

Happy to make any changes the team wants on the PR.

Minura.

[ShalkiWenushika]

I have opened an internal ticket for this, and the progress will be tracked through that issue.

Thanks, @MinuraKariyawasam, for sending a PR. The team will review it.