Cilium failed to start with kubernetes-operator installed in Kubernetes Cluster

[janisii]

Before posting

• I searched existing discussions and issues, including closed ones, and checked the relevant docs.
• I believe this is a product bug rather than a configuration or setup question.
• I can reproduce this issue, or for intermittent issues I've included trigger, frequency, and timing details below.
• I removed or anonymized sensitive data from logs, screenshots, and configuration.

Affected area

Kubernetes / Operator

Deployment type

Self-hosted - advanced/custom deployment

Operating system or environment

Kubernetes

NetBird version and upgrade status

Kubernetes: kubernetes-operator-0.2.2

Did this work before?

Yes, this worked before

Regression details

Kubernetes: kubernetes-operator-0.2.2

Summary

Cilium pods could not start when netbird-operator-kubernetes-operator-webhook-service were not available. On upgrade Kubernetes cluster upgrade this left nodes in Not Ready state.

kubectl -n kube-system describe ds cilium
(....)
Events:
  Type     Reason        Age                  From                  Message
  ----     ------        ----                 ----                  -------
  Warning  FailedCreate  95s (x12 over 102m)  daemonset-controller  Error creating: Internal error occurred: failed calling webhook "mpod-v1.netbird.io": failed to call webhook: Post "https://netbird-operator-kubernetes-operator-webhook-service.netbird.svc:443/mutate--v1-pod?timeout=10s": No agent available

Current behavior

I had to uninstall netbird kubernetes-operator to make cluster available again.

Expected behavior

Cilium should start whether the NetBird webhook service is available or not.

Steps to reproduce

Install netbird kubernetes-operator and perform Kubernetes upgrade. When new node will be provisioned, it will not became available because of cilium starting issue.

Environment and topology

Scaleway Kapsule Managed Kubernetes service (Kubernetes version v1.34.6).

Self-hosted details, if available

No response

Logs, status output, or debug evidence

kubectl -n kube-system describe ds cilium

Events:
  Type     Reason        Age                  From                  Message
  ----     ------        ----                 ----                  -------
  Warning  FailedCreate  95s (x12 over 102m)  daemonset-controller  Error creating: Internal error occurred: failed calling webhook "mpod-v1.netbird.io": failed to call webhook: Post "https://netbird-operator-kubernetes-operator-webhook-service.netbird.svc:443/mutate--v1-pod?timeout=10s": No agent available

Related issues or discussions

No response

Impact

No response

Additional context

No response

[phillebaba]

@janisii I have been doing a lot of refactoring work with the operator, implementing completely new resources. Could you try the latest version of the operator and see if you still have issues?

[janisii]

@janisii I have been doing a lot of refactoring work with the operator, implementing completely new resources. Could you try the latest version of the operator and see if you still have issues?

helm repo update
helm search repo netbirdio
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION                   
netbirdio/kubernetes-operator           0.2.2           0.2.2           NetBird Kubernetes Operator   
netbirdio/netbird-operator              0.1.0           v0.1.0          A Helm chart for Kubernetes   
netbirdio/netbird-operator-config       0.1.0           0.2.0           NetBird Operator Configuration

helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                            APP VERSION
netbird-operator        netbird         5               2026-05-07 20:17:48.474469 +0300 EEST   deployed        kubernetes-operator-0.2.2        0.2.2

I don't see any new version.

[phillebaba]

The Helm chart is now pushed as an OCI artifact. It now lives here.

https://github.com/netbirdio/kubernetes-operator/pkgs/container/helm-charts%2Fnetbird-operator

I have also updated the getting started docs. It is totally fine to upgrade from one chart source to another.

https://docs.netbird.io/manage/integrations/kubernetes#steps

Sorry about the the large changes, but it was necessary to ensure long term stability.

[janisii]

The Helm chart is now pushed as an OCI artifact. It now lives here.

https://github.com/netbirdio/kubernetes-operator/pkgs/container/helm-charts%2Fnetbird-operator

I have also updated the getting started docs. It is totally fine to upgrade from one chart source to another.

https://docs.netbird.io/manage/integrations/kubernetes#steps

Sorry about the the large changes, but it was necessary to ensure long term stability.

helm upgrade -i netbird-operator oci://ghcr.io/netbirdio/helm-charts/netbird-operator -f values.yaml --version 0.5.0 -n netbird --create-namespace

Upgrade to 0.5.0 failing.

2026-05-21T08:10:42Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate	{"cert": "/tmp/k8s-webhook-server/serving-certs/tls.crt", "key": "/tmp/k8s-webhook-server/serving-certs/tls.key"}
2026-05-21T08:10:42Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "/v1, Kind=Pod", "path": "/mutate--v1-pod"}
2026-05-21T08:10:42Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate--v1-pod"}
2026-05-21T08:10:42Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "netbird.io/v1, Kind=NBGroup", "path": "/validate-netbird-io-v1-nbgroup"}
2026-05-21T08:10:42Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-netbird-io-v1-nbgroup"}
2026-05-21T08:10:42Z	ERROR	setup	Failed to create controller	{"controller": "NetworkResource", "error": "unable to retrieve the complete list of server APIs: netbird.io/v1alpha1: no matches for netbird.io/v1alpha1, Resource="}
main.main
	./main.go:312
runtime.main
	runtime/proc.go:290

My values.yaml file.

webhook:
  namespaceSelectors:
    - key: kubernetes.io/metadata.name
      operator: In
      values:
      - netbird
      - vds-dev
  objectSelector:
    matchExpressions:
      - key: netbird.io/managed
        operator: Exists

ingress:
  enabled: true
  router:
    enabled: true

cluster:
  dns: svc.cluster.local
  name: kubernetes

managementURL: https://netbird.<redacted>.com

netbirdAPI:
  keyFromSecret:
   name: "netbird-mgmt-api-key"
   key: "NB_API_KEY"

[janisii]

I see, that ingress is DEPRECATED, but cannot find any information in values.yaml on NetworkRouters and NetworkResources, how to configure them.

# DEPRECATED: Use NetworkRouters and NetworkResources instead.