Problem
When the DOCA-OFED driver container runs openibd restart during init (or pod restart), all Mellanox interfaces go down. On nodes where the primary uplink (OAM/management) is on a Mellanox NIC, that interface often doesn’t come back up in time, or systemd-networkd hits "DHCP4 CLIENT: Failed to set IAID+DUID: Device or resource busy" and marks the interface failed. The node then loses its primary IP and kubelet loses the lease, so the node goes NotReady and doesn’t recover without a reboot. We’ve seen this consistently when the ConnectX card is used for OAM (similar to #2176).
What would help
A way to run a configurable step on the host after the driver is ready (e.g. after .driver-ready is written). For example:
- A list of host commands to run (e.g.
networkctl reconfigure <iface>, networkctl renew <iface> for the primary uplink), or
- An optional “post-ready” hook or sidecar that the operator can run in the driver pod (or a separate pod) with access to run commands in the host’s network namespace (e.g. via nsenter).
That would let us bring the primary interface back and renew DHCP after the driver is up, without maintaining a custom driver daemonset or a separate recovery DaemonSet.
Current workaround
We run a separate DaemonSet that watches /run/mellanox/drivers/.driver-ready and, when it appears, runs nsenter -t 1 -m -n -- networkctl reconfigure / renew on the OAM interface. We’d prefer to do this through the operator once there’s a supported mechanism so we can drop the custom component.
Thanks for considering this.
Problem
When the DOCA-OFED driver container runs
openibd restartduring init (or pod restart), all Mellanox interfaces go down. On nodes where the primary uplink (OAM/management) is on a Mellanox NIC, that interface often doesn’t come back up in time, or systemd-networkd hits "DHCP4 CLIENT: Failed to set IAID+DUID: Device or resource busy" and marks the interface failed. The node then loses its primary IP and kubelet loses the lease, so the node goes NotReady and doesn’t recover without a reboot. We’ve seen this consistently when the ConnectX card is used for OAM (similar to #2176).What would help
A way to run a configurable step on the host after the driver is ready (e.g. after
.driver-readyis written). For example:networkctl reconfigure <iface>,networkctl renew <iface>for the primary uplink), orThat would let us bring the primary interface back and renew DHCP after the driver is up, without maintaining a custom driver daemonset or a separate recovery DaemonSet.
Current workaround
We run a separate DaemonSet that watches
/run/mellanox/drivers/.driver-readyand, when it appears, runsnsenter -t 1 -m -n -- networkctl reconfigure/renewon the OAM interface. We’d prefer to do this through the operator once there’s a supported mechanism so we can drop the custom component.Thanks for considering this.