[Nexthop][m4062nhp] Fix agent EventBase double-drive abort on graceful exit

[vvasavada-nexthop]

Pre-submission checklist

• I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
• pre-commit run

Summary

The agent (both hw_agent and sw_agent) could abort with a prevLoopTid LOG(FATAL) ("Driving an EventBase while it is already being driven", EventBase.cpp) at the tail of shutdown. The abort happened because shutdown was terminated inline (exit() / std::exit()) from a thread whose event base was still live -- the thrift server's event base thread inside serve(), or a thrift stream-cleanup thread. Terminating there runs at-exit handlers that destroy folly's EventBaseManager and re-drive the still-live event base, tripping the CHECK and turning a clean exit into a SIGABRT/coredump. This fix makes every shutdown trigger (except if FLAGS_exit_for_any_hw_disconnect is set) converge on a single, thread-safe path: stop the services, return, let serve() unwind back to main(), and exit from the main thread once the thrift workers are joined - never calling exit() from an event base thread. Note that we are only making this agent in split switch agent paths.

Test Plan

Ran *AgentHwAcl* tests. Without fix, saw multiple
crashes:

[root@wdg152 netops]# grep "Check failed: expected == prevLoopTid" /var/facebook/logs/fboss/wedge_agent.log
F20260806 19:16:47.595602  3407 EventBase.cpp:580] Check failed: expected == prevLoopTid (-2 vs. 2277) Driving an EventBase (in thread 3407) while it is already being driven (in thread 2277) is forbidden.
...

No crashes with fix.

Also wrote a script to be run on DUT which signals SIGTERM to sw and hw agents and also disconnects while the serve() is live. Verified that no crash is seen with fix. But seen without fix.

With fix:

[root@wdg152 netops]# ./repro_mbist.sh --path sigterm
== [setup] clean cold boot: clear all warm/cold flags ==
   waiting 45s for CONFIGURED...
== [trigger:sigterm] SIGTERM to SW agent while serve() is live ==
   waiting 60s for teardown + next boot...
== [result] (from this cycle only) ==
  EventBase double-drive (must be 0)      : 0
  create_port INVALID PARAMETER (must be 0): 0
  reached CONFIGURED                      : 1
  lone HW can_warm_boot_0 (asymmetric)    : 1
  graceful exit ran (warm boot saved)     : 1

[root@wdg152 netops]# ./repro_mbist.sh --path disconnect
== [setup] clean cold boot: clear all warm/cold flags ==
   waiting 45s for CONFIGURED...
== [trigger:disconnect] SIGKILL ALL hw agents; SW loses last connection ==
   waiting 60s for teardown + next boot...
== [result] (from this cycle only) ==
  EventBase double-drive (must be 0)      : 0
  create_port INVALID PARAMETER (must be 0): 0
  reached CONFIGURED                      : 0
  detected all-HW-lost (disconnect path)  : 1
  SW cold boot marker created             : 1
  HW cold boot marker(s) created          : 1
  graceful exit SKIPPING warm boot save   : 1
  handler-missing error (must be 0)       : 0
== [markers on disk] ==
-rw-r--r--. 1 root root 0 Aug  6 18:59 /dev/shm/fboss/warm_boot/hw_cold_boot_once_0

[root@wdg152 netops]# ./repro_mbist.sh --path hw-sigterm
== [setup] clean cold boot: clear all warm/cold flags ==
   waiting 45s for CONFIGURED...
== [trigger:hw-sigterm] SIGTERM to each hw agent while its serve() is live ==
   waiting 60s for teardown + next boot...
== [result] (from this cycle only) ==
  EventBase double-drive (must be 0)      : 0
  create_port INVALID PARAMETER (must be 0): 0
  [Exit] Signal received (DBG2)           : 2
  [Exit] Total graceful Exit (DBG2)       : 2
  [Exit] destroying hardware agent (HW)   : 0
  [Exit] Cold boot detected (HW,if marker): 0
== [HW cold-boot markers on disk] ==
  idx=0 /dev/shm/fboss/exit_hw_for_cold_boot_0 : absent

Without fix:

[root@wdg152 netops]# ./repro_mbist.sh --path sigterm
== [setup] clean cold boot: clear all warm/cold flags ==
   waiting 45s for CONFIGURED...
== [trigger:sigterm] SIGTERM to SW agent while serve() is live ==
   waiting 60s for teardown + next boot...
== [result] (from this cycle only) ==
  EventBase double-drive (must be 0)      : 2
  create_port INVALID PARAMETER (must be 0): 0
  reached CONFIGURED                      : 1
  lone HW can_warm_boot_0 (asymmetric)    : 1
  graceful exit ran (warm boot saved)     : 1


[root@wdg152 netops]# ./repro_mbist.sh --path disconnect
== [setup] clean cold boot: clear all warm/cold flags ==
   waiting 45s for CONFIGURED...
== [trigger:disconnect] SIGKILL ALL hw agents; SW loses last connection ==
   waiting 60s for teardown + next boot...
== [result] (from this cycle only) ==
  EventBase double-drive (must be 0)      : 1
  create_port INVALID PARAMETER (must be 0): 0
  reached CONFIGURED                      : 0
  detected all-HW-lost (disconnect path)  : 1
  SW cold boot marker created             : 1
  HW cold boot marker(s) created          : 1
  graceful exit SKIPPING warm boot save   : 0
  handler-missing error (must be 0)       : 0
== [markers on disk] ==
-rw-r--r--. 1 root root 0 Aug  6 19:19 /dev/shm/fboss/warm_boot/hw_cold_boot_once_0

[root@wdg152 netops]# ./repro_mbist.sh --path hw-sigterm
== [setup] clean cold boot: clear all warm/cold flags ==
   waiting 45s for CONFIGURED...
== [trigger:hw-sigterm] SIGTERM to each hw agent while its serve() is live ==
   waiting 60s for teardown + next boot...
== [result] (from this cycle only) ==
  EventBase double-drive (must be 0)      : 2
  create_port INVALID PARAMETER (must be 0): 0
  [Exit] Signal received (DBG2)           : 1
  [Exit] Total graceful Exit (DBG2)       : 1
  [Exit] destroying hardware agent (HW)   : 0
  [Exit] Cold boot detected (HW,if marker): 0
== [HW cold-boot markers on disk] ==
  idx=0 /dev/shm/fboss/exit_hw_for_cold_boot_0 : absent