Stop networks that a test recovers into

Two governance_test nodes outlived their test by about 8.5 minutes in CI,
logging roughly 660 lines a minute on a join retry loop through the rest of
the bucket.

`test_all_members` stops the network it is given and recovers into a new one,
which it returns. `gov` rebound its local `network` to that, but the enclosing
`infra.network.network` context manager still owns the original, so nothing
ever stopped the recovered network. The nodes that leaked were the ones
`test_no_quote` and `test_node_data` deliberately add without ever trusting,
which is why they sat retrying rather than exiting.

Give `gov` explicit ownership of the recovered network and stop it in a
finally block. The node logs of the leaked nodes were clean, so this reaps
them without newly failing on anything.

`partitions_test.run_recovery_elections` has the same shape:
`test_recovery_elections` stops both the original and its intermediate network
but returns a third that nobody stopped. Fix it the same way, which matters
more now that the partitions groups run concurrently and a stray node is a
neighbour of a live test rather than of an idle runner.

The pattern is wider than these two: any test that recovers into a new network
and returns it leaks unless its caller takes ownership, and recovery.py does
this in several places. Left alone here.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 24ad6978-a80f-45bc-b73b-268aeaf2b30e

Author: achamayou

tests/governance.py

@@ -568,16 +568,26 @@ def gov(args):
         test_consensus_status(network, args)
         test_member_data(network, args)
         test_ack_state_digest_update(network, args)
-        network = test_all_members(network, args)
-        test_user(network, args)
-        test_jinja_templates(network, args)
-        test_no_quote(network, args)
-        test_node_data(network, args)
-        test_each_node_cert_renewal(network, args)
-        test_binding_proposal_to_service_identity(network, args)
-        test_all_nodes_cert_renewal(network, args)
-        test_service_cert_renewal(network, args)
-        test_service_cert_renewal_extended(network, args)
+
+        # test_all_members stops this network and recovers into a new one, which
+        # the enclosing context manager does not own: it still holds the
+        # original. Stop the recovered network here, or its nodes outlive the
+        # test. That includes the deliberately untrusted nodes added by
+        # test_no_quote and test_node_data, which then sit in a join retry loop
+        # for the rest of the CI job.
+        recovered_network = test_all_members(network, args)
+        try:
+            test_user(recovered_network, args)
+            test_jinja_templates(recovered_network, args)
+            test_no_quote(recovered_network, args)
+            test_node_data(recovered_network, args)
+            test_each_node_cert_renewal(recovered_network, args)
+            test_binding_proposal_to_service_identity(recovered_network, args)
+            test_all_nodes_cert_renewal(recovered_network, args)
+            test_service_cert_renewal(recovered_network, args)
+            test_service_cert_renewal_extended(recovered_network, args)
+        finally:
+            recovered_network.stop_all_nodes(skip_verification=True)
 
 
 # These tests requiring starting up + shutting down a node with specific

tests/partitions_test.py

@@ -1607,8 +1607,16 @@ def run_forwarding_and_sessions(args):
 
 def run_recovery_elections(args):
     with partitioned_network(args) as network:
-        network = test_recovery_elections(network, args)
-        test_ledger_invariants(network, args)
+        # test_recovery_elections stops this network and recovers into a new
+        # one, which the context manager does not own: it still holds the
+        # original. Stop the returned network here, or its nodes outlive the
+        # test.
+        recovery_network = test_recovery_elections(network, args)
+        try:
+            test_ledger_invariants(recovery_network, args)
+        finally:
+            if recovery_network is not network:
+                recovery_network.stop_all_nodes(skip_verification=True)
 
 
 if __name__ == "__main__":