See if `pyop2.safe_noncollective` could used for parallel error checking.

[JHopeCollins]

These lines to check whether a directory exists do all the error checking only on rank 0 then broadcast the result to all ranks to fail/succeed collectively:

gusto/gusto/core/io.py

Lines 393 to 423 in bf6ed72

	if any([self.output.dump_vtus, self.output.dump_nc,
	self.output.dumplist_latlon, self.output.dump_diagnostics,
	self.output.point_data, self.output.checkpoint and not pick_up]):
	# setup output directory and check that it does not already exist
	self.dumpdir = path.join("results", self.output.dirname)
	running_tests = '--running-tests' in sys.argv or "pytest" in self.output.dirname
	# Raising exceptions needs to be done in parallel
	if self.mesh.comm.rank == 0:
	# Create results directory if it doesn't already exist
	if not path.exists(self.dumpdir):
	try:
	makedirs(self.dumpdir)
	except OSError as e:
	error = e
	raise_parallel_exception = 2
	elif not (running_tests or pick_up):
	# Throw an error if directory already exists, unless we
	# are picking up or running tests
	raise_parallel_exception = 1
	# Gather errors from each rank and raise appropriate error everywhere
	# This allreduce also ensures that all ranks are in sync wrt the results dir
	raise_exception = self.mesh.comm.allreduce(raise_parallel_exception, op=MPI.MAX)
	if raise_exception == 1:
	raise GustoIOError(f'results directory {self.dumpdir} already exists')
	elif raise_exception == 2:
	if error:
	raise error
	else:
	raise OSError('Check error message on rank 0')

pyop2 now has a safe_noncollective helper function for doing exactly this type of logic, which would the amount of MPI-aware logic that gusto has to maintain.
https://github.com/firedrakeproject/firedrake/blob/175549a2805066331fde99803b2af31f8fd1b01c/pyop2/mpi.py#L569-L604

There might also be other places where safe_noncollective could be used but I haven't looked in detail yet.