Should the NUOPC cap assume hybrid runs will initialize from restarts (but allow users to override that choice)?

[mnlevy1981]

In #397 we pointed out that there is no mechanism to have the MARBL tracers initialized from a restart file in a hybrid run, we always use the provided IC file. I started working on a PR to introduce MARBL_TRACERS_IC_FILE_IS_Z use MOM_read_data() instead of MOM_initialize_tracer_from_Z() when that flag is false (similar to what is done in the CFC module), but I ran into an issue where there are so many MARBL tracers that they are split across multiple restart files in our workhorse 2/3° grid.

Thinking about it more, the physics is going to run into a similar problem in the 1/12° grid (and possibly the 1/4° grid); for the former, we are only writing one variable per restart file so T might be in ${CASE}.mom6.r.${DATE}.nc but S will be in ${CASE}.mom6.r.${DATE}_1.nc and other fields will be in other files.

One possible solution would be to have the NUOPC cap treat RUN_TYPE=hybrid the same way it treats branch runs and continuation runs -- assume all tracers will be initialized from restart files. There are two key points to this setup:

1. By default we should assume all tracers will be read from restarts, so we want may_reinit = .false. to avoid silently falling over to different IC files if tracers happen to be missing from the restarts (e.g. you are running with MARBL tracers and don't realize the case you are starting a hybrid run from did not have MARBL turned on)
2. Each tracer module should have a must_reinit flag that allows users to say "in this run, even if my tracers are in the restart files I actually want to initialize from this provided IC file." By default must_reinit should be false, and in each tracer module must_reinit = .true. implies may_reinit = .true. as well (if you want MARBL tracers to be initialized from an IC file, it doesn't matter if they are in the restart or not).

Once this is in place, we can clean up a lot of the logic in MOM_input.yaml over in MOM_interface because we won't need blocks like

    THICKNESS_FILE:
        description: "The name of the thickness file"
        datatype: string
        value:
            $RUN_TYPE == "hybrid":
                = f'./{$RUN_REFCASE}.mom6{$INST_SUFFIX}.r.{$RUN_REFDATE}-{$RUN_REFTOD}.nc'

and

    TS_FILE:
        description: "The initial condition file for the temperature and salinity."
        datatype: string
        value:
            $RUN_TYPE == "hybrid":
                = f'./{$RUN_REFCASE}.mom6{$INST_SUFFIX}.r.{$RUN_REFDATE}-{$RUN_REFTOD}.nc'
    TEMP_IC_VAR:
        description: "The initial condition variable for potential temperature"
        datatype: string
        value:
            $RUN_TYPE == "hybrid": "Temp"
    SALT_IC_VAR:
        description: "The initial condition variable for the salinity."
        datatype: string
        value:
            $RUN_TYPE == "hybrid": "Salt"

and

    VELOCITY_CONFIG:
        description: "A string that determines how the initial velocities are specified for a new run."
        datatype: string
        value:
            $RUN_TYPE == "hybrid": "file"
    VELOCITY_FILE:
        description: "The name of the velocity initial condition file."
        datatype: string
        value:
            $RUN_TYPE == "hybrid":
                = f'./{$RUN_REFCASE}.mom6{$INST_SUFFIX}.r.{$RUN_REFDATE}-{$RUN_REFTOD}.nc'

@alperaltuntas, @klindsay28, and I discussed this briefly, and think we need to wait until after the CESM Workshop (June 15 - 17) before tackling this to make sure we get the default behavior right and test all the edge cases thoroughly. I'm happy to help with the MARBL tracers side of things, and even to use that as an example of how to implement the must_reinit flag, but don't know what needs to happen in the NUOPC cap or where to add the must_reinit flag for the physics side of things.