Allow for user overridable boundary conditions

[lroberts36]

PR Summary

Add the option for injecting user specified boundary conditions in AddBoundaryExchangeTasks. These changes were duplicated in #1250 and #1236, so it seemed better to make a single small PR. Additionally, it switches to a task list for boundary communication in mesh.cpp.

PR Checklist

• Code passes cpplint
• New features are documented.
• Adds a test for any bugs fixed. Adds tests for new features.
• Code is formatted
• Changes are summarized in CHANGELOG.md
• Change is breaking (API, behavior, ...)
  • Change is additionally added to CHANGELOG.md in the breaking section
  • PR is marked as breaking
  • Short summary API changes at the top of the PR (plus optionally with an automated update/fix script)
• CI has been triggered on Darwin for performance regression tests.
• Docs build
• (@lanl.gov employees) Update copyright on changed files

[lroberts36]

This piece is new and is required for non-cell centered fields.

[Yurlungur]

If I understand correctly, this is adding the ability to have AddBoundaryExchangeTasks have an arbitrary boundary condition function... Is this only useful for the multigrid stuff? It might be good to document it.

[lroberts36]

If I understand correctly, this is adding the ability to have AddBoundaryExchangeTasks have an arbitrary boundary condition function... Is this only useful for the multigrid stuff? It might be good to document it.

@Yurlungur: It is useful for multigrid (e.g. if you want to apply a different boundary conditions on the same variable in different containers) and it was also useful for the forest of octrees stuff (hence why I made it its own PR).

[Yurlungur]

👍 I think adding a sentence to the docs would be useful. Otherwise LGTM.

[pdmullen]

@lroberts36 Hoping you can help me better understand this MR:

I originally thought this might be an additional API for enrolling custom boundary conditions... for example, if I had a boundary condition MyCustomBCPackageA that operates on a MeshData subset containing only variables from PackageA, then it might be useful to call AddBoundaryExchangeTasks(md_subset, old_args..., MyCustomBCPackageA). This might be nice so that I can avoid having checks in my custom boundary condition to see which meshdata subset register I am currently working with. However, looking at this PR's companion PR #1315, it looks like this new utility is instead being used as a selector between e.g., SetBoundary and ApplyBoundaryConditionsOnCoarseOrFineMD . Furthermore, since our custom boundary conditions typically are specific to e.g., InnerX1, I guess this new utility is not really for that... do I understand correctly? As such, I then wanted to confirm that independent of what function I pass to AddBoundaryExchangeTasks, that logical BCs are still always applied.

[lroberts36]

@lroberts36 Hoping you can help me better understand this MR:

I originally thought this might be an additional API for enrolling custom boundary conditions... for example, if I had a boundary condition MyCustomBCPackageA that operates on a MeshData subset containing only variables from PackageA, then it might be useful to call AddBoundaryExchangeTasks(md_subset, old_args..., MyCustomBCPackageA). This might be nice so that I can avoid having checks in my custom boundary condition to see which meshdata subset register I am currently working with. However, looking at this PR's companion PR #1315, it looks like this new utility is instead being used as a selector between e.g., SetBoundary and ApplyBoundaryConditionsOnCoarseOrFineMD . Furthermore, since our custom boundary conditions typically are specific to e.g., InnerX1, I guess this new utility is not really for that... do I understand correctly? As such, I then wanted to confirm that independent of what function I pass to AddBoundaryExchangeTasks, that logical BCs are still always applied.

@pdmullen: All this PR does is allow you to override ApplyBoundaryConditionsOnCoarseOrFineMD when calling AddBoundaryExchangeTasks with some user defined task(s). The user is responsible for making sure that the new task(s) actually set all the physical boundaries, deal with the coarse/fine flag, etc. It would be straightforward to add a wrapper that takes a BC of the form

template <CoordinateDirection DIR, BCSide SIDE>
void MyPackageBC(std::shared_ptr<MeshBlockData<Real>> &rc, bool coarse)

and goes over the required template parameters and blocks.

The interior ghost communication is unchanged by this PR.

[pdmullen]

@lroberts36 Hoping you can help me better understand this MR:
I originally thought this might be an additional API for enrolling custom boundary conditions... for example, if I had a boundary condition MyCustomBCPackageA that operates on a MeshData subset containing only variables from PackageA, then it might be useful to call AddBoundaryExchangeTasks(md_subset, old_args..., MyCustomBCPackageA). This might be nice so that I can avoid having checks in my custom boundary condition to see which meshdata subset register I am currently working with. However, looking at this PR's companion PR #1315, it looks like this new utility is instead being used as a selector between e.g., SetBoundary and ApplyBoundaryConditionsOnCoarseOrFineMD . Furthermore, since our custom boundary conditions typically are specific to e.g., InnerX1, I guess this new utility is not really for that... do I understand correctly? As such, I then wanted to confirm that independent of what function I pass to AddBoundaryExchangeTasks, that logical BCs are still always applied.

@pdmullen: All this PR does is allow you to override ApplyBoundaryConditionsOnCoarseOrFineMD when calling AddBoundaryExchangeTasks with some user defined task(s). The user is responsible for making sure that the new task(s) actually set all the physical boundaries, deal with the coarse/fine flag, etc. It would be straightforward to add a wrapper that takes a BC of the form

template <CoordinateDirection DIR, BCSide SIDE>
void MyPackageBC(std::shared_ptr<MeshBlockData<Real>> &rc, bool coarse)

and goes over the required template parameters and blocks.

The interior ghost communication is unchanged by this PR.

Got it! 😄 Thanks! I had a disconnect between user-defined BCs as I typically interface with them as opposed to an override to ApplyBoundaryConditionsOnCoarseOrFineMD. Makes sense now.