MOOSE - MFEM benchmarks

[1wertyuiop5]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

Hi, I am investigating using MOOSE and/or MFEM for an elasticity problem and was interested in the comparative performance of MOOSE with both the Libmesh and MFEM backends, and pure MFEM. The flexibility of MOOSE is a big plus for my application but performance is also pretty key.

A few quick benchmark tests using the MFEM example 2, MOOSE-MFEM linear elasticity example and a comparative MOOSE beam bending test show MFEM being quite a bit faster than MOOSE, but I was surprised to see such a large gap between pure MFEM and MFEM-MOOSE. All of these are using BoomerAMG as a preconditioner with the same settings, and PCG as the solver.

Below is the table showing my results (please note this was quite a quick study), I'm interested to see if anyone has any comments on the reason for the slowdown with MFEM-MOOSE compared to pure MFEM. Or if anyone has done any other performance studies on MOOSE-MFEM I would be grateful to hear! I've also attached the MOOSE input file I used.

Cores	Elements	Degrees of freedom	MOOSE run time [s]	MFEM – MOOSE run time [s]	MFEM run time [s]
2	4096	15795	3.07	2.15	1.60
4	4096	15795	2.28	1.61	0.91
8	4096	15795	1.91	1.33	0.82
16	4096	15795	1.85	1.33	0.77
2	32768	111843	21.33	19.87	8.55
4	32768	111843	13.53	10.81	4.96
8	32768	111843	9.45	6.23	3.21
16	32768	111843	7.75	4.11	2.31
4	262144	839619	116.90	121.67	41.61
8	262144	839619	77.29	67.30	25.26
16	262144	839619	60.84	40.90	16.29
24	262144	839619	62.35	35.41	15.31

[cantilever.txt](https://github.com/user-attachments/files/32195886/cantilever.txt)

[alexanderianblair]

Hi @1wertyuiop5 - do you happen to have your MOOSE-MFEM inputs that you were comparing to MFEM here?

I can think of a few possibilities (particularly around some re-assembly in deformed problems like this), but it'd be good to rule out any solver differences first. Also tagging @Heinrich-BR, who has last performed some comparison runs between MFEM and MOOSE-MFEM examples

[1wertyuiop5]

Thanks, I've attached the MOOSE-MFEM input file below. I was comparing against MFEM example 2 using the same mesh and preconditioner and solver settings. The only modifications from the base examples are the material properties and refinement levels.

mfem_cantilever.txt

[alexanderianblair]

Thanks - I think there might indeed be a solver difference there, as I believe MFEM example 2 defaults to using BoomerAMG set up using amg->SetSystemsOptions(dim, reorder_space);.

Easy option to add, if this is it - let me run some tests

[1wertyuiop5]

I had noticed this in the code and removing amg->SetSystemsOptions(dim, reorder_space) in the MFEM code actually made it fail to converge for me. Under the hood it sets the HYPRE_BoomerAMGSetNumFunctions(amg_precond, dim) option, which I discovered through some digging is set automatically in MOOSE/PETSc depending on the problem. I had assumed from this and the fact that the MOOSE-MFEM version converged that it might get set somewhere else? But that would definitely explain some of the difference if so.

Thanks, really appreciate the help!

[alexanderianblair]

Quick draft of a branch adding the ability to set the SystemOptions to make things consistent with the solver in MFEM Example 2 here - https://github.com/alexanderianblair/platypus-moose/tree/alexanderianblair/mfem-hypreboomeramg-options

The new user parameters of note to set, for consistency, are Solvers/amg/fespace=H1FESpace and Solvers/amg/vector_treatment=by_component. Does that bring your timings closer to native MFEM? Looks like it's making a significant difference my side, locally

[1wertyuiop5]

Thanks, this does make a significant difference for me as well, timings are now much closer to pure MFEM code.

[Heinrich-BR]

@1wertyuiop5 happy to hear that the SetSystemsOptions issue closes the gap. Just for reference, as it might be useful for future problems, let me point out that what's slowing down your runs is probably a twofold issue. I notice that in your [Solvers] block you don't pass to HypreBoomerAMG the fespace parameter which is to be set for elasticity problems. Not setting SetElasticityOptions (nor SetSystemsOptions since that wasn't available) makes it so that HypreBoomerAMG treats the problem as a scalar one, and that slows down convergence quite a bit. In this particular case, going the SetSystemsOptions route in @alexanderianblair's branch is still probably the most performant solution, but I believe you can already recover a large chunk of the performance in the main MOOSE branch by setting your Solvers block to this:

[Solvers]
  [boomeramg]
    type = MFEMHypreBoomerAMG
    fespace = H1FESpace   # Set elasticity options for HypreBoomerAMG
    l_max_its = 1
    l_tol = 0
    strength_threshold = 0.7
    print_level = 1
  []
  [main]
    type = MFEMHyprePCG
    preconditioner = boomeramg
    l_max_its = 500
    l_tol = 1e-8
    #l_abs_tol = 0.0
    print_level = 1
  []
[]

[1wertyuiop5]

Thanks very much! I'd missed that that option was available in the MFEM-MOOSE implementation

[alexanderianblair]

Quick draft of a branch adding the ability to set the SystemOptions to make things consistent with the solver in MFEM Example 2 here - https://github.com/alexanderianblair/platypus-moose/tree/alexanderianblair/mfem-hypreboomeramg-options

The new user parameters of note to set, for consistency, are Solvers/amg/fespace=H1FESpace and Solvers/amg/vector_treatment=by_component. Does that bring your timings closer to native MFEM? Looks like it's making a significant difference my side, locally

For completeness - this functionality has now been merged into next following the merge of #33749

[nmnobre]

Hi @1wertyuiop5, would it be too much work to repeat the experiment above and get a new table while setting (a) -elast for ex2p and (b) fespace = H1FESpace on the MOOSE input file (like @Heinrich-BR just suggested)? Note you can already do this with the upstream version of MOOSE. I will also note that SetElasticityOptions() calls SetSystemsOptions().

Annoyingly, there are still some very small FP discrepancies in the matrix, which means residuals and sparsity patterns aren't exactly the same on my side between MFEM and MOOSE-MFEM... I spent all morning on this, and couldn't quite figure out why. EDIT: it's because we call SetNodalFESpace() before the solve! Skipping that step, we get the exact same results, verbatim.

[nmnobre]

Cores	Elems	DoFs	MFEM-MOOSE (rbm) [s]	MFEM (-elast) [s]	MFEM–MOOSE (bc) [s]	MFEM (-sys) [s]
4	262144	839619	33.94	32.63	16.33	15.06
8	262144	839619	21.57	20.46	10.67	9.88
16	262144	839619	17.38	16.53	10.38	9.95
-	Overhead	geomean	4.9%	-	6.6%	-

[1wertyuiop5]

Hi, I just got round to running these new tests and I'm actually seeing quite a bit of a slowdown when I do this. I am currently running these tests on a virtual machine and although I am the only person using it, it might be oversubscribed by the hypervisor. I've asked for MOOSE with MFEM backend to be installed on the cluster I have access to so I can run a node exclusively and I should be able to get some more reliable performance measures. Will let you know when I've run them!

[nmnobre]

Hey, the idea was simply to produce a like-for-like comparison, but I have done that now, so don't worry.

[1wertyuiop5]

Thanks, I think I'll still have a look when I get a chance on the cluster, but appreciate it

[lindsayad]

@1wertyuiop5 could you also share your libmesh-MOOSE input?

[1wertyuiop5]

Hi Alex, I've attached it below
cantilever.txt