Multi-core operation reports an error

[Jinko12]

Hello!
This kind of error always occurs when I use parallel computing （ImpactZ）
Number of grids less than number of processorsin y!
Number of grids less than number of processorsin y!
Number of grids less than number of processorsin y!
Number of grids less than number of processorsin y!
Number of grids less than number of processorsin y!
(Sometimes in the z direction)
May I ask what the reason is? How should I solve it

[Jinko12]

Sorry, this is an issue with impactZ rather than lume.

[ken-lauer]

Here's my understanding of the problem. @ChristopherMayes may have further thoughts or be able to clarify further:

On line 1 of your input file is:

num_processes_y num_processes_z

The total number of processes used for the parallel computation will be num_processes_y * num_processes_z.

and on line 3 is:

num_gridx num_gridy num_gridz ...

where this is the total number of grid points in the x, y, and z directions, respectively.

Importantly, though, there is partitioning of the input in the directions y and z. The simulation is split in those directions, but not in x:

• x isn't partitioned
• y is partitioned among num_processes_y processes, requiring num_gridy >= num_processes_y
• z is partitioned among num_processes_z processes, requiring num_gridz >= num_processes_z

The error is indicating that you have more processors specified than really needed as there aren't enough grid points to span them fully, specifically in the z direction. It may work as-is but it's not optimally distributed.

In short, increase your num_gridz or reduce your num_processes_z in order to address the issue.

[Jinko12]

Thank you very much for your answer. I did do it in the way you said. Here is my code. The number of my meshes is 128128128. Even though I set num_processes to 2*2, an error still occurred

L=rho1bend1
input = IZ.ImpactZInput(
reference_kinetic_energy=energy0,
nx=128,
ny=128,
nz=128,
radius_x=0.001,
radius_y=0.001,
ncpu_y=2,
ncpu_z=2,
reference_particle_mass=mec2,
reference_particle_charge=-1.0,
average_current=P0.average_current,
reference_frequency=1300000000.0,
boundary_type="trans_open_longi_open",
lattice=[
IZ.ToggleSpaceCharge(enable=True,map_steps=10,),
IZ.WriteFull(name="initial_particles", file_id=100),
IZ.Dipole(
length=L,
steps=100,
map_steps=10,
angle=bend1,
k1=0.0,
input_switch=00.0, # input switch 150., if >200, include 1D CSR
hgap=0.001,
e1=bend1/2,
e2=bend1/2,
entrance_curvature=0.0,
exit_curvature=0.0,
fint=0.000,
misalignment_error_x=0.0,
misalignment_error_y=0.0,
rotation_error_x=0.0,
rotation_error_y=0.0,
rotation_error_z=0.0,
name="dipole1",
),
IZ.WriteFull(file_id=200, name="final_particles"),
],
)
P0.drift_to_z()
par=IZ.ImpactZParticles.from_particle_group(P0,reference_frequency=1.3e9,reference_kinetic_energy=energy0-mec2 )
I = ImpactZ(input,initial_particles=par)
I.nproc=input.ncpu_yinput.ncpu_z
I.run(verbose=False, progress_bar=False)

IMPACT-Z output was:

Read input data from file - ImpactZ.in:
!-----------------------------------------------------------
! IMPACT-Z: Integrated Map and PArticle Tracking Code: Version 2.7
! Copyright of The Regents of the University of California
!-----------------------------------------------------------
check random: 0 0.320923269
check random: 2 0.170972645
check random: 3 0.875587583
check random: 1 0.928635657
pass generating initial distribution...
pass setting up lattice...
enter elment (type code): 1 -14
zedge: 0.0000000000000000
enter elment (type code): 2 -2
zedge: 0.0000000000000000
enter elment (type code): 3 4
zedge: 0.0000000000000000
wrong in the z direction local mesh number! -2147483647 128 0 0
wrong in the z direction local mesh number! -2147483521 128 1 0
wrong in the z direction local mesh number! -2147483647 128 0 1
wrong in the z direction local mesh number! -2147483521 128 1 1
IMPACT-Z exited with an error: Command '['/.conda/envs/impactz/bin/mpirun', '-n', '4',

[ken-lauer]

These messages:

wrong in the z direction local mesh number! -2147483647 128 0 0

Relate to how the spatial range is split up with the grid/mesh size. I unfortunately don't fully understand this bit (it's very complicated and undocumented).

Normally, this message arises when:

• The local mesh size for a single processor is less than 0
• The local mesh size for a single processor is larger than nz

In this case, you can see that Mshlocal(3) which is the mesh size in Z for the specific processor is a garbage number -2147483647 - which indicates to me there's some logic bug in IMPACT-Z, or perhaps another bad parameter in your input file.

At this point, though, I think we need Ji's help. I'd recommend opening an issue on https://github.com/impact-lbl/IMPACT-Z and including the ImpactZ.in file for him to look over. If you don't get a response there, e-mail might be better.