Timeclass work branch for sorting out the mess by alhom · Pull Request #1271 · fmihpc/vlasiator

alhom · 2026-04-27T06:58:18Z

A draft pull request of the quite WIP timeclasses, for poring over and porting code in a controlled manner to a fresh PR (todo).

…on. Scaled up some comm, others down.

…-implemented translation flags, not bug-free yet (hangs after a while)

…ired for loger unidirectional searches (up to VLASOV_STENCIL_WIDTH+1 neighbours). This version crashes with a std::out_of_range on a map. Making the main Vlasov neighbourhood huge (pm6 in every direction) did not fix that...

…d dt, calculations and datareducers implemented.

…ing to fix small discrepancies in acceleration, but this might be floating-point precision issues with dt

… anymore to begin with

…in blocks adjustment

Gt devmerge

…rder.

…Zneighbors

…mented pencil threading with tc

…erformance for a long time. Rule of five additions; not sure if const copy constructors are actually good here

…erence issue, dt issue is still and it was not the uint cast

alhom

A bunch of notes from going through the timeclasses codebase, down to grid.cpp.

alhom · 2026-04-27T07:12:14Z

Mostly for diagnostics. Depends on having some new CellParams available.

alhom · 2026-04-27T07:12:39Z

+            continue;
+         }
+      }
+      if(P::systemWriteAllDROs || lowercase == "timeclass_rank" || lowercase == "vg_timeclass_rank") {


_rank things we can dispense with - these were an earlier development iteration that was surpassed by new halo definitions.

alhom · 2026-04-27T07:15:17Z

+	      continue;
+	 }
+      }
+      if(P::diagnosticWriteAllDROs || lowercase == "perturbedvolb" || lowercase == "vg_b_perturbed_vol") {


Unnecessary

alhom · 2026-04-27T07:15:37Z

            continue;
         }
      }
+      if(P::diagnosticWriteAllDROs || lowercase == "vg_p" || lowercase == "p") {


unnecessary

alhom · 2026-04-27T07:16:29Z

Adding diagnostics for timeghost block counting (re: memory footprint)

alhom · 2026-04-27T09:53:17Z

+//          "\n";        
+         // for(auto tc:timeclasses_handled){
+         for (int tc = 0; tc <= P::maxTimeclass; tc++){ // Filter to necessary tcs - adjustVelocityBlocks comms need COMM-WORLD yet
+            adjustVelocityBlocks(mpiGrid, cells, true, popID, tc);


to be squinted at wrt. hashmap troubles

alhom · 2026-04-27T09:55:03Z

+      TIME_V,           /*!< Current time of the cell due to timeclass timestepping*/
+      TIMESTEP_V,           /*!< Current timestep of the cell due to timeclass timestepping - obs float precision*/
+      TIMESTEP_FRACTIONAL_V,           /*!< Current timestep of the cell due to timeclass timestepping*/
+      TIMECLASS_RANK,           /*!< timeclass of this rank*/


_RANK not needed

alhom · 2026-04-27T09:55:43Z

      VLASOV_SOLVER_Z_GHOST, /*!< up to third(PPM+ghost) neighbor in z face directions */
      VLASOV_SOLVER_GHOST, /*!< all required neighbors for ghost translation */
      VLASOV_SOLVER_GHOST_REQNEIGH, /*!< all ghost translation neighbors which require own neighbor information */
+      VLASOV_SOLVER_TIMEGHOST_EXACT_HALO,


"EXACT", or those ones that have full translation even in ghost regions.

alhom · 2026-04-27T09:56:40Z

      VLASOV_SOLVER_GHOST, /*!< all required neighbors for ghost translation */
      VLASOV_SOLVER_GHOST_REQNEIGH, /*!< all ghost translation neighbors which require own neighbor information */
+      VLASOV_SOLVER_TIMEGHOST_EXACT_HALO,
+      VLASOV_SOLVER_TIMEGHOST_OUTER_HALO,


Outer halo are only translation sources, copied over and accelerated to correct time.

alhom · 2026-04-27T09:58:29Z

      for (size_t popID=0; popID<getObjectWrapper().particleSpecies.size(); ++popID) {
-         // Set active population
-         SpatialCell::setCommunicatedSpecies(popID);
+         for (int timeclass = 0; timeclass <= P::currentMaxTimeclass; ++timeclass) {


This is now communicating everything, see where and how to select only for modified timeclasses.

alhom · 2026-04-27T10:18:06Z

   mpiGrid.set_initial_length(grid_length)
      .set_load_balancing_method(&P::loadBalanceAlgorithm[0])
-      .set_neighborhood_length(neighborhood_size)
+      .set_neighborhood_length(neighborhood_size+4)


This to be checked

alhom · 2026-04-27T11:24:28Z

+      // if (P::propagateVlasovAcceleration)
+      // When using the FS-SPLIT functionality, Jaro Hokkanen reported issues with using the regular
+      // CellParams::LBWEIGHTCOUNTER, so use of blockscounts + 1 might be required.
+      mpiGrid.set_cell_weight(cells[i], (Real)1 + mpiGrid[cells[i]]->parameters[CellParams::LBWEIGHTCOUNTER] * ((int)pow(P::timeclassLBmantissa, mpiGrid[cells[i]]->parameters[CellParams::TIMECLASS])));


ad-hoc tuning parameter for LB here

alhom · 2026-04-27T11:27:24Z

      const vector<CellID>& localCells = getLocalCells();
-      prepareGhostTranslationCellLists(mpiGrid,localCells);
+
+      prepareGhostTranslationCellLists(mpiGrid, localCells, ghostTranslate_source, ghostTranslate_active);


NB here as well ghostTranslate vs timeghost below

alhom · 2026-04-27T11:27:57Z

+      const vector<CellID>& localCells = getLocalCells();
+      std::cerr << __FILE__<<":" << __LINE__ <<"\n";
+      const vector<CellID> remote_cells = mpiGrid.get_remote_cells_on_process_boundary(Neighborhoods::FULL);
+      mpiGrid.really_force_update_cell_neighborhoods(remote_cells);


Likely unnecessary

alhom · 2026-04-27T11:29:54Z

+      }
+   }
+
+   if (timeclass == P::currentMaxTimeclass){


just diagnostics

alhom · 2026-04-27T11:35:40Z

+      phiprof::Timer timeclassOuter {"Stencils init, timeclass, outer"};
+      std::vector<neigh_t> neighborhood_outer;
+
+      for(auto n : neighborhood){


This is a horrible loop and should just loop over timeclassOuterHaloExtent+timeclassExactHaloExtent.

alhom · 2026-04-27T11:41:12Z

+
+   // vmesh::VelocityBlockContainer<vmesh::LocalID> emptyvmesh& = vmesh::VelocityBlockContainer<vmesh::LocalID>();
+   // Write velocity block IDs for all timeghosts
+   for(int timeclass = 0; timeclass <= P::currentMaxTimeclass; timeclass++){


This loop writes VDFs to population_timeclass for diagnostics.

To check proper restart handling.

alhom · 2026-04-27T11:42:44Z

Needs ghost VDF support for readGrid

alhom · 2026-04-27T11:45:24Z

Trivial merges

alhom · 2026-04-27T11:48:53Z

+   //    }
+
+   // }
+   if(P::tc_test_type == 1){


The test types here have some archeological layering.

markusbattarbee and others added 30 commits November 4, 2022 20:42

Re-activated translation of blocks of target cells in non-local versi…

d7c537d

…on. Scaled up some comm, others down.

more communication after all, trying GCC11 on Carrington

cf56b1c

Merge branch 'dev' into AMR_translation_optimizations_dev

b535bd2

Merge branch 'dev' into AMR_translation_optimizations_dev

481ca3a

created new (huge, asymmetric) neighborhood for local translation, re…

b437ce6

…-implemented translation flags, not bug-free yet (hangs after a while)

initial timeclasses commit. CellParams for rank and cell timeclass an…

af2c7c9

…d dt, calculations and datareducers implemented.

Some LB awareness

f7b3cb1

a todo

eda974a

Fiddling with the dt and TC calculation for better desing

0cae83e

Timeclasses calculation done

579fd3c

Tooling and musings.

642d345

A note

4e26506

Prototyping the async time.

92b879b

Outputs, determination of tcs, but hangs.

36ddf5f

Tics now good, dt2 interp on, on to VDF time-ghosts

4b72210

Translation enabled, Flowthrough testing

ecc1123

Looking at the moment time interpolation properly. Doesnt look proper.

7892ab2

No need to poke these interpolations, actually

1f07bdd

Add rescaleDensity to Flowthrough to stabilize testing.

06be535

Properly integrating dynamic dt

9e81335

Missing cell_dts in acc subcycles, trying if forced maxvdt does anyth…

8a67b32

…ing to fix small discrepancies in acceleration, but this might be floating-point precision issues with dt

Disabling some scrap code on time ghosts

ce9c181

... now actually disabled

cb63c19

Config parameters for tc testing, cleaning up old crud

5bcf923

Cleaning out mixed tool changes

90b610a

Amending commit

2d41759

Dt debug, cleaned up some leftover code that shouldnt have been there…

d8e9b58

… anymore to begin with

Disable acc walls check if not not accelerating

536dca7

Proper guard against not running zero-dt accelerations instead of pla…

473d132

…in blocks adjustment

veetihaaja and others added 28 commits June 24, 2025 10:04

Merge pull request markusbattarbee#24 from alhom/GTDevmerge

73c28b5

Gt devmerge

fixed small cast problem

2e5ec78

added guard to check if tc, then tgt must be on

2578038

added guard to check if amr is on in tc runs

9d91ead

changed tc test type 4 to scale automatically

db94f0c

added new tc test type

4dd9742

changed timeclass halo default params

b9bc01b

changed tc halo default params

b4c1103

Initial commit of MPI-enabling timeclasses

15d0835

fixed alfven normalization bug

48c3c7e

added possibility to add constant radian to alfven angle

9c78b08

added alfvencascadeproject for tc testing

bb000b5

added moment prints for before field propagation

8f71fee

added moment printing, also fixed offdiagonal moment being in wrong o…

7529a26

…rder.

Runs 1D, missed flows from fine to coarse tc, crashes for 2D getting …

a41f5be

…Zneighbors

Revert definitions.h for the dev version of enum Neighborhoods

76425c7

Forceful debugging to get 2D tc-mpi working, to no avail

a9df6c6

Merge branch 'dev' into GTDevmerge, 21.7.2025 fresh dev rebase, imple…

c80e917

…mented pencil threading with tc

added new tc test case, with timeclass levels inside each other

20d656a

MPI segfault fixed for now!

7d3a181

Found a strange uint-int cast issue that has been bugging TC branch p…

fc7d929

…erformance for a long time. Rule of five additions; not sure if const copy constructors are actually good here

Found a strange uint-int cast issue that has been bugging TC branch p…

be3258b

…erformance for a long time. Rule of five additions; not sure if const copy constructors are actually good here

Sanity-checking copy and move constructors, fixed an out of scope ref…

58e8141

…erence issue, dt issue is still and it was not the uint cast

small comment addition

4282c0a

Merge remote-tracking branch 'alhom/timeclass-mpi' into timeclass-mpi

00b5200

deleted const from copy constructor

62541fd

Fix timeclass VDF outputs

8ce6107

Fresh freezer make-file!

facea7c

alhom commented Apr 27, 2026

View reviewed changes

Uh oh!

Conversation

alhom commented Apr 27, 2026

Uh oh!

alhom left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants