Replica propagation times systematically change in cyclic patterns that may slow simulations

[jchodera]

In plotting the total replica propagation time per iteration from one of @hannahbrucemacdonald's experiments (/data/chodera/brucemah/relative_paper/dec/jnk1/lig20to8/complex_0.stderr), it appears that there is a pattern:

Connecting successive iterations with lines reveals a clearer pattern:

Zooming in shows the pattern more clearly:

This is currently just something intriguing, but something we should investigate in the new year since it may lead to speed improvements if we can understand what is going on here.

[jchodera]

@hannahbrucemacdonald : If you have a chance before I do, here’s what I was doing to look into the variable timing issue.
I inserted these lines into my in-place installation of openmmtools. You can find where this is installed with

python -c "from openmmtools import multistate; print(multistate.__file__)"

I edited multistatesampler.py and inserted the following at the end of _propagate_replica
at line 1235:
https://github.com/choderalab/openmmtools/blob/master/openmmtools/multistate/multistatesampler.py#L1235
Here are the lines:

 # DEBUG: Report nvidia-smi                                                                                                                                                                                                                                                                                                                                          
 import subprocess
 process = subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE, universal_newlines=True)
 logger.debug(process.stdout)

This should write the output of nvidia-smi each replica propagation step to report current temperature, voltage, and speed:

Tue Jan 21 12:19:31 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.39       Driver Version: 418.39       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    On   | 00000000:02:00.0 Off |                  N/A |
|  0%   31C    P8     5W / 180W |      1MiB /  8119MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

The relevant numbers are
• the drive rversion
• the GPU type
• the fan %
• the temperature
• the power usage
• the process table

[hannahbrucemacdonald]

Did you only see this in complex, or solvent and vacuum too?

[hannahbrucemacdonald]

Hey - I've got so far as to run a short bit of complex with Thrombin and here are the minimal plots. Looks like we have oscillations not correlated to the GPU temp. It looks like it's stabilising around 80C which should be fine. I will have a go with the other parameters that you've suggested too

System: Thrombin
GPU type: NVIDIA-SMI 418.39
Driver version: 418.39

• the driver version

• the GPU type

• the fan %

• the temperature

• the power usage

• the process table

(edited to fix x-axis labels)

[hannahbrucemacdonald]

Fan speed is quite variable, but also doesn't seem to correlate

[jchodera]

Are you plotting this every iteration or every replica propagation time? I was hoping to get more fine grained information by printing the output of nvidia-smi every replica propagation substep

[hannahbrucemacdonald]

Ahhhh I see what you mean, I have that I think - give me a min.

[hannahbrucemacdonald]

Ok so I have the time of each replica iteration from the nvidia-smi output, but it's in seconds, while each replica propogation is ~1 second. I can't see anything useful about each replica taking longer or not because of the rounding

Unless I'm getting the wrong end of the stick.

I can increase the amount of MD (~factor 10?) per step to push this up? We currently do... 250*4 fs per replica per iteration (1 picosecond).

[jchodera]

Did you only see this in complex, or solvent and vacuum too?

@hannahbrucemacdonald : Great question! I used this jupyter notebook to generate this plot for all phases from /data/chodera/brucemah/relative_paper/dec/jnk1/lig20to8/---just change the paths in the notebook to plot a different calculation!

GitHub seems to have trouble with image uploads right now, but check out the notebook:

• cyclic behavior is only observed with complex
• the iteration time for solvent keeps growing with iteration number---is this a NetCDF issue?
• the vacuum time keeps growing, but also switches back and forth between fast and slow in long time blocks (!!!); are we using the CPU or Reference platform for vacuum?

[jaimergp]

That temperature plot looks suspicious to me. Temp throttling can definitely happen at intervals.

Some data pulled from this article:

Ideally you want temperatures to be as low as possible, anything below 80 degrees is normal and should keep throttling in check. Nvidia's GTX 1080 Ti, for example, has a throttling point of 84 degrees. If you keep the temperature below 80 degrees you leave yourself with a bit of breathing room, so you can focus on having fun instead of monitoring GPU frequencies.

It's important to remember that every graphics card has a different throttling point. The previous-gen GTX 980 and 970, for example, throttle at 80 degrees, while AMD's Vega series cards can reach a maximum temperature of 85 degrees before they throttle. You will need to find out the throttling point for your specific card in order to set an effective fan curve and voltage.

So we might be hitting 80ºC, decreasing performance to "calm down" (but keeping the temperature just a bit below 80ºC!), then we think we can use more power and hit 80ºC, chill down, etc.

[jaimergp]

Power usage might give more details.

[hannahbrucemacdonald]

Sorry got a little further --- but not much - I can expand get_details() to wrangle in more details

biggest addition is looking at /data/chodera/brucemah/relative_paper/amber_starting/off100/thrombin/test/lig1to10/

h_edits_to_ipynb.zip

timestep is still too small to really see the finetooth timing... I'll run something slower in the same directory - hopefully it will be there for european morning time!