save_last to save last checkpoint

[felix-e-h-p]

Related to #542

• save_last: True was meant to save the model from the most recent epoch to last.ckpt.
• L 2.6.0 last.ckpt is written when a top-k checkpoint is saved at the same step.
• Used save_top_k: 1, so only happens when validation loss improves, and last.ckpt ends up being a copy of the best ckpt.
• Before 2.6, save_last: True did save real last epoch.

Changes:

• configs.example/callbacks/default.yaml: turned off save_last. added second ModelCheckpoint with last_checkpoint. Hence saves after every validation and overwrites last.ckpt each time.
• pvnet/training/train.py: Both callbacks now to be saved to the same run folder. The config files are still saved once, as before.
• tests/training/test_train.py: Updated fixture accordingly

[felix-e-h-p]

Should be marked as relevant issue in summation model also right?

[Sukh-P]

@felix-e-h-p have you tested this already on a dummy model run? If so could you please link that wandb run and detail any checking you did to check this works as expected? Obviously no test coverage on this so would be good to know it's already been tried and works as expected

[felix-e-h-p]

@felix-e-h-p have you tested this already on a dummy model run? If so could you please link that wandb run and detail any checking you did to check this works as expected? Obviously no test coverage on this so would be good to know it's already been tried and works as expected

Yeah will do so later today

In here - https://wandb.ai/openclimatefix/pvnet_uk_test/runs/

Saving locally then gonna read step count and early stopping counter - have limited at low train and val batches so shouldn't be too long to check it

[felix-e-h-p]

All good. 50 epochs with 100 train sample limit, best is 49, I loaded best and last just for a direct check: last step 5000 | best step 4900