Hello,
Thanks for making this wonderful work available.
I have a question regarding the interaction between weight_decay and ortho_fraction in Dion3. weight_decay is applied to the whole weight matrix whereas scatter_add_ is applied on the top_k. configs/dion3_160m.yaml and configs/normuon_160m.yaml kept weight_decay constant, but I want to understand whether there's a need to scale weight_decay for consistency between Dion3 with ortho_fraction<1 and NorMuon. Would the weight_decay need to be smaller when using a small ortho_fraction since rows are decayed more often than they are selected?
I intended to do hyperparameter tuning, but the recommendation is to use a larger ortho_fraction for smaller model, and a larger one for large models. Would the optimal hyperparameters shift with the change in ortho_fraction and what is your recommendation?
Hello,
Thanks for making this wonderful work available.
I have a question regarding the interaction between
weight_decayandortho_fractionin Dion3.weight_decayis applied to the whole weight matrix whereasscatter_add_is applied on the top_k.configs/dion3_160m.yamlandconfigs/normuon_160m.yamlkeptweight_decayconstant, but I want to understand whether there's a need to scaleweight_decayfor consistency between Dion3 with ortho_fraction<1 and NorMuon. Would theweight_decayneed to be smaller when using a smallortho_fractionsince rows are decayed more often than they are selected?I intended to do hyperparameter tuning, but the recommendation is to use a larger
ortho_fractionfor smaller model, and a larger one for large models. Would the optimal hyperparameters shift with the change inortho_fractionand what is your recommendation?