Skip to content

[Feature] Improve VISReg loss numerical stability and defaults - #431

Open
HaiyuWu wants to merge 1 commit into
galilai-group:mainfrom
HaiyuWu:feature/visreg-loss-stability
Open

[Feature] Improve VISReg loss numerical stability and defaults#431
HaiyuWu wants to merge 1 commit into
galilai-group:mainfrom
HaiyuWu:feature/visreg-loss-stability

Conversation

@HaiyuWu

@HaiyuWu HaiyuWu commented Jul 25, 2026

Copy link
Copy Markdown
Contributor
  • VISRegLoss: clamp std to a minimum instead of adding 1e-6, so the scale term and normalisation stay unbiased for well-conditioned features while still guarding against division-by-zero on collapsed dimensions.
  • VISRegLoss: keep the Gaussian quantile target in fp32 rather than casting it to the input dtype, for a more stable shape term under 16-mixed.
  • VISReg: change the default convex weight lamb from 0.02 to 0.9.

Description

Checklist

  • [ x ] I have read the Contributing document.
  • [ x ] The documentation is up-to-date with the changes I made (check build artifacts).
  • [ x ] All tests passed, and additional code has been covered with new tests.
  • [ x ] I have added the PR to the RELEASES.rst file.

- VISRegLoss: clamp std to a minimum instead of adding 1e-6, so the scale
  term and normalisation stay unbiased for well-conditioned features while
  still guarding against division-by-zero on collapsed dimensions.
- VISRegLoss: keep the Gaussian quantile target in fp32 rather than casting
  it to the input dtype, for a more stable shape term under 16-mixed.
- VISReg: change the default convex weight lamb from 0.02 to 0.9.
@HaiyuWu
HaiyuWu force-pushed the feature/visreg-loss-stability branch from 40c4fb6 to 3657caa Compare July 25, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant