Loss Surface and Generalization

Stochastic Weight Averaging

Izmailov et al. (2018) suggested

Figure 1: Stochastic Weighted Averaging Illustration (view pdf)

The visualization method of the above figure can be found at Orthogonal Projection.

References

Izmailov, Pavel, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. “Averaging Weights Leads to Wider Optima and Better Generalization.” arXiv Preprint arXiv:1803.05407.