Visualizing High Dimensional Space

Orthogonal Projection

Izmailov et al. (2018) proposed to project weight tensors \(w\) into 2d plane (i.e., obtain the projector \(w(x,y)\)), by constructing the orthogonal basis \((\hat{u},\hat{v})\) from three given weight tensors \((w_1, w_2, w_3)\): \[ \begin{align} u &= w_2 - w_1 \\ v &= (w_3 - w_1) - \frac{\left<w_3-w_1, w_2-w_1\right>}{\left\| w_2-w_1 \right\|^2} \cdot (w_2 - w_1) \\ \hat{u} &= \frac{u}{\left\| u \right\|} \\ \hat{v} &= \frac{v}{\left\| v \right\|} \\ w(x,y) &= w_1 + x\cdot \hat{u} + y \cdot \hat{v} \end{align} \]

References

Izmailov, Pavel, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. “Averaging Weights Leads to Wider Optima and Better Generalization.” arXiv Preprint arXiv:1803.05407.