# Visualizing High Dimensional Space

## Orthogonal Projection

Izmailov et al. (2018) proposed to project weight tensors $$w$$ into 2d plane (i.e., obtain the projector $$w(x,y)$$), by constructing the orthogonal basis $$(\hat{u},\hat{v})$$ from three given weight tensors $$(w_1, w_2, w_3)$$: \begin{align} u &= w_2 - w_1 \\ v &= (w_3 - w_1) - \frac{\left<w_3-w_1, w_2-w_1\right>}{\left\| w_2-w_1 \right\|^2} \cdot (w_2 - w_1) \\ \hat{u} &= \frac{u}{\left\| u \right\|} \\ \hat{v} &= \frac{v}{\left\| v \right\|} \\ w(x,y) &= w_1 + x\cdot \hat{u} + y \cdot \hat{v} \end{align}

# References

Izmailov, Pavel, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. “Averaging Weights Leads to Wider Optima and Better Generalization.” arXiv Preprint arXiv:1803.05407.