Differential Equations on Probability Distributions

Example Problems

Solve the equation \(-z_i \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_i}\,q_{\phi}(\mathbf{z})\)

Let \(q_{\phi}(\mathbf{z}) = q_{\phi}(z_1|z_2,\dots,z_K) \cdot q_{\phi}(z_2,\dots,z_K)\), where \(K\) is the number of dimensions of \(\mathbf{z}\). We shall first solve the differential equation on \(z_1\):

\[ \begin{align} & -z_1 \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_1}\,q_{\phi}(\mathbf{z}) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) = q_{\phi}(z_2,\dots,z_K)\cdot \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K) = \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -z_1\,\partial z_1 = \frac{1}{q_{\phi}(z_1|z_2,\dots,z_K)}\,\partial q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & \int -z_1\,\partial z_1 = \int \frac{1}{q_{\phi}(z_1|z_2,\dots,z_K)}\,\partial q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -\frac{1}{2} z_1^2 + C(z_2,\dots,z_K) = \log q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & \exp\left(-\frac{1}{2} z_1^2\right)\cdot\exp\left(C(z_2,\dots,z_K))\right) = q_{\phi}(z_1|z_2,\dots,z_K) \end{align} \]

Since \(q_{\phi}(z_1|z_2,\dots,z_K)\) is a probability distribution, we have:

\[ \begin{equation*} \exp\left(C(z_2,\dots,z_K)\right) = 1 / \int \exp\left(-\frac{1}{2} z_1^2\right) \,\mathrm{d}z_1 = \frac{1}{\sqrt{2\pi}} \end{equation*} \]

thus we have:

\[ \begin{equation*} q_{\phi}(\mathbf{z}) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) \end{equation*} \]

We then solve the differential equation on \(z_2\). We have:

\[ \begin{align} & -z_2 \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_2}\,q_{\phi}(\mathbf{z}) \\ \implies & -z_2 \cdot \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_2} \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_2 \cdot \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot\frac{\partial}{\partial z_2} q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_2 \cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_2} q_{\phi}(z_2,\dots,z_K) \end{align} \]

If we let \(\mathbf{z}’ = z_2,\dots,z_K\), the form of the differential equation on \(z_2\) is now exactly the same as the differential equation on \(z_1\). Use the same method, we can solve the equation on \(z_2\), and further on \(z_3, \dots, z_K\). Finally ,we can get the solution:

\[ \begin{equation*} q_{\phi}(\mathbf{z}) = \frac{1}{\left(\sqrt{2\pi}\right)^K}\prod_{i=1}^K \exp\left(-\frac{1}{2} z_i^2\right) \end{equation*} \]

which is a \(K\)-dimensional unit Gaussian.