Differential Equations on Probability Distributions

Example Problems

Solve the equation \(-z_i \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_i}\,q_{\phi}(\mathbf{z})\)

Let \(q_{\phi}(\mathbf{z}) = q_{\phi}(z_1|z_2,\dots,z_K) \cdot q_{\phi}(z_2,\dots,z_K)\), where \(K\) is the number of dimensions of \(\mathbf{z}\). We shall first solve the differential equation on \(z_1\):

\[ \begin{align} & -z_1 \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_1}\,q_{\phi}(\mathbf{z}) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) = q_{\phi}(z_2,\dots,z_K)\cdot \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K) = \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -z_1\,\partial z_1 = \frac{1}{q_{\phi}(z_1|z_2,\dots,z_K)}\,\partial q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & \int -z_1\,\partial z_1 = \int \frac{1}{q_{\phi}(z_1|z_2,\dots,z_K)}\,\partial q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -\frac{1}{2} z_1^2 + C(z_2,\dots,z_K) = \log q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & \exp\left(-\frac{1}{2} z_1^2\right)\cdot\exp\left(C(z_2,\dots,z_K))\right) = q_{\phi}(z_1|z_2,\dots,z_K) \end{align} \]

Since \(q_{\phi}(z_1|z_2,\dots,z_K)\) is a probability distribution, we have:

\[ \begin{equation*} \exp\left(C(z_2,\dots,z_K)\right) = 1 / \int \exp\left(-\frac{1}{2} z_1^2\right) \,\mathrm{d}z_1 = \frac{1}{\sqrt{2\pi}} \end{equation*} \]

thus we have:

\[ \begin{equation*} q_{\phi}(\mathbf{z}) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) \end{equation*} \]

We then solve the differential equation on \(z_2\). We have:

\[ \begin{align} & -z_2 \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_2}\,q_{\phi}(\mathbf{z}) \\ \implies & -z_2 \cdot \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_2} \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_2 \cdot \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot\frac{\partial}{\partial z_2} q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_2 \cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_2} q_{\phi}(z_2,\dots,z_K) \end{align} \]

If we let \(\mathbf{z}’ = z_2,\dots,z_K\), the form of the differential equation on \(z_2\) is now exactly the same as the differential equation on \(z_1\). Use the same method, we can solve the equation on \(z_2\), and further on \(z_3, \dots, z_K\). Finally ,we can get the solution:

\[ \begin{equation*} q_{\phi}(\mathbf{z}) = \frac{1}{\left(\sqrt{2\pi}\right)^K}\prod_{i=1}^K \exp\left(-\frac{1}{2} z_i^2\right) \end{equation*} \]

which is a \(K\)-dimensional unit Gaussian.

Calculus of Variations

Notation

  • \(C_n[a,b]​\): the set of continuous functions defined on \([a,b]​\), which are continuous and have derivatives up to order \(n​\). Specifically, \(C[a,b]​\) is the set of continuous functions.
  • \(D_n[a,b]​\): the set of functions defined on \([a,b]​\) which are continuous and have continuous derivatives up to order \(n​\).

Fundamental lemma in the calculus of variations

If \(\alpha(x)\) is continuous in \([a,b]\), and if \[ \int_a^b \alpha(x) h(x) = 0 \] for every function \(h(x) \in \mathcal{l}(a,b)\) such that \(h(a) = h(b) = 0\), then \(\alpha(x)=0\) for all \(x\) in \([a,b]\).

Euler's Equation

All the functionals in following cases should have boundary conditions specified.

Single variate, single functional case

A necessary condition for \(J[y]\):

\[ J[y] = \int_a^b F(x,y,y') dx \]

to have an extremum for \(y(x)\in D_1[a,b]​\) is:

\[ F_y - \frac{\partial}{\partial x} \frac{\partial F}{\partial (\partial_x y)} = 0 \]

where \(\partial_x y = \partial y / \partial x = y'\).

Singla variate, multi functional case

A necessary condition for \(J[y_1,\dots,y_n]​\): \[ J[y_1,\dots,y_n] = \int_a^b F(x,y_1,\dots,y_n,y_1',\dots,y_n') \,dx \] to have an extremum is : \[ F_{y_i} - \frac{\partial}{\partial x} \frac{\partial F}{\partial(\partial_x y_i)} = 0 \qquad (i=1,\dots,n) \]

Multi variate, single functional case

A necessary condition for \(J[u]​\):

\[ J[u] = \int\cdots\int_R F(x_1, \dots, x_n, u, \partial_{x_1} u, \dots, \partial_{x_n} u) \,dx_1 \cdots dx_n \]

to have an extremum is:

\[ F_u - \sum_{i=1}^n \frac{\partial}{\partial x_i} \frac{\partial F}{\partial(\partial_{x_i} u)} = 0 \]

Linear space vs functional space

Linear spaceFunctional space (on \([a, b]\))
Element\(\mathbf{x} = [x_1, x_2, \dots, x_n]\)\(f = [f(x_i), \dots]\), where \(i \in I\) is an uncountable index on \([a,b]\)
Inner product\(\langle\mathbf{x},\mathbf{y}\rangle = [x_1 y_1, x_2 y_2, \dots, x_n y_n]\)\(\langle f,g\rangle = \int_a^b f(x) \,g(x)\, \mathrm{d}x\)
Orthogonal basis\(\mathbf{x} = \sum_{k=1}^n \alpha_k \mathbf{e}_k\)\(f=\sum_k \alpha_k f_k\)
\(\alpha_k = \langle \mathbf{x},\mathbf{e}_k \rangle\)\(\alpha_k = \langle f, f_k \rangle\)