# Differential Equations on Probability Distributions

## Example Problems

### Solve the equation $$-z_i \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_i}\,q_{\phi}(\mathbf{z})$$

Let $$q_{\phi}(\mathbf{z}) = q_{\phi}(z_1|z_2,\dots,z_K) \cdot q_{\phi}(z_2,\dots,z_K)$$, where $$K$$ is the number of dimensions of $$\mathbf{z}$$. We shall first solve the differential equation on $$z_1$$:

\begin{align} & -z_1 \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_1}\,q_{\phi}(\mathbf{z}) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K)\cdot q_{\phi}(z_2,\dots,z_K) = q_{\phi}(z_2,\dots,z_K)\cdot \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -z_1 \cdot q_{\phi}(z_1|z_2,\dots,z_K) = \frac{\partial}{\partial z_1}\,q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -z_1\,\partial z_1 = \frac{1}{q_{\phi}(z_1|z_2,\dots,z_K)}\,\partial q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & \int -z_1\,\partial z_1 = \int \frac{1}{q_{\phi}(z_1|z_2,\dots,z_K)}\,\partial q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & -\frac{1}{2} z_1^2 + C(z_2,\dots,z_K) = \log q_{\phi}(z_1|z_2,\dots,z_K) \\ \implies & \exp\left(-\frac{1}{2} z_1^2\right)\cdot\exp\left(C(z_2,\dots,z_K))\right) = q_{\phi}(z_1|z_2,\dots,z_K) \end{align}

Since $$q_{\phi}(z_1|z_2,\dots,z_K)$$ is a probability distribution, we have:

$\begin{equation*} \exp\left(C(z_2,\dots,z_K)\right) = 1 / \int \exp\left(-\frac{1}{2} z_1^2\right) \,\mathrm{d}z_1 = \frac{1}{\sqrt{2\pi}} \end{equation*}$

thus we have:

$\begin{equation*} q_{\phi}(\mathbf{z}) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) \end{equation*}$

We then solve the differential equation on $$z_2$$. We have:

\begin{align} & -z_2 \cdot q_{\phi}(\mathbf{z}) = \frac{\partial}{\partial z_2}\,q_{\phi}(\mathbf{z}) \\ \implies & -z_2 \cdot \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_2} \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_2 \cdot \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot q_{\phi}(z_2,\dots,z_K) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2} z_1^2\right)\cdot\frac{\partial}{\partial z_2} q_{\phi}(z_2,\dots,z_K) \\ \implies & -z_2 \cdot q_{\phi}(z_2,\dots,z_K) = \frac{\partial}{\partial z_2} q_{\phi}(z_2,\dots,z_K) \end{align}

If we let $$\mathbf{z}’ = z_2,\dots,z_K$$, the form of the differential equation on $$z_2$$ is now exactly the same as the differential equation on $$z_1$$. Use the same method, we can solve the equation on $$z_2$$, and further on $$z_3, \dots, z_K$$. Finally ,we can get the solution:

$\begin{equation*} q_{\phi}(\mathbf{z}) = \frac{1}{\left(\sqrt{2\pi}\right)^K}\prod_{i=1}^K \exp\left(-\frac{1}{2} z_i^2\right) \end{equation*}$

which is a $$K$$-dimensional unit Gaussian.

# Calculus of Variations

## Notation

• $$C_n[a,b]​$$: the set of continuous functions defined on $$[a,b]​$$, which are continuous and have derivatives up to order $$n​$$. Specifically, $$C[a,b]​$$ is the set of continuous functions.
• $$D_n[a,b]​$$: the set of functions defined on $$[a,b]​$$ which are continuous and have continuous derivatives up to order $$n​$$.

## Fundamental lemma in the calculus of variations

If $$\alpha(x)$$ is continuous in $$[a,b]$$, and if $\int_a^b \alpha(x) h(x) = 0$ for every function $$h(x) \in \mathcal{l}(a,b)$$ such that $$h(a) = h(b) = 0$$, then $$\alpha(x)=0$$ for all $$x$$ in $$[a,b]$$.

## Euler's Equation

All the functionals in following cases should have boundary conditions specified.

### Single variate, single functional case

A necessary condition for $$J[y]$$:

$J[y] = \int_a^b F(x,y,y') dx$

to have an extremum for $$y(x)\in D_1[a,b]​$$ is:

$F_y - \frac{\partial}{\partial x} \frac{\partial F}{\partial (\partial_x y)} = 0$

where $$\partial_x y = \partial y / \partial x = y'$$.

### Singla variate, multi functional case

A necessary condition for $$J[y_1,\dots,y_n]​$$: $J[y_1,\dots,y_n] = \int_a^b F(x,y_1,\dots,y_n,y_1',\dots,y_n') \,dx$ to have an extremum is : $F_{y_i} - \frac{\partial}{\partial x} \frac{\partial F}{\partial(\partial_x y_i)} = 0 \qquad (i=1,\dots,n)$

### Multi variate, single functional case

A necessary condition for $$J[u]​$$:

$J[u] = \int\cdots\int_R F(x_1, \dots, x_n, u, \partial_{x_1} u, \dots, \partial_{x_n} u) \,dx_1 \cdots dx_n$

to have an extremum is:

$F_u - \sum_{i=1}^n \frac{\partial}{\partial x_i} \frac{\partial F}{\partial(\partial_{x_i} u)} = 0$

# Linear space vs functional space

Linear spaceFunctional space (on $$[a, b]$$)
Element$$\mathbf{x} = [x_1, x_2, \dots, x_n]$$$$f = [f(x_i), \dots]$$, where $$i \in I$$ is an uncountable index on $$[a,b]$$
Inner product$$\langle\mathbf{x},\mathbf{y}\rangle = [x_1 y_1, x_2 y_2, \dots, x_n y_n]$$$$\langle f,g\rangle = \int_a^b f(x) \,g(x)\, \mathrm{d}x$$
Orthogonal basis$$\mathbf{x} = \sum_{k=1}^n \alpha_k \mathbf{e}_k$$$$f=\sum_k \alpha_k f_k$$
$$\alpha_k = \langle \mathbf{x},\mathbf{e}_k \rangle$$$$\alpha_k = \langle f, f_k \rangle$$