The forward diffusion process, which goes from $x_t$ to $x_{t+1} $ is Gaussian, which is very reasonable as we go the next state by adding random gaussian noise. However, I do not understand why the reverse of the process ( which goes from $x_{t+1} $ to $x_t $) is not also a Gaussian.
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
You can start off with $X_0$ having any distribution you like: Gaussian or Bernoulli or the distribution over 255x255 colour photographs of poodles, or whatever.
Since you're adding Gaussian noise to it, the conditional distributions $X_t|X_{t-1}$ are just Gaussian -- if you know $X_{t-1}$, then $X_t$ is just that plus Gaussian noise.
The reverse distributions are trying to estimate $X_0|X_t$ (in little steps). This can't be Gaussian, because $X_0$ can be any distribution you like. In the special case when $X_0$ is Gaussian, the reverse distributions are also Gaussian, but if $X_0$ is the distribution over 255x255 colour photographs of poodles it won't be.
For a very simple case, suppose $X_0$ is Bernoulli(0.5). $X_t$ for small $t$ looks like a two-part Gaussian mixture, and $X_t|X_0$ is adding Gaussian noise. The reverse conditional $X_0|X_t$ maps a Gaussian mixture into a binary distribution, which can't be Gaussian.
answered Oct 11 at 7:18
$\begingroup$
$\endgroup$
The forward process is a blurring process and a stochastic random process. The output is noisy.
The reverse process is a de-noising process and a deterministic process. The output is not noisy.
answered Oct 11 at 7:51
$\begingroup$
$\endgroup$
The exact reverse process is intractable since it requires computations involving the data distribution. Therefore, it is approximated with a parameterized model, such as a neural network. If the diffusion step sizes are small enough, the reverse process is also Gaussian.
