12
$\begingroup$

In the forward diffusion process described by Ho, et al. the probability distribution for the next step is:

$$q(\mathbf{x}_t|\mathbf{x}_{t-1}) = N(\mathbf{x}_t;\sqrt{1-\beta_t}\mathbf{x}_{t-1},\beta_t\mathbf{I})$$

What is the purpose of scaling the mean of the distribution by $\sqrt{1 - \beta_t}$? I think this has something to do with maintaining the same variance across steps, but don't understand why keeping variance fixed would be helpful.

Ho, J., Jain, A., Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. Proceedings of the 34th Conference on Neural Information Processing Systems. NeurIPS 2020, Vancouver, CA.

$\endgroup$

1 Answer 1

17
$\begingroup$

The scaling factor is needed just to avoid making the variance "explode".

Intuition. In a forward diffusion process you start with some vector $\mathbf{x}$ and add a bit of noise at each step $t$. If you do not scale $\mathbf{x}$ before adding the noise, all the noise will add up to increase the overall variance. This is undesired, because the range of magnitude of the vectors you get at the end will depend on the number $T$ of steps you perform.

If you start with just a number $x_0 = 0$, after $T$ steps you will have a value $x_T \in [-T, T]$.

Proof. To solve that, at step $t$, we need to "scale down a bit" $\mathbf{x_{t-1}}$ before adding the noise.

For simplicity, consider $x_0 \sim \mathcal{N}(\mu, 1)$, so we can perform exact computations (note that the input is normalized to unit variance). We would like to keep the variance at $1$ at every step $t$.

So, for the first step $t=1$, we scale $x_0$ by a factor $a$, and then add $\epsilon_1 \sim \mathcal{N}(0, \beta_1)$:

$$ x_1 = a\, x_0 + \sqrt{\beta_1} \epsilon_1 $$

The variance of $x_1$ is given by:

$$ \mathrm{Var}(x_1) = a^2 + \beta_1 $$

If we force $ \mathrm{Var}(x_1) = 1 $ we can solve for the required scaling factor $a$ to keep the variance unchanged:

$$ a^2 + \beta_1 = 1 \Rightarrow a = \sqrt{1 - \beta_1} $$

This of course works at every step $t$, not just the first one. So we get:

$$ \mathbf{x_t} = \sqrt{1-\beta_t} \mathbf{x_{t-1}} + \sqrt{\beta_t} \mathbf{\epsilon_{t-1}} $$

which corresponds to the probability distribution you started from:

$$ q(\mathbf{x_t} | \mathbf{x_{t-1}} ) = \mathcal{N}(\mathbf{x_t}; \sqrt{1-\beta_t} \mathbf{x_{t-1}}, \beta_t \mathbb{I} ) $$

$\endgroup$
3
  • $\begingroup$ From the way the variance is computed $Var(x_1) = a^2 +\beta_1$, the Gaussian $\epsilon_1$ and $x_0$ should have covariance zero. Why is that the case? $\endgroup$
    – user25004
    Commented Feb 7, 2023 at 21:34
  • $\begingroup$ @user25004 Exactly. This is because $\epsilon_1$ is independent of $x_0$, which implies that their covariance is zero. $\endgroup$
    – Einlar
    Commented Apr 18, 2023 at 17:18
  • $\begingroup$ > If you start with just a number $x_0 = 0$, after $T$ steps you will have a value $x_T \in [-T, T]$, Why is this true? I thought the normal distribution has a range from $-\infty$ to $\infty$ $\endgroup$ Commented Nov 10, 2023 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.