Diffusion Models - modeling noise? In this post about diffusion models, IIUC, we want to use a neural network to approximate the mean of the reverse diffusion: $p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t))$. That is, we train a neural network to predict the quantity $\tilde{\boldsymbol{\mu}}_t = \frac{1}{\sqrt{\alpha_t}} \Big( \mathbf{x}_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_t \Big)$.
Instead, according to the article, since $\mathbf{x}_t$ is available at training time, we predict $\boldsymbol{\epsilon}_t$ instead.
But according to the forward diffusion process, we have:
$
\begin{aligned}
\mathbf{x}_t 
&= \sqrt{\alpha_t}\mathbf{x}_{t-1} + \sqrt{1 - \alpha_t}\boldsymbol{\epsilon}_{t-1} & \text{ ;where } \boldsymbol{\epsilon}_{t-1}, \boldsymbol{\epsilon}_{t-2}, \dots \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) \\
&= \sqrt{\alpha_t \alpha_{t-1}} \mathbf{x}_{t-2} + \sqrt{1 - \alpha_t \alpha_{t-1}} \bar{\boldsymbol{\epsilon}}_{t-2} & \text{ ;where } \bar{\boldsymbol{\epsilon}}_{t-2} \text{ merges two Gaussians (*).} \\
&= \dots \\
&= \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\boldsymbol{\epsilon} \\
q(\mathbf{x}_t \vert \mathbf{x}_0) &= \mathcal{N}(\mathbf{x}_t; \sqrt{\bar{\alpha}_t} \mathbf{x}_0, (1 - \bar{\alpha}_t)\mathbf{I})
\end{aligned}
$
So $\boldsymbol{\epsilon}_t$ is a random sample from the standard normal. I don't fully understand this; why are we trying to train a neural network to predict on the noise? If this is a random draw from a normal that is independent of any covariates, isn't this also available at training time, and so we can just directly compute a sample?
 A: TLDR; the noise $\boldsymbol{\epsilon}_\theta$ is the only thing we need learn so we can model reverse diffusion process. We can use $\boldsymbol{\epsilon}_\theta$ to get mean and variance of the Gaussian distributions we are looking for!
I also had the exact same question when I encountered the DDPM paper! I think we need to clarify a few things to understand the purpose of predicting the noise!
1- Why are we learning the noise?

*

*So in diffusion models (including DDPM), given the forward diffusion (i.e., $q(x_t | x_{t-1})$), our goal is to learn how to reverse that process (i.e., $p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)$) with a neural network.

*It turns out after doing lots of math, $p_\theta$ is Guassian in the form of $p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t))$. Hence, all we need to do is to find $\mathbf{\mu_\theta}$ and $\mathbf{\Sigma_\theta}$

*Now, we can do the math and see:
$$\begin{aligned}
\boldsymbol{\mu}_\theta(\mathbf{x}_t, t) &= \color{black}{\frac{1}{\sqrt{\alpha_t}} \Big( \mathbf{x}_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t) \Big)}
\end{aligned}$$
where $\mathbf{\alpha}_t$ depends on noise values and non-trainable. So the only part we actually need to learn, is $\boldsymbol{\epsilon}_\theta$.

*Once we learn $\boldsymbol{\epsilon}_\theta$, we can put it in the equation and get $\mathbf{p}_\theta$

2- If the noise is independent of data, why/how can we learn it?

*

*Well, I think given the algorithm 1 below, all our NN has to do is to predict $\mathcal{N}(0, I)$. But that's not possible. We can't predict random noise.


*However, there's a catch! I think the confusion comes from the fact that we don't want to learn a random noise. We want to learn the noise component, given a noisy input!  So the input to our model is image + noise, and we want to learn the noise part.

So let's see a pseudo-code of DDPM for clarification:
def train_loss(denoise_model, x_0, t):
    noise = torch.randn_like(x_0)

    x_noisy = q_sample(x_0=x_0, t=t, noise=noise)
    predicted_noise = denoise_model(x_noisy, t)

    loss = F.l2_loss(noise, predicted_noise)

    return loss


We can see that the model is trained to denoise a noisy input in the code.
