# Why is the forward process referred to as the "ground truth" in diffusion models?

I've seen in many tutorials on diffusion models refer to the distribution of the latent variables induced by the forward process as "ground truth". I wonder why. What we can actually see is only the original data point $$x_0$$, while all other $$x_t$$'s are "imaginary" latent variables. The forward process is just a distribution we plug in for the ELBO. Why are the $$x_t$$ "true" when they are in fact latent variables that we can't observe?

• In diffusion models, the forward process gradually transforms the original data $x_0$ into noise through a series of latent variables $x_t$. These $x_t$ values are considered your so called 'ground truth' in the sense that they represent the true distribution over time steps of the denoising generating process, even though they aren't directly observable. Can you actually reference such 'ground truth' usage? Commented Jun 30 at 3:59
• @cinch For example, the tutorial "Understanding Diffusion Models: A Unified Perspective." Also, from the score-based perspective, we indeed view the forward process as a "true" distribution for each latent variables to obtain the score function. Commented Jun 30 at 4:15

From your reference "Understanding Diffusion Models: A Unified Perspective", the ground truth claims for score function appear in places like:

The score model can be optimized by minimizing the Fisher Divergence with the ground truth score function... What does the score function represent? For every $$\mathbf{x}$$, taking the gradient of its log likelihood with respect to $$\mathbf{x}$$ essentially describes what direction in data space to move in order to further increase its likelihood. Intuitively, then, the score function defines a vector field over the entire space that data $$\mathbf{x}$$ inhabits, pointing towards the modes... Note that the objective in Equation 157 relies on having access to the ground truth score function, which is unavailable to us for complex distributions such as the one modeling natural images. Fortunately, alternative techniques known as score matching [14, 15, 16, 17] have been derived to minimize this Fisher divergence without knowing the ground truth score, and can be optimized with stochastic gradient descent.

Thus there's no surprise here to treat score function which provides essential information about the data distribution guiding the model in reconstructing data from noise as some usually complex or intractable 'ground truth' function in practice, just like the true intractable posterior distribution of latent variable or the true marginal likelihood of evidence in VAEs.

The only other places mentioning 'ground truth' only refer to the true intractable posterior distribution of latent variable in the ELBO derivation section.

Directly computing and maximizing the likelihood $$p(\mathbf{x})$$ is difficult because it either involves integrating out all latent variables $$\mathbf{z}$$ in Equation 1, which is intractable for complex models, or it involves having access to a ground truth latent encoder $$p(\mathbf{z}|\mathbf{x})$$ in Equation 2.

Unfortunately, it is intractable to minimize this KL Divergence term directly, as we do not have access to the ground truth $$p(\mathbf{z}|\mathbf{x})$$ distribution. However... the likelihood of our data is always a constant with respect to $$\mathbf{φ}$$... and does not depend on $$\mathbf{φ}$$ whatsoever. Since the ELBO and KL Divergence terms sum up to a constant, any maximization of the ELBO term with respect to $$\mathbf{φ}$$ necessarily invokes an equal minimization of the KL Divergence term. Thus, the ELBO can be maximized as a proxy for learning how to perfectly model the true latent posterior distribution; the more we optimize the ELBO, the closer our approximate posterior gets to the true posterior.

Since the latent encoder $$p(\mathbf{z}|\mathbf{x})$$ represents the distribution we are trying to approximate or learn from the data, it's considered as the 'ground truth' in variational inference. And these references also explain the relation between ELBO and KL quite well and hopefully could address your other concerns about variational inference.