Why does higher dimensional data has higher likelihood?

Question

I am reading about generative models. I came across an example a few times but I cannot come up with an explanation for it.

Imagine data is generated according to $p_\text{data}(x)$. It is often said that for higher dimensional data, the log-likelihood of the model also increases linearly with the dimension. Why is that the case ? I cannot wrap my head around this fact and any help would be very appreciated. I imagine $p_\text{data}(x)$ is a density and it might relate to the curse of dimensionality, but I still cannot show it semi-formally.

EDIT: I am not asking whether the likelihood increases with more samples, I am asking about two models with different input dimension, e.g. $x \in \mathbb R^{100}$ and $x \in \mathbb R^{500}$. Apparently, the likelihood of the true data generation process is higher when the dimension is larger, see deepgenerativemodels.github.io/assets/slides/cs236_lecture9.pdf, slide 5, and I do not understand why

Answer 1

Actually this isn't always the case, it's just often the case when the data is sufficiently spread out or discrete.

Let's start with the discrete case because it's the easiest. Suppose in one dimension, you have $P(x_1)$. If you add a second dimension to this, you will now have $P(x_1 \cap x_2)$.

Note that $P(x_1 \cap x_2) = P(x_1) P(x_2 | x_1)$.

Because $P(x_2 | x_1) \le 1$, for all $x_2$, $P(x_1) \ge P(x_1 \cap x_2)$. So the maximum likelihood for the 2D distribution cannot be greater than the likelihood for a 1D distribution.

The concept is similar for continuous variables. However, the inequality does not always hold. In particular,

$f_{x_1 \cap x_2}(x_1, x_2) = f_{x_1}(x_1) f_{x_2|x_1}(x_2, x_1)$

So in this case, we will have $f_{x_1}(x_1) > f_{x_1 \cap x_2}(x_1, x_2)$ if and only if $f_{x_2|x_1}(x_2, x_1) < 1$. This isn't always true. As a very silly but illustrative example, suppose that somehow $f_{x_2|x_1}(x_2, x_1)$ was a Uniform(0, $\theta$) distribution. Then the pdf is $1/\theta$, so if $\theta > 1$, then yes, $f_{x_1 \cap x_2}(x_1, x_2) < f_{x_1}(x_1)$. But if $\theta < 1$, then $f_{x_1 \cap x_2}(x_1, x_2) > f_{x_1}(x_1)$!

It's a little more complicated than this, but if the new dimensions are well spread out beyond a unit interval then the higher dimension likelihood is (probably) lower. But if it's well concentrated within a unit interval, this may not hold!