I am reading about generative models. I came across an example a few times but I cannot come up with an explanation for it.

Imagine data is generated according to $p_\text{data}(x)$. It is often said that for higher dimensional data, the log-likelihood of the model also increases linearly with the dimension. Why is that the case ? I cannot wrap my head around this fact and any help would be very appreciated. I imagine $p_\text{data}(x)$ is a density and it might relate to the curse of dimensionality, but I still cannot show it semi-formally.

EDIT: I am not asking whether the likelihood increases with more samples, I am asking about two models with different input dimension, e.g. $x \in \mathbb R^{100}$ and $x \in \mathbb R^{500}$. Apparently, the likelihood of the true data generation process is higher when the dimension is larger, see deepgenerativemodels.github.io/assets/slides/cs236_lecture9.pdf, slide 5, and I do not understand why

  • $\begingroup$ Not sure which scaling exactly this refers to but, a likelihood $f(x;\theta)$ is still scales with the dimension of $x$ $\endgroup$ Sep 3 at 0:55
  • 1
    $\begingroup$ When you refer to "higher dimensional data", do you mean "larger sample size?" $\endgroup$
    – jbowman
    Sep 3 at 1:13
  • 1
    $\begingroup$ I don't believe this is a duplicate. This is concerning the dimension of each sample, not the number of samples. An example: compare 50 samples from a multivariate Gaussian $(0, I)$ with 100 dimensions to 50 samples from a multivariate Gaussian $(0, I)$ with 10 dimensions. Numerically, the former will have larger log-likelihoods. (Larger, not higher—they're more negative.) $\endgroup$ Sep 3 at 16:38
  • 1
    $\begingroup$ Please - in your edit - bear in mind that the likelihood does not become higher. It is a negative quantity whose absolute value grows. This is stated in the slides you cite as well. $\endgroup$ Sep 3 at 17:01
  • 1
    $\begingroup$ @Alf Likelihood can be positive, but what's important is the direction of the growth! $\endgroup$ Sep 4 at 6:29

1 Answer 1


Actually this isn't always the case, it's just often the case when the data is sufficiently spread out or discrete.

Let's start with the discrete case because it's the easiest. Suppose in one dimension, you have $P(x_1)$. If you add a second dimension to this, you will now have $P(x_1 \cap x_2)$.

Note that $P(x_1 \cap x_2) = P(x_1) P(x_2 | x_1)$.

Because $P(x_2 | x_1) \le 1$, for all $x_2$, $P(x_1) \ge P(x_1 \cap x_2)$. So the maximum likelihood for the 2D distribution cannot be greater than the likelihood for a 1D distribution.

The concept is similar for continuous variables. However, the inequality does not always hold. In particular,

$f_{x_1 \cap x_2}(x_1, x_2) = f_{x_1}(x_1) f_{x_2|x_1}(x_2, x_1)$

So in this case, we will have $f_{x_1}(x_1) > f_{x_1 \cap x_2}(x_1, x_2)$ if and only if $f_{x_2|x_1}(x_2, x_1) < 1$. This isn't always true. As a very silly but illustrative example, suppose that somehow $f_{x_2|x_1}(x_2, x_1)$ was a Uniform(0, $\theta$) distribution. Then the pdf is $1/\theta$, so if $\theta > 1$, then yes, $f_{x_1 \cap x_2}(x_1, x_2) < f_{x_1}(x_1)$. But if $\theta < 1$, then $f_{x_1 \cap x_2}(x_1, x_2) > f_{x_1}(x_1)$!

It's a little more complicated than this, but if the new dimensions are well spread out beyond a unit interval then the higher dimension likelihood is (probably) lower. But if it's well concentrated within a unit interval, this may not hold!

  • $\begingroup$ I think it exactly what I needed, thanks a lot for the explanation :) $\endgroup$
    – Alf
    Sep 4 at 5:29
  • $\begingroup$ Just wanted to drop a congrats for the great explanation👍🏻 $\endgroup$
    – Alberto
    Sep 4 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.