7
$\begingroup$

Does higher variance usually mean lower probability density? Despite the type of distribution. Thank you.

Update:

Sorry for confusion. Please allow me to clarify. If I sample the same number of data points from two distributions of the same type, but one distribution with a lower variance and another one with a higher variance, would the former sample have higher likelihood?

An example with Gaussian distributions:

  1. Sample $\mathbf{X_1}$ from $N(\mu_1, \sigma_1^2)$

  2. Sample $\mathbf{X_2}$ from $N(\mu_2, \sigma_2^2)$, where $\sigma_1 < \sigma_2$,

is it true that $P(\mathbf{X_1}; N(\mu_1, \sigma_1^2)) < P(\mathbf{X_2}; N(\mu_2, \sigma_2^2))$?

$\endgroup$
11
  • 3
    $\begingroup$ Hi: the density integrates to 1 regardless of the distribution so it's not clear what you mean by lower ? $\endgroup$
    – mlofton
    Apr 18, 2023 at 4:53
  • 1
    $\begingroup$ What do you mean by "distributions of the same type"? $\endgroup$
    – Dave
    Apr 18, 2023 at 18:00
  • 1
    $\begingroup$ That's one example. How about two Gamma distributions? Or two Student $t$ distributions? It appears what you're trying to ask concerns scale families of distributions, where changing the scale must change the density uniformly so that it continues to integrate to unity, as every density must. But that's only a guess. $\endgroup$
    – whuber
    Apr 18, 2023 at 18:12
  • 2
    $\begingroup$ Re the edited question: the answer is a very resounding no, because the laws of chance imply that sometimes a sample from one distribution will look like it's a sample from a very different distribution (having the same support). There aren't any guarantees: that's why the sample is called random. $\endgroup$
    – whuber
    Apr 18, 2023 at 18:41
  • 1
    $\begingroup$ This tells you nothing about likelihood, which is a relative or ratio concept only proportional to density; looking at different samples from different distributions does not really give anything to compare. If I sample heights in centimetres from a $N(175, 10^2)$ distribution seeing the values $162, 177, 183$, I cannot say this is "less likely" than sampling heights in metres from a $N(1.75, 0.10^2)$ distribution seeing $1.62, 1.77, 1.83$ - they are clearly equally likely in any meaningful sense as they are exactly the same measurements but on a different scale. $\endgroup$
    – Henry
    Apr 19, 2023 at 17:44

6 Answers 6

17
$\begingroup$

Up to a point. Because the density integrates to 1, the typical value of the density will be higher if the distribution has a lower variance and lower if it has a higher variance. For example, the maximum density of a Normal distribution with variance $\sigma^2$ is $1/(\sigma\sqrt{2\pi}$, which gets lower as $\sigma$ gets higher. On the other hand, because the distribution with larger variance is more spread out, it has higher density out in the tails. For example, here are two Normal distributions, with variance 1 and 4.

enter image description here

The distribution with variance 1 has higher density in the middle, but lower density at the edges.

However, if you have two distributions with different shapes, it's harder to make generalisations.

$\endgroup$
3
  • $\begingroup$ Thank you so much. What if I sample the same number of data points from the two distributions? Would the samples from the distribution with lower variance have a higher likelihood? $\endgroup$ Apr 18, 2023 at 14:09
  • $\begingroup$ Note that the OP has specified different means, $\mu_1$ and $\mu_2$, though perhaps that happened after you had posted your answer. $\endgroup$ Apr 19, 2023 at 6:50
  • 1
    $\begingroup$ Your question in the comment @TaroYamPotato is about $$\mathbb E_{X \sim p}[ \log p(X)] = \int p(x) \log p(x) \mathrm d x,$$ for a distribution with density $p$. This is known as the negative of the differential entropy and, for Gaussians, would indeed be lower if the variance is lower; the formula for Gaussians is $\frac12 + \frac12 \ln(2 \pi) + \ln \sigma$. $\endgroup$
    – Danica
    Apr 20, 2023 at 4:36
2
$\begingroup$

Standard deviation and probability density are exactly inversely correlated in one common and important case: scaled distributions, i.e. the distributions of $a\cdot X$ for different $a$ and same random variable $X$. Thomas' answer has a great example of this.

In the more general case, there does not have to be any relation. For example, take the mixture of two Gaussians at $+a$ and $-a$, with stdev $\sigma$ each. As long as $a \gg \sigma$ (so the Gaussians don't overlap), the maximum density is $1/(2\sqrt{2\pi}\, \sigma)$ and thus independent of $a$, while the variance $a^2 + \sigma^2$ depends strongly on $a$.

Edit: If we now increase $a$ while decreasing $\sigma$, variance and peak density both increase simultaneously, making it a true counterexample.

$\endgroup$
2
  • 2
    $\begingroup$ The effect is slight in the second case, but the maximum density is lower the farther apart the Gaussians are - there is no distance at which the Gaussians don't overlap. The effect can be made arbitrarily small, but not eliminated. This is an example of association between variance and peak density, not a counterexample. $\endgroup$ Apr 18, 2023 at 13:59
  • 1
    $\begingroup$ It's simpler to mix two uniform distributions than two Gaussian distributions. $\endgroup$
    – kaya3
    Apr 19, 2023 at 21:13
1
$\begingroup$

Response to updated question: If I sample 100 data points from two distributions of the same type, but one with a lower variance and one with a higher variance, would the former one have higher likelihood?

The likelihood $L(\theta|X)$ depends on both the parameter $\theta$ and the random sample $X$, so it's hard to know what is meant by a "higher likelihood." If

  • you have one sample $X$ taken from one distribution $P$, and
  • all parameters of $P$ are known save the variance $\sigma^2$, and
  • you know that $\sigma^2\in\{\sigma^2_1,\sigma^2_2\}$, where $\sigma^2_1<\sigma^2_2$,

then the likelihood ratio $\frac{L(\sigma^2_1|X)}{L(\sigma^2_2|X)}\equiv \prod_{i=1}^{100} \frac{ p(x_i|\sigma^2_1)}{ p(x_i|\sigma^2_2)}$ reflects the evidence in favor of either of the two possible values of $\sigma^2$. A very high dispersion of $X$ will result in $L(\sigma^2_1|X)<L(\sigma^2_2|X)$, but this is a higher likelihood only in the sense that $\sigma_2^2$ is more likely than $\sigma_1^2$ to be the true value of $\sigma^2$. If the dispersion of $X$ is very low, of course, you will have $L(\sigma^2_1|X)>L(\sigma^2_2|X)$.

$\endgroup$
1
$\begingroup$

In addition to the other answers. If we interpret the question as about the expected density (or likelihood) of the sample, then the answer is affirmative. If $X$ is a random variable with the density function $f$, then the expected density at the observation $X$ is $$ \DeclareMathOperator{\E}{\mathbb{E}} \E f(X) = \int f(x)^2 \; dx $$ and this will be smaller if the variance is larger. Specifically, for a scale family with $f_\sigma(x) = \frac1\sigma f(\frac{x}{\sigma})$ we find that $$ \E f_\sigma(X) = \int f_\sigma(x)^2 \; dx = \frac1\sigma \int f(x)^2\; dx $$ so it scales with the standard deviation. See also Does the integral of the probability density function squared mean something?

If we instead of using the density itself, focuses on the log density, as we often do when using the density as a likelihood function, what we get is $$ \int \log(f(x)) \cdot f(x)\; dx$$ which is the (negative of) differential entropy. This scales with variance as shown in How does entropy depend on location and scale?

$\endgroup$
0
$\begingroup$

"Probability density" is not a single number that can be lower or higher. It's a function $p(x)$ of $x$. Usually it also depends on some additional parameters $q_i$. For example, in the normal distribution, we have $q_1 = \mu$ and $q_2 = \sigma$.

To change variance, you would change these parameters. You could also change the function itself, if you hadn't said "same distribution". When you change those parameters, you will also alter $p(x)$, which may become higher or lower depending on the exact function. So no, it is not true that higher variance will always reduce probability density. For example, for a normal distribution, extreme values will have higher density.

To give a specific counter example, let's talk about the probability of getting $2 < x < 3$:

  • With a normal distribution $N(0, 1)$, $p(x > 2) = 0.023$ and $p(x > 3) = 0.001$ therefore $p(2 < x < 3) = 0.022$.
  • With a normal distribution $N(0, 4)$, $p(x > 2) = 0.158$ and $p(x > 3) = 0.067$ therefore $p(2 < x < 3) = 0.091$

In this case, the probability density between 2 and 3 has increased when variance went from 1 to 4.

I had to change your example a little, because this is trivial:

is it true that $P(\mathbf{X_1}; N(\mu_1, \sigma_1^2)) < P(\mathbf{X_2}; N(\mu_2, \sigma_2^2))$?

You've left too many independent variables, so I can say $X_1 = \mu_1$ and $X_2 = \mu_2 + 10 \sigma_2$ and then it is false, or vice versa and it is true.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. I was wondering the likelihood of the samples from such two distributions. In your example, if we sample many data points from $N(0,1)$ and from N(0,4), respectively, the two samples would be different. I was wondering the likelihoods of the two samples given the distribution where they were sampled. $\endgroup$ Apr 19, 2023 at 5:40
  • $\begingroup$ @TaroYamPotato The likelihood you describe is exactly what my calculations show. If you sampled 1000 points from either distribution, about 22 would be between 2 and 3 for the first distribution, and about 91 for the second. $\endgroup$ Apr 21, 2023 at 22:53
-1
$\begingroup$

just take 10 points : 1,2,3,4,5,6,7,8,9,10 (discrete samples) the mean is: 5.5 and you may take variance to be approx. = 3 this standardizes the data to -1.5,-1.167,-0.83,-0.5,-0.1667,0.1667,0.5,0.833,1.1667,1.5

(now imagine the same thing for a continuous domain) so if you observe the plot, the PDF will be a curve closer to 1 at E(X) which is well, the most probable value of X and that after standardization is 0. So, the curve is denser around X=0 and not so dense as you move to sides which is why you observe a bell curve post standardization. but if you look at the original data; the variance = 3(approx.) > 1. If the data were continuous; the mean being 5.5 would imply that the curve peaks at X=5.5(well, not exactly X=5.5 because then f(x) = 0, lets say in an interval epsilon around 5.5) yes; however the variance being larger means the next value is at a greater distance. Notice how 2-1 = 1, 3-2 = 1 but; -0.83-(-1.167) = 0.337 < 1. think of balls scattered in a playground vs. gathered in one place. the density is higher in that one place but negligent in other parts of the playground. in the first case, the density is well, not that high except at some place where you have 2 balls or something.

this also helps answer the question: why would you want to collect them all in one place? well because then it's easier to find.

$\endgroup$
1
  • $\begingroup$ I lost you at "observe the plot:" what plot, exactly? What is the PDF of "just ... 10 points"? If by "E(X)" you mean (as is conventional) the expectation, then it's invalid to characterize it as "the most probable value," which would be a mode. (The same error occurs in characterizing the sample mean as where a curve "peaks.") A "larger" variance (larger than what?) says little or nothing about gaps in the data, such as "the next value." The problems in this post go on and on, but if you would like to fix it, consider starting with these issues. $\endgroup$
    – whuber
    Aug 8, 2023 at 16:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.