Does higher variance usually mean lower probability density?

Question

Does higher variance usually mean lower probability density? Despite the type of distribution. Thank you.

Update：

Sorry for confusion. Please allow me to clarify. If I sample the same number of data points from two distributions of the same type, but one distribution with a lower variance and another one with a higher variance, would the former sample have higher likelihood?

An example with Gaussian distributions:

1. Sample $\mathbf{X_1}$ from $N(\mu_1, \sigma_1^2)$

2. Sample $\mathbf{X_2}$ from $N(\mu_2, \sigma_2^2)$, where $\sigma_1 < \sigma_2$,

is it true that $P(\mathbf{X_1}; N(\mu_1, \sigma_1^2)) < P(\mathbf{X_2}; N(\mu_2, \sigma_2^2))$?

Answer 1

Up to a point. Because the density integrates to 1, the typical value of the density will be higher if the distribution has a lower variance and lower if it has a higher variance. For example, the maximum density of a Normal distribution with variance $\sigma^2$ is $1/(\sigma\sqrt{2\pi}$, which gets lower as $\sigma$ gets higher. On the other hand, because the distribution with larger variance is more spread out, it has higher density out in the tails. For example, here are two Normal distributions, with variance 1 and 4.

The distribution with variance 1 has higher density in the middle, but lower density at the edges.

However, if you have two distributions with different shapes, it's harder to make generalisations.

Answer 2

Standard deviation and probability density are exactly inversely correlated in one common and important case: scaled distributions, i.e. the distributions of $a\cdot X$ for different $a$ and same random variable $X$. Thomas' answer has a great example of this.

In the more general case, there does not have to be any relation. For example, take the mixture of two Gaussians at $+a$ and $-a$, with stdev $\sigma$ each. As long as $a \gg \sigma$ (so the Gaussians don't overlap), the maximum density is $1/(2\sqrt{2\pi}\, \sigma)$ and thus independent of $a$, while the variance $a^2 + \sigma^2$ depends strongly on $a$.

Edit: If we now increase $a$ while decreasing $\sigma$, variance and peak density both increase simultaneously, making it a true counterexample.

Answer 3

Response to updated question: If I sample 100 data points from two distributions of the same type, but one with a lower variance and one with a higher variance, would the former one have higher likelihood?

The likelihood $L(\theta|X)$ depends on both the parameter $\theta$ and the random sample $X$, so it's hard to know what is meant by a "higher likelihood." If

• you have one sample $X$ taken from one distribution $P$, and
• all parameters of $P$ are known save the variance $\sigma^2$, and
• you know that $\sigma^2\in\{\sigma^2_1,\sigma^2_2\}$, where $\sigma^2_1<\sigma^2_2$,

then the likelihood ratio $\frac{L(\sigma^2_1|X)}{L(\sigma^2_2|X)}\equiv \prod_{i=1}^{100} \frac{ p(x_i|\sigma^2_1)}{ p(x_i|\sigma^2_2)}$ reflects the evidence in favor of either of the two possible values of $\sigma^2$. A very high dispersion of $X$ will result in $L(\sigma^2_1|X)<L(\sigma^2_2|X)$, but this is a higher likelihood only in the sense that $\sigma_2^2$ is more likely than $\sigma_1^2$ to be the true value of $\sigma^2$. If the dispersion of $X$ is very low, of course, you will have $L(\sigma^2_1|X)>L(\sigma^2_2|X)$.

Answer 4

"Probability density" is not a single number that can be lower or higher. It's a function $p(x)$ of $x$. Usually it also depends on some additional parameters $q_i$. For example, in the normal distribution, we have $q_1 = \mu$ and $q_2 = \sigma$.

To change variance, you would change these parameters. You could also change the function itself, if you hadn't said "same distribution". When you change those parameters, you will also alter $p(x)$, which may become higher or lower depending on the exact function. So no, it is not true that higher variance will always reduce probability density. For example, for a normal distribution, extreme values will have higher density.

To give a specific counter example, let's talk about the probability of getting $2 < x < 3$:

• With a normal distribution $N(0, 1)$, $p(x > 2) = 0.023$ and $p(x > 3) = 0.001$ therefore $p(2 < x < 3) = 0.022$.
• With a normal distribution $N(0, 4)$, $p(x > 2) = 0.158$ and $p(x > 3) = 0.067$ therefore $p(2 < x < 3) = 0.091$

In this case, the probability density between 2 and 3 has increased when variance went from 1 to 4.

I had to change your example a little, because this is trivial:

is it true that $P(\mathbf{X_1}; N(\mu_1, \sigma_1^2)) < P(\mathbf{X_2}; N(\mu_2, \sigma_2^2))$?

You've left too many independent variables, so I can say $X_1 = \mu_1$ and $X_2 = \mu_2 + 10 \sigma_2$ and then it is false, or vice versa and it is true.

Answer 5

In addition to the other answers. If we interpret the question as about the expected density (or likelihood) of the sample, then the answer is affirmative. If $X$ is a random variable with the density function $f$, then the expected density at the observation $X$ is $$ \DeclareMathOperator{\E}{\mathbb{E}} \E f(X) = \int f(x)^2 \; dx $$ and this will be smaller if the variance is larger. Specifically, for a scale family with $f_\sigma(x) = \frac1\sigma f(\frac{x}{\sigma})$ we find that $$ \E f_\sigma(X) = \int f_\sigma(x)^2 \; dx = \frac1\sigma \int f(x)^2\; dx $$ so it scales with the standard deviation. See also Does the integral of the probability density function squared mean something?

If we instead of using the density itself, focuses on the log density, as we often do when using the density as a likelihood function, what we get is $$ \int \log(f(x)) \cdot f(x)\; dx$$ which is the (negative of) differential entropy. This scales with variance as shown in How does entropy depend on location and scale?

Answer 6

just take 10 points : 1,2,3,4,5,6,7,8,9,10 (discrete samples) the mean is: 5.5 and you may take variance to be approx. = 3 this standardizes the data to -1.5,-1.167,-0.83,-0.5,-0.1667,0.1667,0.5,0.833,1.1667,1.5

(now imagine the same thing for a continuous domain) so if you observe the plot, the PDF will be a curve closer to 1 at E(X) which is well, the most probable value of X and that after standardization is 0. So, the curve is denser around X=0 and not so dense as you move to sides which is why you observe a bell curve post standardization. but if you look at the original data; the variance = 3(approx.) > 1. If the data were continuous; the mean being 5.5 would imply that the curve peaks at X=5.5(well, not exactly X=5.5 because then f(x) = 0, lets say in an interval epsilon around 5.5) yes; however the variance being larger means the next value is at a greater distance. Notice how 2-1 = 1, 3-2 = 1 but; -0.83-(-1.167) = 0.337 < 1. think of balls scattered in a playground vs. gathered in one place. the density is higher in that one place but negligent in other parts of the playground. in the first case, the density is well, not that high except at some place where you have 2 balls or something.

this also helps answer the question: why would you want to collect them all in one place? well because then it's easier to find.