What is the difference between finite and infinite variance ? My stats knowledge is rather basic; Wikipedia / Google wasn't much help here.

  • 8
    Distributions with infinite variance are heavy-tailed; there are lots of outliers, and can have properties that are different from what one is used to seeing. For example, the sample mean of samples drawn from a Cauchy distribution has the same (Cauchy) distribution as the individual samples. This is quite different from the usual belief that the sample mean is a better "estimator" than any individual sample. – Dilip Sarwate Apr 19 '14 at 22:40
  • So heavy-tailed with = infinite variance ? – AfterWorkGuinness Apr 19 '14 at 23:02
  • 3
    No, heavy-tailed is not the same as having infinite variance, or at least, not in my view. However, I am not a statistician and so you should wait for for a more authoritative answer from highly-ranked users in this forum. – Dilip Sarwate Apr 19 '14 at 23:27
  • 3
    Infinite variance happens when the integral (sum) defining the population variance increases beyond any finite bound as the limit is taken. Some discussion of examples here – Glen_b Apr 20 '14 at 1:31
  • 1
    I think most importantly, most central limit theorems fail to hold for such a population and thus some common results will collapse. – Henry.L Sep 13 '15 at 18:37

$\DeclareMathOperator{\E}{E} \DeclareMathOperator{\var}{var}$ What does it mean for a random variable to have "infinite variance"? What does it mean for a random variable to have infinite expectation? The explanation in both cases are rather similar, so let us start with the case of expectation, and then variance after that.

Let $X$ be a continuous random variable (RV) (our conclusions will be valid more generally, for the discrete case, replace integral by sum). To simplify exposition, lets assume $X \ge 0$.

Its expectation is defined by the integral $$ \E X = \int_0^\infty x f(x) \, d x $$ when that integral exists, that is, is finite. Else we say the expectation does not exist. That is an improper integral, and by definition is $$ \int_0^\infty x f(x) \, d x = \lim_{a \rightarrow \infty} \int_0^a x f(x) \, d x $$ For that limit to be finite, the contribution from the tail must vanish, that is, we must have $$ \lim_{a \rightarrow \infty} \int_a^\infty x f(x) \, d x =0 $$ A necessary (but not sufficient) condition for that to be the case is $\lim_{x\rightarrow \infty} x f(x) =0 $. What the above displayed condition says, is that, the contribution to the expectation from the (right) tail must be vanishing. If so is not the case, the expectation is dominated by contributions from arbitrarily large realized values. In practice, that will mean that empirical means will be very unstable, because they will be dominated by the infrequent very large realized values. And note that this instability of sample means will not disappear with large samples --- it is a built-in part of the model!

In many situations, that seems unrealistic. Lets say an (life) insurance model, so $X$ models some (human) lifetime. We know that, say $X > 1000$ doesn't occur, but in practice we use models without an upper limit. The reason is clear: No hard upper limit is known, if a person is (say) 110 years old, there is no reason he cannot live one more year! So a model with a hard upper limit seems artificial. Still, we do not want the extreme upper tail to have much influence.

If $X$ has a finite expectation, then we can change the model to have a hard upper limit without undue influence to the model. In situations with a fuzzy upper limit that seems good. If the model have infinite expectation, then, any hard upper limit we introduce to the model will have dramatic consequences! That is the real importance of infinite expectation.

With finite expectation, we can be fuzzy about upper limits. With infinite expectation, we cannot.

Now, much the same can be said about infinite variance, mutatis mutandi.

To make clearer, let us see at an example. For the example we use the Pareto distribution, implemented in the R package (on CRAN) actuar as pareto1 --- single-parameter Pareto distribution also known as Pareto type 1 distribution. It has probability density function given by $$ f(x) = \begin{cases} \frac{\alpha m^\alpha}{x^{\alpha+1}} &, x\ge m \\ 0 &, x<m \end{cases} $$ for some parameters $m>0, \alpha>0$. When $\alpha > 1 $ the expectation exists and is given by $\frac{\alpha}{\alpha-1}\cdot m$. When $\alpha \le 1$ the expectation do not exist, or as we say, it is infinite, because the integral defining it diverges to infinity. We can define the First moment distribution (see the post When would we use tantiles and the medial, rather than quantiles and the median? for some information and references) as $$ E(M) = \int_m^M x f(x) \, d x = \frac{\alpha}{\alpha-1} \left( m - \frac{m^\alpha}{M^{\alpha-1}} \right) $$ ( this exists without regard to if the expectation itself exists). (Later edit: I invented the name "first moment distribution, later I learned this is related to what is "officially" names partial moments).

When the expectation exists ($\alpha> 1$) we can divide by it to get the relative first moment distribution, given by $$ Er(M) = E(m)/E(\infty) = 1-\left(\frac{m}{M}\right)^{\alpha-1} $$ When $\alpha$ is just a little bit larger than one, so the expectation "just barely exists", the integral defining the expectation will converge slowly. Let us look at the example with $m=1, \alpha=1.2$. Let us plot then $Er(M)$ with the help of R:

### Function for opening new plot file:
open_png  <-  function(filename) png(filename=filename,
                                     type="cairo-png")

library(actuar) # from CRAN
### Code for Pareto type I distribution:
# First plotting density and "graphical moments" using ideas from http://www.quantdec.com/envstats/notes/class_06/properties.htm   and used some times at cross validated

m  <-  1.0
alpha <- 1.2
# Expectation:
E   <-  m * (alpha/(alpha-1))
# upper limit for plots:
upper  <- qpareto1(0.99, alpha, m)   
#
open_png("first_moment_dist1.png")
Er  <- function(M, m, alpha) 1.0 - (m/M)^(alpha-1.0)
### Inverse relative first moment distribution function,  giving
#   what we may call "expectation quantiles":
Er_inv  <-   function(eq, m, alpha) m*exp(log(1.0-eq)/(1-alpha))     

plot(function(M) Er(M, m, alpha), from=1.0,  to=upper)
plot(function(M) ppareto1(M, alpha, m), from=1.0,  to=upper, add=TRUE,  col="red")
dev.off()

which produces this plot:

enter image description here

For example, from this plot you can read that about 50% of the contribution to the expectation come from observations above around 40. Given that the expectation $\mu$ of this distribution is 6, that is astounding! (this distribution do not have existing variance. For that we need $\alpha > 2$).

The function Er_inv defined above is the inverse relative first moment distribution, an analogue to the quantile function. We have:

> ### What this plot shows very clearly is that most of the contribution to the expectation come from the very extreme right tail!
# Example   
eq  <-  Er_inv(0.5, m, alpha)
ppareto1(eq, alpha, m)
eq

> > > [1] 0.984375
> [1] 32
> 

This shows that 50% of the contributions to the expectation comes from the upper 1.5% tail of the distribution! So, especially in small samples where there is a high probability that the extreme tail is not represented, the arithmetic mean, while still being an unbiased estimator of the expectation $\mu$, must have a very skew distribution. We will investigate this by simulation: First we use a sample size $n=5$.

set.seed(1234)
n  <-  5
N  <-  10000000  # Number of simulation replicas
means  <-  replicate(N,  mean(rpareto1(n, alpha, m) ))


> mean(means)
[1] 5.846645
> median(means)
[1] 2.658925
> min(means)
[1] 1.014836
> max(means)
[1] 633004.5
length(means[means <=100])
[1] 9970136

To get a readable plot we only show the histogram for the part of the sample with values below 100, which is a very large part of the sample.

open_png("mean_sim_hist1.png")
hist(means[means<=100],  breaks=100, probability=TRUE)
dev.off()

enter image description here

The distribution of the arithmetical means is very skew,

> sum(means <= 6)/N
[1] 0.8596413
> 

almost 86% of the empirical means are less or equal than the theoretical mean, the expectation. That is what we should expect, since most of the contribution to the mean comes from the extreme upper tail, which is unrepresented in most samples.

We need to go back to reassess our earlier conclusion. While the existence of the mean makes it possible to be fuzzy about upper limits, we see that when "the mean just barely exists", meaning that the integral is slowly convergent, we cannot really be that fuzzy about upper limits. Slowly convergent integrals has the consequence that it might be better to use methods that do not assume that the expectation exists. When the integral is very slowly converging, it is in practice as if it didn't converge at all. The practical benefits that follow from an convergent integral is a chimera in the slowly convergent case! That is one way to understand N N Taleb's conclusion in http://fooledbyrandomness.com/complexityAugust-06.pdf

  • 1
    Fantastic answer. – Karl Aug 10 '16 at 19:09

Variance is the measure of dispersion of the distribution of values of a random variable. It's not the only such measure, e.g. mean absolute deviation is one of alternatives.

The infinite variance means that random values don't tend to concentrate around the mean too tightly. It could mean that there's large enough probability that the next random number will be very far away from the mean.

The distributions like Normal (Gaussian) can produce random numbers very far away from the mean, but the probability of such events decrease very rapidly with magnitude of the deviation.

In that regard when you look at the plot of Cauchy distribution or a Gaussian (normal) distribution, they don't look very different visually. However, if you try to compute the variance of the Cauchy distribution it'll be infinite, while Gaussian's is finite. So, normal distribution is more tight around its mean compared to Cauchy's.

Btw, if you talk to mathematicians, they'll insist that Cauchy distribution doesn't have a well defined mean, that it's infinite. This sounds ridiculous to physicists who'd point out to the fact that Cauchy's symmetrical, hence, it's bound to have a mean. In this case they'd argue the problem is with your definition of mean, not with the Cauchy's distribution.

  • 2
    Are you sure about mathematicians and physicists? My impression is that physisicst can be very rigouros about such things! See my answer, slow convergence makes a value of little value! Also, no mathematician would say the Cauchy has infinite mean, the proper limit defing the integral simply does not exist, since it diverges in both tail. Talking about the expectation being $\infty$ or $-\infty$ only makes sense when the divergence is in one tail only. – kjetil b halvorsen Sep 9 '15 at 20:11
  • @kjetilbhalvorsen, "no mathematician would say the Cauchy has infinite mean" - that mean isn't well defined is exactly what I've been told by my stats professor, while my theor Physcis advisor was surprised there's even a question about the mean, "of course it's zero, and if you disagree then there's something wrong with your definition of mean" – Aksakal Jan 27 '16 at 14:58
  • Did you ask him about his definition of the mean? – kjetil b halvorsen Jan 27 '16 at 15:03
  • @kjetilbhalvorsen, Riemann integral if you're talking about math prof. His argument is that in Riemann sum you don't define a certain order of sum or partitioning of sum, so your sum will be infinite. Physicists point is a symmetry, clearly, it "has to be zero" – Aksakal Jan 27 '16 at 15:51
  • 1
    Then maybe you can tell him he defined the median, not the mean. – kjetil b halvorsen Jan 27 '16 at 15:55

An alternative way to look at is by the quantile function.

$$Q(F(x)) = x$$

Then we can compute a moment or expectation

$$E(T(x)) = \int_{-\infty}^\infty T(x) f(x) dx\\$$

alternatively as (replacing $f(x)dx = dF$):

$$E(T(x)) = \int_{0}^1 T(Q(F)) dF \\$$

Say we wish to compute the first moment then $T(x) = x$. In the image below this corresponds to the area between F and the vertical line at $x=0$ (where the area on the left side may count as negative when $T(x)<0$). The second moment would correspond to the volume that the same area sweeps when it is rotated along the line at $x=0$ (with a factor $\pi$ difference).

Cauchy versus Normal

The curves in the image show how much each quantile contributes in the computation.

For the normal curve there are only very few quantiles with a large contribution. But for the Cauchy curve there are many more quantiles with a large contribution. If the curve $T(Q(F))$ goes sufficiently fast enough to infinity when F approaches zero or one, then the area can be infinite.

This infinity may not be so strange since the integrand itself distance (mean) or squared distance (variance) can become infinite. It is only a question how much weight, how much percent of F, those infinite tails have.

In the summation/integration of distance from zero (mean) or squared distance from the mean (variance) a single point that is very far away will have more influence on the average distance (or squared distance) than a lot of points nearby.

Thus when we move towards infinity the density may decrease, but the influence on the sum of some (increasing) quantity, e.g. distance or squared distance does not necessarily change.

If for each amount of mass at some distance $x$ there is half or more mass at a distance $\sqrt{2}x$ then you will get that the sum of total mass $\sum \frac{1}{2^n}$ will converge because the contribution of mass decreases, but the variance becomes infinite since that contribution does not decrease $\sum ((\sqrt{2}x)^n)^2 \frac{1}{2^n} \to \infty$

Most distributions you encounter probably have finite variance. Here is a discrete example $X$ that has infinite variance but finite mean:

Let its probability mass function be $ p(k) = c/|k|^3$, for $k \in \mathbb{Z} \setminus\{0\}$, $p(0) = 0$, where $c = (2\zeta(3))^{-1} := (2\sum_{k=1}^\infty 1/k^3)^{-1} < \infty$. First of all because $\mathbb{E} \mid X\mid < \infty$ it has finite mean. Also it has infinite variance because $2 \sum_{k=1}^\infty k^2 / |k|^3 = 2\sum_{k=1}^\infty k^{-1} = \infty$.

Note: $\zeta(x) :=\sum_{k=1}^\infty k^{-x}$ is the Riemann zeta function. There are many other examples, just not so pleasant to write down.

  • 4
    Just because the distribution is symmetric (i.e. an even function), does not necessarily make the mean $0$; the mean may not exist because the sum/integral turns out to be of the form $\infty - \infty$ – Dilip Sarwate Jun 10 '15 at 16:30

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.