21
$\begingroup$

Many PDFs range from minus to positive infinity, yet some means are defined and some are not. What common trait makes some computable?

$\endgroup$
  • 14
    $\begingroup$ Convergent integrals. $\endgroup$ – Reinstate Monica Sep 2 '16 at 3:08
  • 1
    $\begingroup$ This distributions are mathematical abstractions. If the integral doesn't converge then mean is not defined. However, what's not mentioned in answers below is that PDFs with minus infinity to plus infinity cannot model real data sources. There is no such physical process to generate such data in real life. In my opinion all real data sources will be bounded and you will be able to approximate the mean. $\endgroup$ – Cagdas Ozgenc Sep 2 '16 at 12:56
  • 3
    $\begingroup$ @Cagdas That remark does not appear to be correct. There are plenty of heavy-tailed processes. Their divergent expectations are manifest as extreme variability in long-run averages. For a convincing application of a Cauchy model, for instance, see Douglas Zare's post at stats.stackexchange.com/a/36037/919. $\endgroup$ – whuber Sep 2 '16 at 13:29
  • 2
    $\begingroup$ @CagdasOzgenc: You should read Black Swan by Taleb to see just how wrong that reasoning is. While heuristically there might not be a process which perfectly generates a distribution with an undefined mean or infinite mean, there are plenty of examples where people underestimate just how fat the tails are of their distribution and proceed to calculate means, whereas the true distribution has a mean that is completely different and usually right-skewed. This kind of improper reasoning led to many risk-assessment gafs in finance where risk is underestimated by many orders of magnitude. $\endgroup$ – Alex R. Sep 2 '16 at 19:10
  • 1
    $\begingroup$ @Cagdas Ozgenc: For a discussion why your argument is wrong see stats.stackexchange.com/questions/94402/… $\endgroup$ – kjetil b halvorsen Sep 8 '16 at 9:56
23
$\begingroup$

The mean of a distribution is defined in terms of an integral (I'll write it as if for a continuous distribution - as a Riemann integral, say - but the issue applies more generally; we can proceed to Stieltjes or Lebesgue integration to deal with these properly and all at once):

$$E(X) = \int_{-\infty}^\infty x f(x)\, dx$$

But what does that mean? It's effectively a shorthand for

$$\stackrel{\lim}{_{a\to\infty,b\to\infty}} \int_{-a}^b x\, f(x)\, dx$$

or

$$\stackrel{\lim}{_{a\to\infty}} \int_{-a}^0 x f(x)\, dx \, +\, \stackrel{\lim}{_{b\to\infty}} \int_{0}^b x f(x)\, dx$$

(though you could break it anywhere not just at 0)

The problem comes when the limits of those integrals are not finite.

So for example, consider the standard Cauchy density, which is proportional to $\frac{1}{1+x^2}$ ... note that

$$\stackrel{\lim}{_{b\to\infty}} \int_{0}^b \frac{x}{1+x^2}\, dx$$

let $u=1+x^2$, so $du=2x\,dx$

$$=\,\stackrel{\lim}{_{b\to\infty}}\frac12 \int_{1}^{1+b^2} \frac{1}{u}\, du$$

$$=\,\stackrel{\lim}{_{b\to\infty}} \frac{_1}{^2}\ln(u)\Bigg |_{1}^{1+b^2} $$

$$=\,\stackrel{\lim}{_{b\to\infty}} \frac{_1}{^2}\ln(1+b^2)$$

which isn't finite. The limit in the lower half is also not finite; the expectation is thereby undefined.

Or if we had as our random variable the absolute value of a standard Cauchy, its entire expectation would be proportional to that limit we just looked at (i.e. $\stackrel{\lim}{_{b\to\infty}} \frac12\ln(1+b^2)$).

On the other hand, some other densities do continue out "to infinity" but their integral does have a limit.

$\endgroup$
  • 1
    $\begingroup$ You can (of course) also see the same thing in similar discrete probability distributions. Take a distribution where the probability if $n$ occurring, for integer $n>0$, is proportional to $\frac{1}{n^2}$. The sum of probabilities is finite (which is just as well since it needs to have limit 1: actually our constant must be $\frac{6}{\pi^2}$ or whatever it is), but since the sum of $\frac{1}{n}$ diverges it has no mean. Whereas if we choose a probability proportional to $\frac{1}{n^3}$ then the mean involves a sum of $\frac{1}{n^2}$ and we're fine, that's "small enough" that it converges. $\endgroup$ – Steve Jessop Sep 2 '16 at 11:27
  • 1
    $\begingroup$ Yes, $\frac{6}{\pi^2}$ is the scaling constant for that (to make it sum to1). $\endgroup$ – Glen_b Sep 2 '16 at 11:45
8
$\begingroup$

The other answers are good, but might not convince everyone, especially people who take one look at the Cauchy distribution (with $x_0 = 0$) and say it's still intuitively obvious that the mean should be zero.

The reason the intuitive answer is not correct from the mathematical perspective is due to the Riemann rearrangement theorem (video).

Effectively what you're doing when you're looking at a Cauchy and saying that the mean "should be zero" is that you're splitting down the "center" at zero, and then claiming the moments of the two sizes balance. Or in other words, you're implicitly doing an infinite sum with "half" the terms positive (the moments at each point to the right) and "half" the terms negative (the moments at each point to the left) and claiming it sums to zero. (For the technically minded: $\int_{0}^\infty f(x_0+r)r\, dr - \int_{0}^{\infty} f(x_0-r)r\, dr = 0$)

The Riemann rearrangement theorem says that this type of infinite sum (one with both positive and negative terms) is only consistent if the two series (positive terms only and negative terms only) are each convergent when taken independently. If both sides (positive and negative) are divergent on their own, then you can come up with an order of summation of the terms such that it sums to any number. (Video above, starting at 6:50)

So, yes, if you do the summation in a balanced manner from 0 out, the first moments from the Cauchy distribution cancel out. However, the (standard) definition of mean doesn't enforce this order of summation. You should be able to sum the moments in any order and have it be equally valid. Therefore, the mean of the Cauchy distribution is undefined - by judiciously choosing how you sum the moments, you can make them "balance" (or not) at practically any point.

So to make the mean of a distribution defined, the two moment integrals need to each be independently convergent (finite) around the proposed mean (which, when you do the math, is really just another way of saying that the full integral ($\int_{-\infty}^\infty f(x)x\, dx$) needs to be convergent). If the tails are "fat" enough to make the moment for one side infinite, you're done. You can't balance it out with an infinite moment on the other side.


I should mention that the "counter intuitive" behavior of things like the Cauchy distribution is entirely due to problems when thinking about infinity. Take the Cauchy distribution and chop off the tails - even arbitrarily far out, like at plus/minus the xkcd number - and (once re-normalized) you suddenly get something that's well behaved and has a defined mean. It's not the fat tails in-and-of-themselves that are an issue, it's how those tails behave as you approach infinity.

$\endgroup$
  • $\begingroup$ Nice. I wonder if its possible to give an exlicit "order of summation" that leads to, say, two. $\endgroup$ – Matthew Drury Sep 2 '16 at 20:56
  • $\begingroup$ @MatthewDrury: p_i and n_i denote positive and negative numbers. Successively find p_i and n_i so that integral over [n_i , p_i] is 2+(1/i) and integral over [n_{i+1},p_i] is 2-(1/i). One could do this explicitly using R, matlab or mathematica, but only for a finite number of terms. $\endgroup$ – David Epstein Sep 8 '16 at 10:06
7
$\begingroup$

General Abrial and Glen_b had perfect answers. I just want to add a small demo to show you the mean of Cauchy distribution does not exist / does not converge.

In following experiment, you will see, even you get a large sample and calcluate the empirical mean from the sample, the numbers are quite different from experiment to experiment.

set.seed(0)
par(mfrow=c(1,2))
experiments=rep(1e5,100)
mean_list_cauchy=sapply(experiments, function(n) mean(rcauchy(n)))
mean_list_normal=sapply(experiments, function(n) mean(rnorm(n)))
plot(mean_list_cauchy,ylim=c(-10,10))
plot(mean_list_normal,ylim=c(-10,10))

enter image description here

You can observe that we have $100$ experiments, and in each experiment, we sample $1\times 10^5$ points from two distributions, with such a big sample size, the empirical mean across different experiments should be fairly close to true mean. The results shows Cauchy distribution does not have a converging mean, but normal distribution has.

EDIT:

As @mark999 mentioned in the chat, we should argue the two distributions used in the experiment has similar "variance" (the reason I use quote is because Cauchy distribution variance is also undefined.). Here is the justification: their PDF are similar.

Note that, by looking at the PDF of Cauchy distribution, we would guess it is $0$, but from the experiments we can see, it does not exist. That is the point of the demo.

curve(dnorm, -8,8)
curve(dcauchy, -8,8)

enter image description here

$\endgroup$
  • 4
    $\begingroup$ I don't think this shows that the Cauchy distribution has no mean. You could get similar results if you replaced the Cauchy distribution by a normal distribution with a suitably large variance. $\endgroup$ – mark999 Sep 2 '16 at 11:34
  • $\begingroup$ good point @mark999, I will edit my answer to address this problem. $\endgroup$ – Haitao Du Sep 2 '16 at 11:41
  • $\begingroup$ Is it possible to figure out from PDF of Cauchy distribution that it has no mean, probably by looking at it's fat tails ? $\endgroup$ – ks1322 Sep 2 '16 at 13:18
  • $\begingroup$ Perhaps you had something like this in mind? stats.stackexchange.com/questions/90531/… $\endgroup$ – Reinstate Monica Sep 2 '16 at 15:16
2
$\begingroup$

By definition of Lebesgue-Stieltjes integral, the mean exists if:

$$\int \vert x\vert dF(x)<\infty.$$

https://en.wikipedia.org/wiki/Moment_(mathematics)#Significance_of_the_moments

https://en.wikipedia.org/wiki/Lebesgue_integration

$\endgroup$
2
$\begingroup$

The Cauchy distribution is a disguised form of a very fundamental distribution, namely the uniform distribution on a circle. In formulas, the infinitesimal probability is $d\theta/2\pi$, where $\theta$ is the angle coordinate. The probability (or measure) of an arc $A\subset \mathbb S^1$ is $\mathtt{length}(A)/2\pi$. This is different from the uniform distribution $U(-\pi,\pi)$, though measures are indeed the same for arcs not containing $\pi$. For example, on the arc from $\pi-\varepsilon$ counter-clockwise to $-\pi+\varepsilon\ (=\pi+\varepsilon \mod 2\pi)$, the mean of the distribution on the circle is $\pi$. But the mean of the uniform distribution $U(-\pi,\pi)$ on the corresponding union of two disjoint intervals, each of length $\varepsilon/2\pi$, is zero.

Since the distribution on the circle is rotationally symmetric, there cannot be a mean, median or mode on the circle. Similarly, higher moments, such as variance, cannot make sense. This distribution arises naturally in many contexts. For example, my current project involves microscope images of cancerous tissue. The very numerous objects in the image are not symmetric and a "direction" can be assigned to each. The obvious null hypothesis is that these directions are uniformly distributed.

To disguise the simplicity, let $\mathbb S^1$ be the standard unit circle, and let $p=(0,1)\in\mathbb S^1$. We define $x$ as a function of $\theta$ by stereographical projection of the circle from $p$ onto the $x$-axis. The formula is $x=\tan(\theta/2)$. Differentiating, we find $d\theta/2 = dx/(1+x^2)$. The infinitesimal probability is therefore $\frac{d\theta}{\pi(1+x^2)}$, the usual form of the Cauchy distribution, and "Hey, presto!", simplicity becomes a headache, requiring treatment by the subtleties of integration theory.

In $\mathbb S^1 \setminus \{p\}$, we can ignore the absence of $p$ (in other words, reinstate $p\in\mathbb S^1$) for any consideration such as mean or higher order moment, because the probability of $p$ (its measure) is zero. So therefore the non-existence of mean and of higher moments carries over to the real line. However, there is now a special point, namely $-p = (0,-1)$, which maps to $0\in\mathbb R$ under stereographic projection and this becomes the median and mode of the Cauchy distribution.

$\endgroup$
  • 2
    $\begingroup$ The Cauchy distribution has a median and mode. $\endgroup$ – jkabrg Sep 9 '16 at 12:51
  • $\begingroup$ quite right. I got a bit carried away. But the argument for the non-existence of the mean is correct.. I will edit my answer. $\endgroup$ – David Epstein Sep 10 '16 at 13:01
  • $\begingroup$ Why is it that "there cannot be a mean because there isn't one on the circle"? There's a lot missing in your argument. I'm assuming what you mean by it being the uniform distribution "on the circle" is that $\theta \sim U(-\pi,\pi)$ and $X = \tan(\theta/2)$, but then $\mathbb E[\theta] = 0$ so I don't understand what you're talking about. $\endgroup$ – jkabrg Sep 10 '16 at 13:41
  • $\begingroup$ @jkabrg: I hope the new edits make this more comprehensible $\endgroup$ – David Epstein Sep 10 '16 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.