What makes the mean of some distributions undefined?

Question

Many PDFs range from minus to positive infinity, yet some means are defined and some are not. What common trait makes some computable?

Answer 1

The mean of a distribution is defined in terms of an integral (I'll write it as if for a continuous distribution - as a Riemann integral, say - but the issue applies more generally; we can proceed to Stieltjes or Lebesgue integration to deal with these properly and all at once):

$$E(X) = \int_{-\infty}^\infty x f(x)\, dx$$

But what does that mean? It's effectively a shorthand for

$$\stackrel{\lim}{_{a\to\infty,b\to\infty}} \int_{-a}^b x\, f(x)\, dx$$

or

$$\stackrel{\lim}{_{a\to\infty}} \int_{-a}^0 x f(x)\, dx \, +\, \stackrel{\lim}{_{b\to\infty}} \int_{0}^b x f(x)\, dx$$

(though you could break it anywhere not just at 0)

The problem comes when the limits of those integrals are not finite.

So for example, consider the standard Cauchy density, which is proportional to $\frac{1}{1+x^2}$ ... note that

$$\stackrel{\lim}{_{b\to\infty}} \int_{0}^b \frac{x}{1+x^2}\, dx$$

let $u=1+x^2$, so $du=2x\,dx$

$$=\,\stackrel{\lim}{_{b\to\infty}}\frac12 \int_{1}^{1+b^2} \frac{1}{u}\, du$$

$$=\,\stackrel{\lim}{_{b\to\infty}} \frac{_1}{^2}\ln(u)\Bigg |_{1}^{1+b^2} $$

$$=\,\stackrel{\lim}{_{b\to\infty}} \frac{_1}{^2}\ln(1+b^2)$$

which isn't finite. The limit in the lower half is also not finite; the expectation is thereby undefined.

Or if we had as our random variable the absolute value of a standard Cauchy, its entire expectation would be proportional to that limit we just looked at (i.e. $\stackrel{\lim}{_{b\to\infty}} \frac12\ln(1+b^2)$).

On the other hand, some other densities do continue out "to infinity" but their integral does have a limit.

Answer 2

The other answers are good, but might not convince everyone, especially people who take one look at the Cauchy distribution (with $x_0 = 0$) and say it's still intuitively obvious that the mean should be zero.

The reason the intuitive answer is not correct from the mathematical perspective is due to the Riemann rearrangement theorem (video).

Effectively what you're doing when you're looking at a Cauchy and saying that the mean "should be zero" is that you're splitting down the "center" at zero, and then claiming the moments of the two sizes balance. Or in other words, you're implicitly doing an infinite sum with "half" the terms positive (the moments at each point to the right) and "half" the terms negative (the moments at each point to the left) and claiming it sums to zero. (For the technically minded: $\int_{0}^\infty f(x_0+r)r\, dr - \int_{0}^{\infty} f(x_0-r)r\, dr = 0$)

The Riemann rearrangement theorem says that this type of infinite sum (one with both positive and negative terms) is only consistent if the two series (positive terms only and negative terms only) are each convergent when taken independently. If both sides (positive and negative) are divergent on their own, then you can come up with an order of summation of the terms such that it sums to any number. (Video above, starting at 6:50)

So, yes, if you do the summation in a balanced manner from 0 out, the first moments from the Cauchy distribution cancel out. However, the (standard) definition of mean doesn't enforce this order of summation. You should be able to sum the moments in any order and have it be equally valid. Therefore, the mean of the Cauchy distribution is undefined - by judiciously choosing how you sum the moments, you can make them "balance" (or not) at practically any point.

So to make the mean of a distribution defined, the two moment integrals need to each be independently convergent (finite) around the proposed mean (which, when you do the math, is really just another way of saying that the full integral ($\int_{-\infty}^\infty f(x)x\, dx$) needs to be convergent). If the tails are "fat" enough to make the moment for one side infinite, you're done. You can't balance it out with an infinite moment on the other side.

---

I should mention that the "counter intuitive" behavior of things like the Cauchy distribution is entirely due to problems when thinking about infinity. Take the Cauchy distribution and chop off the tails - even arbitrarily far out, like at plus/minus the xkcd number - and (once re-normalized) you suddenly get something that's well behaved and has a defined mean. It's not the fat tails in-and-of-themselves that are an issue, it's how those tails behave as you approach infinity.

Answer 3

General Abrial and Glen_b had perfect answers. I just want to add a small demo to show you the mean of Cauchy distribution does not exist / does not converge.

In following experiment, you will see, even you get a large sample and calcluate the empirical mean from the sample, the numbers are quite different from experiment to experiment.

set.seed(0)
par(mfrow=c(1,2))
experiments=rep(1e5,100)
mean_list_cauchy=sapply(experiments, function(n) mean(rcauchy(n)))
mean_list_normal=sapply(experiments, function(n) mean(rnorm(n)))
plot(mean_list_cauchy,ylim=c(-10,10))
plot(mean_list_normal,ylim=c(-10,10))

You can observe that we have $100$ experiments, and in each experiment, we sample $1\times 10^5$ points from two distributions, with such a big sample size, the empirical mean across different experiments should be fairly close to true mean. The results shows Cauchy distribution does not have a converging mean, but normal distribution has.

EDIT:

As @mark999 mentioned in the chat, we should argue the two distributions used in the experiment has similar "variance" (the reason I use quote is because Cauchy distribution variance is also undefined.). Here is the justification: their PDF are similar.

Note that, by looking at the PDF of Cauchy distribution, we would guess it is $0$, but from the experiments we can see, it does not exist. That is the point of the demo.

curve(dnorm, -8,8)
curve(dcauchy, -8,8)

Answer 4

The Cauchy distribution is a disguised form of a very fundamental distribution, namely the uniform distribution on a circle. In formulas, the infinitesimal probability is $d\theta/2\pi$, where $\theta$ is the angle coordinate. The probability (or measure) of an arc $A\subset \mathbb S^1$ is $\mathtt{length}(A)/2\pi$. This is different from the uniform distribution $U(-\pi,\pi)$, though measures are indeed the same for arcs not containing $\pi$. For example, on the arc from $\pi-\varepsilon$ counter-clockwise to $-\pi+\varepsilon\ (=\pi+\varepsilon \mod 2\pi)$, the mean of the distribution on the circle is $\pi$. But the mean of the uniform distribution $U(-\pi,\pi)$ on the corresponding union of two disjoint intervals, each of length $\varepsilon/2\pi$, is zero.

Since the distribution on the circle is rotationally symmetric, there cannot be a mean, median or mode on the circle. Similarly, higher moments, such as variance, cannot make sense. This distribution arises naturally in many contexts. For example, my current project involves microscope images of cancerous tissue. The very numerous objects in the image are not symmetric and a "direction" can be assigned to each. The obvious null hypothesis is that these directions are uniformly distributed.

To disguise the simplicity, let $\mathbb S^1$ be the standard unit circle, and let $p=(0,1)\in\mathbb S^1$. We define $x$ as a function of $\theta$ by stereographical projection of the circle from $p$ onto the $x$-axis. The formula is $x=\tan(\theta/2)$. Differentiating, we find $d\theta/2 = dx/(1+x^2)$. The infinitesimal probability is therefore $\frac{d\theta}{\pi(1+x^2)}$, the usual form of the Cauchy distribution, and "Hey, presto!", simplicity becomes a headache, requiring treatment by the subtleties of integration theory.

In $\mathbb S^1 \setminus \{p\}$, we can ignore the absence of $p$ (in other words, reinstate $p\in\mathbb S^1$) for any consideration such as mean or higher order moment, because the probability of $p$ (its measure) is zero. So therefore the non-existence of mean and of higher moments carries over to the real line. However, there is now a special point, namely $-p = (0,-1)$, which maps to $0\in\mathbb R$ under stereographic projection and this becomes the median and mode of the Cauchy distribution.

Answer 5

By definition of Lebesgue-Stieltjes integral, the mean exists if:

$$\int \vert x\vert dF(x)<\infty.$$

https://en.wikipedia.org/wiki/Moment_(mathematics)#Significance_of_the_moments

https://en.wikipedia.org/wiki/Lebesgue_integration

Answer 6

It helps to think about such questions from a more abstract point of view.

When we are talking about random variables we implicitly assume the existence of probability space $(\Omega, \mathcal{F}, \mathbb{P})$. Then a random variable $X: \Omega \to \mathbb{R}$ is simply a measurable function in $(\mathbb{R}, \mathcal{B})$. $X$ induces a measure $\mu$ on the measurable space $(\mathbb{R}, \mathcal{B})$ called the pushforward measure defined by $\mu = \mathbb{P} \circ X^{-1}$. The pdf of $X$ exists if $\mu$ is absolutely continuous with respect to the Lebesgue measure on $(\mathbb{R}, \mathcal{B})$. The pdf of $X$ is a function $f : \mathbb{R} \to \mathbb{R}$ and its mean exists if $\mathbb{E}(X) < \infty$ where this expectation is the integral of $X$ with respect to $(\Omega, \mathcal{F}, \mathbb{P})$. Using the law of unconscious statistician we can write $$ \mathbb{E}(X) = \int_{\mathbb{R}} x f(x) dx $$ Check out the construction of Lebesgue integral for more details about when $\mathbb{E}(X) < \infty$.