# When mathematical statistics outsmarts probability theory

This is not a question, but it is too good to pass. I read it is originally due to Enis, Peter. "On the relation $$E (X) = E [E (X∣ Y)]$$." Biometrika 60, no. 2 (1973): 432-433.

Assume $$Y$$ has a Chi-square distribution, with one degree of freedom, so its density is

$$f_Y(y) = \frac{1}{\sqrt{2\pi}}y^{-1/2}\exp\left\{-\frac 12 y\right\},\;\;\; y \in [0,\infty). \tag{1}$$

Suppose now that we define the conditional density of another variable $$X$$ as $$f_{X|Y}(x \mid y) = \frac{1}{\sqrt{2\pi}} y^{1/2}\exp\left\{-\frac 12yx^2\right\}\;\;\; x \in (-\infty, \infty). \tag{2}$$

Namely, conditional on $$Y=y$$, $$X$$ has a zero-mean Normal distribution with variance equal to $$1/y$$. This is perfectly legal and valid. Note that we have $$E(X\mid Y) = 0.$$ Then "automatically", we would conclude $$...\implies E\big[E(X\mid Y)\big] = E(X) = 0...$$

BUT THIS IS NOT THE CASE.

By Bayes theorem for densities, the joint density is $$f_{X,Y}(x,y) = f_{X|Y}(x \mid y)\cdot f_Y(y) = \frac{1}{2\pi}\exp\left\{-\frac 12y(1+x^2)\right\}\;\;\; (x,y) \in (-\infty, \infty) \times (0, \infty). \tag{3}$$

From this we can obtain the marginal density of $$X$$ by $$f_X(x) = \int_0^{\infty}f_{X,Y}(x,y) dy = \int_0^{\infty}\frac{1}{2\pi}\exp\left\{-\frac 12y(1+x^2)\right\}\,dy$$ $$=\frac {1}{\pi}\frac{1}{1+x^2}\int_0^{\infty}\frac{1+x^2}{2}\exp\left\{-\frac 12y(1+x^2)\right\}\,dy.$$ The integral is equal to $$1$$ since it is an Exponential density with rate parameter $$(1+x^2)/2$$, so we obtain $$f_X(x) = \frac {1}{\pi}\frac{1}{1+x^2}, \tag{4}$$

which is the standard Cauchy density, for which $$E(X)$$ is undefined.

So we have obtained that $$E(X\mid Y)$$ can exist and be finite, even when $$E[X]$$ does not exist.

Why has mathematical statistics outsmarted probability theory in this instance?

Because in the latter, in order to define the conditional expectation, we start by assuming a random variable $$X$$ for which $$E(X)$$ exists, and given this, $$E(X\mid Y)$$ is then defined -the "defining property" of the conditional expectation in this approach is exactly that its mean equals the unconditional mean. So if we were told, "let $$X$$ be a standard Cauchy density", we would conclude that "it follows that we cannot define a conditional expectation of it"... But we just saw that this premise of the existence of $$E(X)$$ is not necessary for $$E(X\mid Y)$$ to exist...

...the dark side of this achievement, is that it creates the following additional obligation: whenever we encounter first a conditional expectation, we cannot automatically "average over it" to obtain the unconditional expected value -the latter must be proven to exist by other means.

In other words, the existence of the unconditional expected value is only sufficient for the conditional expectation to exist, and the existence of the conditional expectation is not sufficient for the unconditional expected value to exist.

Ah, and here is a question: Any other such lovely examples?

• Which other examples are you thinking of? Aug 13, 2022 at 8:12
• Probably not a big deal but I believe you could perhaps have framed a question - perhaps in terms of the apparent paradox (e.g. asking if there was case where something like this occurred) and then answered it, thus retaining the Q&A format. Aug 13, 2022 at 10:01
• If you play AI Factory Backgammon on Android (and you play it enough) you'll find that in 8997 occurrences where the cpu is on the bar and has only one point to get in, it massively exceeds the probability. The odds are 11 in 36 of getting back in so 8997*(11/36) = 2749 occasions. The actual times it managed to land its shot are 5188 occasions.....so not 11 in 36 (30.56%) but 19 in 36 so 62.43%. The developer swears their dice are fair....... Aug 13, 2022 at 15:53
• @TimRichards: I am afraid I do not grasp the connection between the question and your comment. Aug 14, 2022 at 8:52
• This could be one more answer to this old Q&A thread on Math Educators SE about "unique candidates that fail". Specifically, from $\mathbb E[X \mid Y] = 0$ we can conclude that $\mathbb E[X]$ must be zero, if it exists. Aug 14, 2022 at 14:17

## 3 Answers

I do not find this any more surprising than saying that if $$Y \sim \mathcal N(0,1)$$ then $$\mathbb E\left[\frac1Y\right]$$ is undefined even though $$\frac1Y$$ has a distribution symmetric about $$0$$.

So let's use this to construct an example using $$Y\sim \mathcal N(0,1)$$:

• let $$X=\pm\frac1Y$$ with equal probability (or $$0$$ in the zero-probability case of $$Y=0$$)
• clearly $$\mathbb E\left[X \mid Y\right]=0$$ so $$\mathbb E\big[\mathbb E\left[X \mid Y\right]\big]=0$$
• but $$X$$ and $$\frac1Y$$ have the same heavy-tailed symmetric distribution so $$\mathbb E\left[X\right]$$ is undefined
• Could there be distributions $X,Y,Z$ such that $\mathbb E[\mathbb E[X|Y]]$ and $\mathbb E[\mathbb E[X|Z]]$ are both defined but distinct? Aug 14, 2022 at 11:06
• @Magma I think that is a different question which you might like to ask. Clearly you would need $\mathbb E[X]$ undefined Aug 14, 2022 at 12:55

While it is a pleasant remark, I do not find this occurrence that surprising or paradoxical, and this for several reasons:

(i) $$\mathbb E^X[0]=0$$ remains true, where $$\mathbb E^X[\cdot]$$ denotes the expectation under the distribution of $$X$$;

(ii) conditional distributions and therefore expectations are only defined with respect to or in terms of a joint distribution, meaning that logically we start from this joint and derive the conditional, rather than the opposite, so logically we do not "encounter first a conditional expectation";

(iii) conditional distributions are usually equipped with lighter tails than marginal ones, hence it is not surprising that the conditional expectation may exist for all realisations of $$Y$$ while the marginal expectation does not exist;

(iv) there is no "defeat of probability theory" there (and even less of a connection with "mathematical statistics"): the law of total expectation states the existence of $$\mathbb E[X]$$ as its main assumption.

• Thank you for your feedback, The whole point of my post was to alert readers that a commonly held belief is not always correct (i.e. the belief being that once we have a conditional expectation, we can obtain the marginal expected value), not to declare that there is a fundamental wrong result in the theory (there isn't). Your remark that we start with the joint distribution is the best way for somebody to gain an intuition as to why this belief could indeed not always be true. But. (CONT'D) Aug 13, 2022 at 11:18
• (CONT"D) ...in practice, users construct conditional distributions in the above way without concerning themselves at all with the joint, and still hold on the assertion discussed. The "defeat" of probability theory (which is evidently a pun) is not that it has made a mistake, but that it has restricted itself by defining conditional expectation as it usually does, requiring the existence of $E(X)$ to avail itself of $E(E(X|Y))$, as I wrote in my post. Aug 13, 2022 at 11:25
• What does $\mathbb E^X[0]$ represent? Aug 13, 2022 at 14:06
• @Henry: the expectation of the random variable $0(X)$, a.s. equal to zero. Aug 13, 2022 at 14:36

Note that we have

$$E(X∣Y)=0.$$

$$E(X|y) = E[X|1] \cdot y$$, which when $$y\to \infty$$ seems to become like a case of the undefined $$0 \times \infty$$.

In a way the example poses the naive statement that a distribution must have $$E[X] = m$$ when it is symmetric about $$m$$. And it does this via the expression $$E (X) = E [E (X∣ Y)]$$ which hides the situation that an undefined term is included.

• I am not sure I understand. The example just shows that the specific conditional mean exists, even though the unconditional mean doesn't. Nov 11, 2022 at 17:12
• @AlecosPapadopoulos the unconditional mean is a sum of conditional means. When one of those conditional means does not exist or is undefined, then there is a problem in the expression of the unconditional mean as a sum of conditional means. Nov 11, 2022 at 20:01
• "The example just shows that the specific conditional mean exists, even though the unconditional mean doesn't." I see this like the example tries to argue the other way around when it uses $$E[E(X|Y)] = E[0] = 0$$ it wrongly states that the conditional mean $E(X|y)=0$ exists for all values of $y \in [0,\infty]$ Nov 11, 2022 at 20:05
• But this is exactly the point of the post. That the expression "take the expected value of a conditional mean" is not automatically valid. Nov 12, 2022 at 14:17
• @AlecosPapadopoulos Why do we need this construction with an example of the Cauchy distribution for this? The probability theory can already know from the start that it is not valid. Nov 12, 2022 at 14:20