6
$\begingroup$

I am confused in understanding the formal definition for independence and how does this differ from the law of total expectations.

I thought that $E[u|x]=E[u]$ formally means independence of $u$ and $x$.

However, when I have $E[u|x]=a$ (const) and use the law of total expectations,

$E[u]=E[E[u|x]]=a$

I obtain $E[u]=a$

Don't I have then $a=E[u|x]$ and $a=E[u] \implies a=E[u|x]=E[u] \implies \text{independence between x and u}$?

Something does not add up, as this would mean that we can always establish independence between two variables. Where is my error in thinking?

$\endgroup$
5
$\begingroup$

It's impossible to read minds, so to locate any error in thinking let's help you work through as simple an example as possible using the most basic possible definitions and axioms of probability.

Consider a sample space $\Omega = \{au, bu, cu, av, bv, cv\}$ ("$au$" etc. are just names of six abstract things) where all subsets are considered measurable. Define a probability $\mathbb{P}$ on $\Omega$ in terms of its values on the atoms via

$$\mathbb{P}(au) = \mathbb{P}(cu) = p,\ \mathbb{P}(bu) = r-2p;\quad \mathbb{P}(av) = \mathbb{P}(cv) = q,\ \mathbb{P}(bv) = 1-r-2q$$

where $p,q,r$ are any numbers for which all six probabilities are positive. For instance, we may take $p=1/6$, $q=1/8$, and $r=1/2$. Because the sum of the six given probabilities is unity, this defines a valid probability measure.

Define the random variables $U$ and $X$ as

$$U(\omega)=-1, 0, 1$$

depending on whether the initial letter in the name of $\omega$ is $a$, $b$, or $c$ respectively; and

$$X(\omega) = 0,1$$

depending on whether the final letter in the name of $\omega$ is $u$ or $v$ respectively.

This can be neatly summarized in a $3$ by $2$ table of probabilities, headed by values of $U$ and $X$, whose interpretation I trust is evident:

$$\begin{array}{r|cc} & \text{X=0} & \text{X=1} \\ \hline \text{U=-1} & p & q \\ \text{U=0} & r-2 p & 1-r-2 q \\ \text{U=1} & p & q \end{array}$$

It is then easy to compute the following:

  1. $\mathbb{P}(X=0) = \mathbb{P}(\{au,bu,cu\}) = p + r-2p + p = r$ (sum the left column in the table).

  2. $\mathbb{P}(U=-1) = \mathbb{P}(\{au,av\}) = p+q$ (sum the top row in the table).

Independence means nothing other than probabilities multiply. This would imply, among other things, that

$$p = \mathbb{P}(au) = \mathbb{P}(X=0, U=-1) = \mathbb{P}(X=0)\mathbb{P}(U=-1) = r(p+q)$$

(investigate the top left entry in the table). But this is rarely the case; for instance, $1/6 \ne (1/2)(1/6 + 1/8)$. Therefore $X$ and $U$ are not independent (except for some special possible combinations of $p$, $q$, and $r$). However,

$$\mathbb{E}(U) = \mathbb{P}(U=-1)(-1) + \mathbb{P}(U=0)(0) + \mathbb{P}(U=1)(1) = 0$$

That is, the expectation of $U$ is zero. (This should be obvious from the symmetry: the chance that $U=1$ balances the chance that $U=-1$ regardless of the value of $X$.) However,

$$\mathbb{E}(U|X=0) = \mathbb{P}(U=-1, X=0)(-1) + \cdots + \mathbb{P}(U=1, X=0)(1) = 0,$$

and similarly $$\mathbb{E}(U|X=1) = 0.$$ That is, the conditional expectation of $U$--which is a function--has the constant value zero. In symbols, we have found that

$$\mathbb{E}(U|X) = 0 = \mathbb{E}(U)$$

and we have a simple, explicit example showing why constant conditional expectation does not imply independence.

$\endgroup$
14
  • $\begingroup$ (+1) Nice answer. A minor typo: in the computation of $\mathbb{E}(U\mid X=0)$, it should be $P(U=-1\mid X = 0)$, etc (the conditional, not the joint). Doesn't change the result, of course. $\endgroup$ – Zen Mar 22 '13 at 16:55
  • 1
    $\begingroup$ Thanks @Zen: you are correct that the conditional notation should be used at first and then that should be equated with the joint probabilities. $\endgroup$ – whuber Mar 22 '13 at 17:16
  • 3
    $\begingroup$ (+1, awhile ago) My favorite example of this is $(X,Y)$ distributed uniform on $\{(1,0), (0,1), (-1,0), (0,-1)\}$. In that case you get $\mathbb E(X \mid Y) = \mathbb E(Y \mid X) = 0 = \mathbb E X = \mathbb E Y$ but $X$ and $Y$ are not independent. $\endgroup$ – cardinal Mar 22 '13 at 17:57
  • 1
    $\begingroup$ @Cardinal I began with that one but somehow--through some error--felt early on that I could not make a counterexample of it. What you see here was the result of the next step. Sorry about the extra complexity... $\endgroup$ – whuber Mar 22 '13 at 19:51
  • 1
    $\begingroup$ ...also, no need to apologize for the "extra complexity". I'm glad it's there because it shows that simple one-off examples like the one I mentioned aren't just parlor tricks. $\endgroup$ – cardinal Mar 22 '13 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.