$X_i, X_j$ independent when $i≠j$, but $X_1, X_2, X_3$ dependent?

Question

I've seen the statement:

It's possible that random variables $X_i, X_j$ are independent for $i≠j$, but $X_1, X_2, X_3$ are dependent.

I haven't been able to find examples of this though.

Any examples?

Answer 1

Here is an example of this, attributed to S. Benstein.

Let $X_1, X_2, X_3$ have the joint pmf

$$p\left(x_1, x_2, x_3 \right) =\begin{cases} \frac{1}{4} & \left(x_1, x_2, x_3 \right) \in \left\{ (1,0,0), (0,1,0), (0,0,1), (1,1,1) \right\} \\ 0 & \text{otherwise} \end{cases}$$

Then by summing out the third variable it is easy to see that the joint pmf of $X_i$ and $X_j$, $i\neq j$ is

$$p_{ij} (x_i, x_j)= \begin{cases} \frac{1}{4} & (x_i, x_j) \in \left\{ (0,0), (1,0), (0,1), (1,1) \right\} \\ 0 & \text{otherwise} \end{cases} $$

Finally, the marginal pmf of $X_i$ is

$$p_i(x_i) = \begin{cases} \frac{1}{2} & x_i= 0, 1 \\ 0 & \text{otherwise} \end{cases}$$

Now, note that for $i\neq j$

$$p_{ij} (x_i, x_j) =p_i (x_i) p_j (x_j)$$

and thus $X_i$ and $X_j$ are independent. However

$$p(x_1, x_2, x_3) \neq p_1 (x_1) p_2 (x_2) p_3 (x_3)$$

and so $X_1, X_2, X_3$ are not independent. Thus pairwise independence does not imply mutual independence. The latter is a stronger condition and it's usually the one we use with random samples.

Answer 2

$X$, $Y$ independent Bernoulli$(\frac 12)$ and $Z= X+Y-2XY$ is an example of three random variables that are pairwise independent but not mutually independent.

It is easy to show that $Z$ is also Bernoulli$(\frac 12)$ and that $(X,Z)$ and $(Y,Z)$ are pairs of independent random variables, and of course, $(X,Y)$ is a pair of independent random variables by assumption. (If you feel too lazy to carry this out for yourself, note that the answer by @JohnK essentially uses $X_1=X, X_2=Y, X_3 = 1-Z$). Thus, $X,Y,Z$ are said to be pairwise independent random variables. However, for $X,Y,Z$ to be called mutually independent random variables, their joint probability mass function must factor into the product of the individual (marginal) probability mass functions, that is, if $X, Y, Z$ take on values in the sets $\{x_i\}, \{y_j\}, \{z_k\}$ respectively, then

$X,Y,Z$ are said to be mutually independent random variables if for all choices of $x_i, y_j, z_k$, $$P\{X=x_i, Y=y_j, Z = z_k\} = P\{X=x_i\}P\{Y = y_j\}P\{Z = z_k\}.$$

In the example above, it is easy to verify that $$P\{X=1,Y=1,Z=1\} = 0 \neq \frac 18 = P\{X=1\}P\{Y=1\}P\{Z=1\}$$ and so $X,Y,Z$ cannot be called mutually independent random variables.

---

Lest you think that it is necessary to use discrete random variables to have examples such as the one above, consider three standard normal random variables $X,Y,Z$ whose joint probability density function $f_{X,Y,Z}(x,y,z)$ is not $\phi(x)\phi(y)\phi(z)$ where $\phi(\cdot)$ is the standard normal density, but rather

$$f_{X,Y,Z}(x,y,z) = \begin{cases} 2\phi(x)\phi(y)\phi(z) & ~~~~\text{if}~ x \geq 0, y\geq 0, z \geq 0,\\ & \text{or if}~ x < 0, y < 0, z \geq 0,\\ & \text{or if}~ x < 0, y\geq 0, z < 0,\\ & \text{or if}~ x \geq 0, y< 0, z < 0,\\ 0 & \text{otherwise.} \end{cases}\tag{1}$$ Note that $X$, $Y$, and $Z$ are not a set of three jointly normal random variables but as will be described below, any two of these is indeed a pair of independent normal random variables.

We can calculate the joint density of any pair of the random variables, (say $X$ and $Z$) by integrating out the joint density with respect to the unwanted variable, that is, $$f_{X,Z}(x,z) = \int_{-\infty}^\infty f_{X,Y,Z}(x,y,z)\,\mathrm dy. \tag{2}$$

• If $x \geq 0, z \geq 0$ or if $x < 0, z < 0$, then $f_{X,Y,Z}(x,y,z) = \begin{cases} 2\phi(x)\phi(y)\phi(z), & y \geq 0,\\ 0, & y < 0,\end{cases}$ and so $(2)$ reduces to $$f_{X,Z}(x,z) = \phi(x)\phi(z)\int_{0}^\infty 2\phi(y)\,\mathrm dy = \phi(x)\phi(z). \tag{3}$$

• If $x \geq 0, z < 0$ or if $x < 0, z \geq 0$, then $f_{X,Y,Z}(x,y,z) = \begin{cases} 2\phi(x)\phi(y)\phi(z), & y < 0,\\ 0, & y \geq 0,\end{cases}$ and so $(2)$ reduces to $$f_{X,Z}(x,z) = \phi(x)\phi(z)\int_{-\infty}^0 2\phi(y)\,\mathrm dy = \phi(x)\phi(z). \tag{4}$$

In short, $(3)$ and $(4)$ show that $f_{X,Z}(x,z) = \phi(x)\phi(z)$ for all $x, z \in (-\infty,\infty)$ and so $X$ and $Z$ are (pairwise) independent standard normal random variables. Similar calculations (left as an exercise for the bemused reader) show that $X$ and $Y$ are (pairwise) independent standard normal random variables, and $Y$ and $Z$ also are (pairwise) independent standard normal random variables. But $X,Y,Z$ are not mutually independent normal random variables. Nor are the three of them together a set of jointly normal random variables. Indeed, their joint density $f_{X,Y,Z}(x,y,z)$ does not equal the product $\phi(x)\phi(y)\phi(z)$ of their marginal densities for any choice of $x, y, z \in (-\infty,\infty)$

Answer 3

One that's perhaps easier to think about comes from a chessboard. Pick a point uniformly on the chessboard and consider

$X_1$: row number (1-8) modulo 2 $X_2$: column number (1-8) modulo 2 $X_3$: color, 0 for white,1 for black.

It's easy to see than any pair of these is independent: rows are independent of columns; row is independent of colour. But if you know $X_1$ and $X_2$, you know $X_3$ exactly.