Having read a little about exchangeability, I went back to thinking about the iid condition required for the central limit theorem. It struck me that if two random variables are drawn from an identical distribution, the occurrence of one event does not make it more or less probable for the occurrence of another event. I have no doubts that I am wrong and there is a reason why both independence and identical distribution are required. I just don't know why! Thanks very much!

  • 6
    Well, $X$ and ... $X$ are identically distributed. – NRH Nov 2 '11 at 22:33
  • 2
    There are various forms of the central limit theorem. For example, identical distribution is not required if the independent random variables are uniformly bounded and $\sum_{i=1}^\infty \sigma_i^2 = \infty$, and mild dependence is also permissible. See, for example, the Wikipedia page on the central limit theorem. – Dilip Sarwate Nov 2 '11 at 23:18
up vote 19 down vote accepted

If random variables are drawn from an identical distribution, why doesn't this guarantee that they are independent?

Since you don't specify how the random variables are drawn, the question has no meaning. It is the manner of drawing that is important. Consider a neoclassical example of an urn with one ball marked $0$ and one ball marked $1$. The random variable $X$ is the number on the ball drawn from the urn, and is a Bernoulli random variable with parameter $p = P\{X = 1\} = \frac{1}{2}$. Now let $X_1$ denote the number on the first ball drawn and $X_2$ the number on the second ball drawn.

  • Case I: drawing with replacement There are 4 equally likely outcomes of the experiment and they can be written as $00, 01, 10, 11$. $X_1$ and $X_2$ are both Bernoulli$\left(\frac{1}{2}\right)$ random variables and are independent.
  • Case II: drawing without replacement Now there are only two equally likely outcomes $01, 10$ but $X_1$ and $X_2$ clearly are Bernoulli$\left(\frac{1}{2}\right)$ random variables just as before, and just as clearly are not independent.

Thus, just getting random variables with identical distribution does not by any means guarantee that they are independent.

  • Thanks very much for the answer. But I am still unclear about one thing. in case II, you say X1, X2 ~ Bernoulli(1/2). But once X1 is drawn, the value of the second draw is now fixed -- so isn't X2 no longer a Bernoulli(1/2) random variable? In which case, X1 and X2 are no longer identically distributed? – HFC Nov 3 '11 at 20:19
  • 2
    Think of the experiment being repeated over and over again, and each time, a student comes in late, never finds out what the experiment consists of or that a first ball has been drawn. He sees each day that the professor draws a ball from the urn, and on roughly half the days the ball bears the number $1$ and the other times it bears the number $0$. He estimates that the ball is equally likely to be numbered $0$ or $1$, right? In Case II, there are two equally likely outcomes $01$ and $10$ and if you look only at the second ball, it is equally likely to bear the number $0$ or $1$. – Dilip Sarwate Nov 3 '11 at 22:08
  • 7
    @HFC Less whimsically and more formally, it is important to not confuse between conditional distributions and unconditional distributions. In Case II, the conditional distribution of $X_2$ given $X_1$ is degenerate, and is not the same as the (unconditional) distribution of $X_1$. But, by the law of total probability, $$P(X_2=1)=P(X_2=1\mid X_1=1)P(X_1=1)+P(X_2=1\mid X_1=0)P(X_1=0)=0\times\frac{1}{2}+1\times\frac{1}{2}=\frac{1}{2}=P(X_1=1)$$ and so the unconditional distribution of $X_2$ is the same as the (unconditional) distribution of $X_1$. – Dilip Sarwate Nov 3 '11 at 23:21
  • Thanks for clearing this up. The explanation using the law of total probability was really helpful. Thanks again! – HFC Nov 5 '11 at 21:08

protected by whuber Oct 16 '12 at 18:24

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.