How would you go about explaining i.i.d (independent and identically distributed) to non-technical people?

migrated from quant.stackexchange.com Oct 21 '11 at 18:42

This question came from our site for finance professionals and academics.

up vote 47 down vote accepted

It means "Independent and identically distributed".

A good example is a succession of throws of a fair coin: The coin has no memory, so all the trows are "independent".

And every throw is 50:50 (heads:tails), so the coin is and stays fair - the distribution from which every throw is drawn, so to speak, is and stays the same: "identically distributed".

A good starting point would be the Wikipedia page.

  • 8
    I wonder if the coin toss example would falsely give the impression that every event must be equiprobable... – Michael McGowan Oct 21 '11 at 18:57
  • So, is it not necessary that the IID random variables should be equi-probable? if they are not equiprobable then how can the "identically distributed" be explained? Thanks a lot in advance... – user20163 Jan 27 '13 at 20:48
  • 5
    @Nalini "equi-probable" is not a synonym for "identically distributed." If $x$ and $y$ are i.i.d., this means they are taken from the same distribution, not that all values in that distribution are equally likely (think the normal distribution). $x$ and $y$ would have the same expected value, though. – Jason Morgan Jan 27 '13 at 21:07
  • If two variables are independent and normal distributed but with different mean and variance, are they still i.i.d? – spurra Apr 13 '15 at 11:29

Nontechnical explanation:

Independence is a very general notion. Two events are said to be independent if the occurrence of one does not give you any information as to whether the other event occurred or not. In particular, the probability that we ascribe to the second event is not affected by the knowledge that the first event has occurred.

  • Example of independent events, possibly identically distributed
    Consider tossing two different coins one after the other. Assuming that your thumb did not get unduly tired when it flipped the first coin, it is reasonable to assume that knowing that the first coin toss resulted in Heads in no way influences what you think the probability of Heads on the second toss is. The two events $$\{\text{first coin toss resulted in Heads}\}~~\text{and}~~\{\text{second coin toss resulted in Heads}\}$$ are said to be independent events.

    • If we know, or obstinately insist, that the two coins have different probabilities of resulting in Heads, then the events are not identically distributed.

    • If we know or assume that the two coins have the same probability $p$ of coming up Heads, then the above events are also identically distributed, meaning that they both have the same probability $p$ of occurring. But notice that unless $p = \frac 12$, the probability of Heads does not equal the probability of Tails. As noted in one of the Comments, "identical distribution" is not the same as "equally probable."

  • Example of identically distributed nonindependent events
    Consider an urn with two balls in it, one black and one white. We reach into it and draw out the two balls one after the other, choosing the first one at random (and this of course determines the color of the next ball). Thus, the two equally likely outcomes of the experiment are (White, Black) and (Black, White), and we see that the first ball is equally likely to be Black or White and so is the second ball also equally likely to be Black or White. In other words, the events $$\{\text{first ball drawn is Black}\}~~\text{and}~~\{\text{second ball drawn is Black}\}$$ certainly are identically distributed, but they are definitely not independent events. Indeed, if we know that the first event has occurred, we know for sure that the second cannot occur. Thus, while our initial evaluation of the probability of the second event is $\frac 12$, once we know that the first event has occurred, we had best revise our assessment of the probability of the second drawn will be black from $\frac 12$ to $0$.

  • "As noted in one of the Comments, "identical distribution" is not the same as "equally probable."" What's the difference? "equally probable" means heads is equally likely as tails? Whereas "identically distributed" means each event has the same likelihood of heads? – The Red Pea Aug 23 '16 at 13:56
  • 3
    @TheRedPea Not quite. If we have a biased coin that turns up H with probability $p \ne \frac 12$, then the events "First Toss is H" and the event "Second Toss is H" are independent as well as equally probable (they both have probability $p$). Furthermore the tosses are identically distributed: they both have the same probabilities ($p$ and $1-p$ for H and T respectively) in the various tosses. But the events "First Toss is H" and "First Toss is T" are not equally probable. Nor are they independent. Identical distribution = all tosses have same distribution. – Dilip Sarwate Aug 23 '16 at 16:08
  • 1
    @TheRedPea (continued) Equally probable means two events have the same probability. The events may be defined across tosses as above, or within a single experiment. The canonical simple model of an experiment is of a sample space with $n$ outcomes in which all the outcomes have the same probability $\frac 1n$. It is common to describe this by saying "a fair coin" or "a fair dice" or saying things like "A ball is chosen at random from an urn with 3 green balls and 2 red balls" etc. It is only the purists who will cavil and insist that it should be "a fair die" .... – Dilip Sarwate Aug 23 '16 at 16:17
  • OK so identical distribution refers to the entire probability distribution, whereas equal probability refers to parts of that probability distribution. I understand now, thank you. – The Red Pea Aug 23 '16 at 18:50
  • I'm not sure about the last example being identically distributed. Is it arguable that "if two events are not independent, they can not be from identical distributions"? E.g. in your example I'd say the second ball-drawing has a different distribution due to the first event. – jiggunjer Sep 5 '16 at 5:59

A random variable is variable which contains the probability of all possible events in a scenario. For example, lets create a random variable which represents the number of heads in 100 coin tosses. The random variable will contain the probability of getting 1 heads, 2 heads, 3 heads.....all the way to 100 heads. Lets call this random variable X.

If you have two random variables then they are IID (independent identically distributed) if:

  1. If they are independent. As explained above independence means the occurrence of one event does not provide any information about the other event. For example, if I get 100 heads after 100 flips, the probabilities of getting heads or tails in the next flip are the same.
  2. If each random variable shares the same distribution. For example, lets take the random variable from above - X. Lets say X represents Obama about to flip a coin 100 times. Now let's say Y represents a Priest about to flip a coin 100 times. If Obama and the Priest flip coins with the same probability of landing on heads, then X and Y are considered identically distributed. If we sample repeatedly from either the Priest or Obama, then the samples are considered identically distributed.

Side note: Independence also means you can multiply probabilities. Lets say the probability of heads is p, then the probability of getting two heads in a row is p*p, or p^2.

That two dependent variables can have the same distribution can be shown with this example:

Assume two successive experiments involving each 100 tosses of a biased coin, where the total number of Head is modeled as a random variable X1 for the first experiment and X2 for the second experiment. X1 and X2 are binomial random variables with parameters 100 and p, where p the bias of the coin.
As such, they are identically distributed. However they are not independent, since the value of the former is quite informative about the value of the latter. That is if the result of the first experiment is 100 Heads this tells us a lot about the bias of the coin and therefore gives us a lot new information regarding the distribution of X2.
Still X2 and X1 are identically distributed since they are derived from the same coin.

What is also true is that if 2 random variables are dependent then the posterior of X2 given X1 will never be the same as the prior of X2 and vice versa. While when X1 and X2 are independent their posteriors are equal to their priors. Therefore, when two variables are dependent, the observation of one of them results in revised estimates regarding the distribution of the second. Still both may be from the same distribution, it is just we learn in the process more about the nature of this distribution. So returning to the coin tosses experiments, initially in the absence of any information we might assume that X1 and X2 follow a Binomial distribution with parameters 100 and 0.5. But after observing 100 Heads on a row we would certainly revise our estimate about the p parameter to make it quite close to 1.

An aggregation of several random draws from the same distribution. An example being pulling a marble out of bag 10,000 times and counting the times you pull the red marble out.

  • Can you expand on how this adds to the existing answers? – mdewey Dec 4 '16 at 10:12

If a random variable $X$ comes from a population having (say) a normal distribution, that is its pdf (probability density function) is that of normal distribution, with a population average $\mu=3$ and population variance $\sigma^2=4$ (the numbers are hypothetical and are just for your understanding and to simplify comparisons) we can describe it as follows: $X \sim N(3 , 4)$.

Now if we have another random variable $Y$ which is also normally distributed and which is $Y \sim N(3, 4)$ then $X$ and $Y$ are identically distributed.

Nevertheless, being identically distributed does not necessarily imply independence.

  • 5
    You must have an interesting set of "nontechnical people" in mind when you rely on technical terms like "random variable," "normal distribution," "pdf," "variance," and "independence." I would venture to say it's the empty set. – whuber Dec 18 '15 at 20:27
  • "being identically distributed does not necessarily imply independence". How can dependence have an effect on two identically distributed variables? It would seem to me, that dependence causes non-identicality, but not all non-identicality is due to dependence. – jiggunjer Sep 5 '16 at 6:05

protected by Dilip Sarwate Dec 4 '16 at 14:34

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.