# What does it mean by independently and identically distributed random variables? [duplicate]

A coin toss is referred as IID in several websites. What I want to know that if I'm understanding the concept right.

If X denote a random variable which means the "result of a coin toss" then $x_1, x_2, ... , x_n$ are the results of repeated coin tossing. Are $x_1, x_2, ... , x_n$ IID? If yes, how can observations be variables?

Or are $X_1, X_2, ... , X_n$ all considered as random variables if we want to consider them IID? How can a random sample be IID?

• An "observation" consists of a unit of data that has been collected. A "random variable" is a theoretical construct used to reason about data. For more about these issues, please search our site for relevant keywords such as "random variable" and "independent."
– whuber
Dec 30, 2016 at 17:19
• I echo what Bill Huber has said. In your case each Xi is a Bernoulli random variable with parameter p=1/2 and the collection of variables X1 through Xn are statistically independent (look up the formal definition) and each have the same Bernoulli distribution. Also the random variance defined as the sum of these n Bernoulli variables is a random variable with a binomial distribution denoted Bin(n,1/2). Dec 30, 2016 at 17:32
• The edit does not change the validity of the above comments. Note that x represents an observation and X denotes a random variable. These are two separate things. Dec 30, 2016 at 17:35
• Please explain the IID thing. I'm so much confused. Dec 30, 2016 at 17:38
• I just told you told you what identically distributed means and Taylor has defined independence. Bill Huber and I both think that you should read about it yourself. Dec 30, 2016 at 17:47

(note: I think OP's issue has more to do with confusing the concepts behind probability theory and random variables than with what "i.i.d." really means. So I'll try to give an informal explanation. For the math of i.i.d., @Taylor's response is quite on point.)

Random variables are abstractions that you use to represent non-determinstic problems, not coins, dice, or other "random objects". So there are no strict rules saying that one coin toss is $X$, two coin tosses are $X_1$ and $X_2$, etc.

The interesting thing about modeling a coin as one single variable $X$ is that it creates an abstraction for all coins, or all possible realizations of a coin tosse. As long as your coin is fair, any time you toss it, you can think of a variable $X$ that follows a Bernoulli distribution.

If you are studying a succession of coin tosses, you could come up with the most contrived ways of modeling it. For instance, you could think of a random variable $Y$ that represents all possible outcomes of a sequence of $n$ coin tosses. Or you could do something even weirder, such as a random variable $Y_3$ for the combined outcomes of the first 3 tosses, and another random variable $Y_r$ for the combined outcomes of the remaining tosses.

But that would be silly. Why? Because there are two interesting things about tossing a coin $n$ times:

1. each toss does not affect the others (independence);
2. all tosses are identical -- the chance of heads coming up in each toss is always 0.5 (identical distributions).

Or are $X_1$, $X_2$,..., $X_n$ all considered as random variables if we want to consider them IID?

You don't "consider" them random variables. There are no such things as $X_1$, $X_2$,..., $X_n$ in the real world and nothing saying that you must think of $X_1$, $X_2$,..., $X_n$ when you think of $n$ coin tosses.

However, when you think of those $n$ variables and their probability distributions, you get to say that they are all i.i.d. because of properties 1 and 2 above. And in probability theory there's all sorts of useful math that other people discovered you can use when you have a collection of i.i.d. variables.

To make it clear, these variables are independent because they represent independent events (each $X_i$ is related to a different, independent toss). And they are identically distributed because they all follow the same distribution: the Bernoulli probability distribution with $p=0.5$.

So what happened to $X$? Nothing. Notice that you invented those random variables $X_1$, $X_2$,..., $X_n$ because tossing a coin $n$ times is a different problem than tossing a coin a single time. The interesting thing about $X$ is that each variable $X_i$ has, individually, the same properties as your original variable $X$.

Now back to your remaining questions:

If $X$ denote a random variable which means the "result of a coin toss" then $x_1$, $x_2$, ..., $x_n$ are the results of repeated coin tossing. Are $x_1$, $x_2$, ..., $x_n$ IID? If yes, how can observations be variables? [...]
How can a random sample be IID?

The terminology in probability theory can be complicated sometimes, because the concepts are all so closely related. But here's what I (think) I know.

Strictly speaking, the correct would be to say that $x_1$, $x_2$, ..., $x_n$ are drawn from a collection of i.i.d. variables $X_1$, $X_2$, ..., $X_n$.

However, even though the observations only exist after the coin has been tossed (and therefore it's not random anymore), we are still speaking in general terms. We are talking about any possible situation where you could have those $n$ observations, not about a particular set of observations that you know because you just tossed a coin many times. So we don't know them. We might just as well treat them as random variables.

Therefore, even though observations are not random variables, it makes sense to associate with them many properties that actually belong to random variables. By extension we can thus say that $x_1$, $x_2$, ..., $x_n$ are i.i.d. because they were all drawn independently and from identical distributions.

• Giving these long tutorial type answers does not serve the OP well and certainly does not help the community either. I think it makes more sense to do what whuber and Xi'an did by referring the OP to other reading material. Dec 30, 2016 at 20:40
• I have no idea what you mean by "tutorial type answer". Or what's wrong with it. Dec 30, 2016 at 21:58
• This is the type of answer that I was looking for. If I feel I'm not understanding any part of your answer, can I ask you for helping me? Dec 31, 2016 at 6:56
• Glad I could help! If you have questions that can be answered with a comment, just leave them here and I'll try to answer. Or you can always post new questions. I'm just not around SO that much to sustain a 1-1 conversation for too long. Dec 31, 2016 at 17:05
• I feel this is a good answer and gives a ton of intuition.
– nan
Dec 27, 2019 at 15:21

"IID" stands for independent and identically distributed. An IID sample is also known as a "random sample." It is referring to the distribution of all of your random variables: $X_1, \ldots, X_n$. In your case, you are looking for the joint mass function $f_{X_1, \ldots, X_n}(x_1, \ldots, x_n)$.

Typically random variables can be thought of as not-yet-observed data and are denoted with capital letters toward the end of the alphabet, like $X_1$. Once you flip the coin and observe the $0$ or $1$ (we code the results this way instead of as "heads" versus "tails", or $-1$ versus $1$), or once you talk about a random variable hypothetically being a certain value, that entity is no longer random; it is a fixed number, and it is usually denoted as a lowercase letter like $x_1$. You could call them "variables" but not "random variables."

Looking at the above notation, if we plug in a set of $n$ numbers, $x_1, \ldots, x_n$ into the function $f_{X_1, \ldots, X_n}(x_1, \ldots, x_n)$, we get a probability for this discrete configuration of coin flips.

In general it's difficult to come up with a joint distribution for your $n$ random variables $X_1, \ldots, X_n$. Most people are more familiar with univariate distributions (e.g. Normal, Gamma, Exponential, Chi-Square, F, t, etc.). In your case we know each $X_i$ has a Bernoulli distribution. In other words, $$f_{X_i}(x_i) = p^{x_i}(1-p)^{1-x_i},$$ where $0 < p < 1$ is some parameter for each random variable.

1. Independent means the joint "splits up" or factors into $n$ individual marginals, one for each random variable. Or in other words:

$$f_{X_1, \ldots, X_n}(x_1, \ldots, x_n) = f_{X_1}(x_1) \times \cdots \times f_{X_n}(x_n).$$

1. Identical means they all are identical probability mass functions. Or in your case, each coin flip random variable shares the same parameter $p$.

$$f_{X_1}(x_1) \times \cdots \times f_{X_n}(x_n) = [p^{x_1}(1-p)^{1-x_1}] \times \cdots \times[p^{x_n}(1-p)^{1-x_n}].$$

Putting these two assumptions together, your joint distribution/probability mass function, is

$$f_{X_1, \ldots, X_n}(x_1, \ldots, x_n) = p ^{\sum_i x_i} (1-p)^{n - \sum_i x_i}.$$

• This is nice but the OP seems to be new to probability and statistics and may really only need to understand the difference between observations and random variables. Dec 30, 2016 at 17:47
• @MichaelChernick oh you're definitely right. I'l make some edits. Dec 30, 2016 at 17:51
• Actually I understood everything Taylor explained. Thanks for your time Dec 31, 2016 at 7:06