# Clarifying the meaning of i.i.d. when describing a set of variables

Let $$Z_1, ..., Z_k$$ be identically and independently distributed (i.i.d.) set of standard normal random variables.

I understand that as part of the i.i.d. independent broadly means that variables occur independently of one another, meaning they can be multiplied. But I would like to ask a couple of basic questions please.

Q1: Can somebody clarify what exactly is meant by identically distributed ? Does it simply mean that each $$Z_k$$ variable has the same distributional properties, i.e. the mean and variance; or does it mean something else?

Q2: I am aware that the i.i.d. is an assumption in itself, but in practice $$Z_k$$ variables rarely, if ever, have identical distributional properties. Is this fact usually acknowledged and ignored or as long as distributional properties of each variable are similar, it is deemed to be a satisfactory situation?

• Identically distributed suggests the same CDF. That would imply the same mean and variance (if these exist) but not the reverse. – Henry Jan 13 '20 at 15:28
• Random sampling (with replacement if necessary) may lead to identical distributions – Henry Jan 13 '20 at 15:29
• @Henry thank you for the clarification. Just a short follow-up re comment 1 to make sure I got this right. When you noted "but not the reserve" did it mean that normal variables with the same mean and variance do not necessarily have the same CDF? – PsychometStats Jan 13 '20 at 15:37
• Standard normal distributed random variables have identical distributions (Gaussian with mean 0 and variance 1). My point was there are other distributions with mean 0 and variance 1 which are not Gaussian (e.g. $\pm1$ with equal probabilities) and so not identically distributed to standard normals – Henry Jan 13 '20 at 15:46
• @Henry thank you so much, I got it!!! – PsychometStats Jan 13 '20 at 15:48

To begin with, let's ignore the part of the first sentence where the $$Z_k$$'s are assumed to be normal random variables: they are just i.i.d. random variables. What does this mean?

Well, there are two distinct properties, viz. independence and identical distribution that are being ascribed to these random variables. The OP claims with regard to independence that he

"understand ..... variables occur independently of one another, meaning they can be multiplied"

Not so. The definition of independent random variables has nothing to do with multiplication of the random variables; what is correct that for independent random variables, the joint CDF is the product of the marginal CDFs of the random variables: \begin{align} F_{Z_1, Z_2, \cdots, Z_n}(z_1, z_2, \cdots, z_n) &= P(Z_1\leq z_1, Z_2\leq z_2, \cdots, Z_n\leq z_n)\\ &= F_{Z_1}(z_1)F_{Z_2}(z_2)F_{Z_n}(z_n)\\ &= P(Z_1\leq z_1)P(Z_2\leq z_2)\cdots P(Z_n\leq z_n). \end{align} If the $$Z_k$$ are independent continuous random variables, then they are jointly continuous too, and the joint pdf factors into the product of the marginal pdfs: $$f_{Z_1, Z_2, \cdots, Z_n}(z_1, z_2, \cdots, z_n) = f_{Z_1}(z_1)f_{Z_2}(z_2)f_{Z_n}(z_n).$$ If the $$Z_k$$ are independent discrete random variables, then their joint pmf factors into the produce of the marginal pmfs: \begin{align} p_{Z_1, Z_2, \cdots, Z_n}(z_1, z_2, \cdots, z_n) &= P(Z_1= z_1, Z_2=z_2, \cdots, Z_n= z_n)\\ &= p_{Z_1}(z_1)p_{Z_2}(z_2)p_{Z_n}(z_n)\\ &= P(Z_1= z_1)P(Z_2= z_2)\cdots P(Z_n= z_n). \end{align}

With respect to identical distributions, the OP asks

Does it simply mean that each $$Z_k$$ variable has the same distributional properties, i.e. the mean and variance; or does it mean something else?

Here, the first part is a true statement: Each $$Z_k$$ has the same CDF. For every real number $$z$$, $$F_{Z_1}(z) = F_{Z_2}(z) =\cdots = F_{Z_k}(z) = \cdots = F_{Z_n}(z).$$ Not every random variable has a mean or variance and so imputing that the meaning of "same distributional properties" is entirely a matter of having the same mean and variance is a stretch too far. It is a stretch too far even if we restrict our attention to only those random variables that have means and variances. As has been pointed out in the comments, two random variables having the same mean and variance does not imply that they have the same CDF. The only saving grace for the OP's thinking is that it is true for normal random variables: normal random variables with the same mean and variance have the same CDF (and the same pdf too).

Finally, turning to random sampling from a finite set, the samples are identically distributed regardless of whether the sampling is with replacement or without replacement. This last part is likely to cause some controversy among beginning statisticians and so I include an example.

Consider the drawing of two balls without replacement from an urn with three balls of different colors, say Red, Blue and Green. The six equally likely possible outcomes thus are $$(R,B), (R,G), (B, R), (B, G), (G, R), (G, B).$$ What is the probability that the first ball is Red? Well, if the first ball is Red, the outcome must be either $$(R,B)$$ or $$(R,G)$$ and so $$P\left(\text{first ball is Red}\right) = \frac 26 = \frac 13.$$ What is the probability that the second ball is Red? Well, if the second ball is Red, the outcome must be either $$(B,R)$$ or $$(G,R)$$ and so $$P\left(\text{second ball is Red}\right) = \frac 26 = \frac 13.$$ These probabilities are the same as when the sampling is done with replacement.

The moral of this cautionary tale is that both sampling with replacement and sampling without replacement give identical distributions for the samples. The difference is that in sampling with replacement, the samples are independent while in sampling without replacement, the samples are dependent. In fact, this is a nice illustration of the fact that while i.i.d. is a shibboleth that we all murmur in times of stress, the two properties of independence and identical distribution that we so casually lump together as seemingly inseparable are in fact distinct, and that one can hold without the other.

• +1 Thank you for the explanation. I have found it clear and informative – PsychometStats Nov 20 '20 at 16:13

Standard normal distributed random variables have identical distributions (Gaussian with mean $$0$$ and variance $$1$$). There are other distributions with mean $$0$$ and variance $$1$$ which are not Gaussian (e.g. $$±1$$ with equal probabilities) and so not identically distributed to standard normals.