what exactly does it mean when we say "Let $X_1, X_2 ...$ be iid random variables"

Question

Every now and then I read that phrase and get confused. When we say

"Let $X_1, X_2, \dots X_n$ be iid random variables"

I thought this meant that we are sampling $X$ random variable n many times for infinitely many times. Let's say we want to know the sample mean distribution. Set $n=3$, and make $n$ many observations of $X$ and get values $2, 4, 3$. Then we add them up to get the sample mean $\bar{x} = 3$. But sample mean is also a random variable so the phrase "Let $X_1, X_2, \dots X_n$ be iid random variables" is equivalent to saying if we make $n$ observations of $X$ infinitely many times, $\bar{X}$ is a random variable with some distribution.

But for weak law of large numbers, we say that "Let $X_1, X_2, ...$ be iid random variables with $E(X_i) = \mu$ then $\lim_{n \to \infty} P(|\bar{X}_n-\mu| < \epsilon) = 1$"

What exactly is $\bar{X}_n$ here? Is it the $n$th sample of $\bar{X}$ or the distribution of $\bar{X}$ after having observed infinitely many $X$s?

Answer 1

$\bar X_n$ is simply the arithmetic mean of the sequence $X_1, X_2, ..., X_n$. In other words $\bar X_n = \frac{1}{n} \sum_{i=1}^n X_i$. And, indeed, it's a random variable in itself (not a distribution).

"Let $X_1, X_2, ..., X_n$ be i.i.d. random variables"

Simply means that you have a list of $n$ independent random variables that all share the same probability distribution.

I thought this meant that we are sampling X random variable n many times for infinitely many times.

It says nothing about how many times these random variables are sampled.

Answer 2

Direct meaning: The acronym IID means (statistically) independent and identically distributed. Hence, when we say, "Let $X_1, X_2, ..., X_n$ be IID random variables", we are simply saying that we have a set of $n$ random variables that we take to be independent, with the same distribution function (usually this is followed by specifying their common distribution). This type of statistical set-up is called the IID model and it is the most common statistical model in use. It is usual to specify a distributional form for the observable values with some unknown parameters, and the statistical problem then becomes one of estimating the unknown parameters in the distribution from the observed data.

Operational meaning: The above gives the direct meaning of these phrase in terms of setting up a statistical model. However, it is worth understanding where this model comes from in operational terms (i.e., in terms of probabilistic description directly on observable values). The IID model comes from a famous probability theorem called the representation theorem, initially due to de Finetti, but extended by others. For a sequence of exchangeable random variables $X_1, X_2, X_3, ...$ this theorem tells us that these random variables are independent and identically distributed (IID) with distribution equal to their empirical distribution function (conditional on that distribution function):

$$F(x) \equiv \lim_{n \rightarrow \infty} \frac{1}{n} \sum_{i=1}^n \mathbb{I}(X_i \leqslant x).$$

Operationally, this tells us that the IID model is the result of an assumption of exchangeability of the observable sequence of random variables (i.e., the order-invariance of the sequence). It also tells us that the underlying distribution of these observable random variables is the empirical distribution of the sequence, so any identifiable parameters must be functions of this distribution. If you would like to read a simple explanation of how this theorem applies to frequentist and Bayesian inference, a useful reference is O'Neill (2011).

The sample mean: You question also asks the meaning of the object $\bar{X}_n$. This is the sample mean of the first $n$ data points, which is defined as $\bar{X}_n \equiv \tfrac{1}{n} \sum_{i=1}^n X_i$. The sample mean is a statistic (meaning it is a function of the observed data) and its distribution is determined by the distribution of the underlying random variables (i.e., the empirical distribution of the sequence). In the IID model it is subject to theorems like the laws-of-large-numbers and the central limit theorem (CLT) (under certain additional assumptions).

Answer 3

"Let X1,X2,…Xn be iid random variables"

It means that you have n random variables, which share the same probability distribution and also these variables are mutually independent.

According to the law of large numbers, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed. In the formal definition you mentioned the number of the trials is described by n.