# The paradox of i.i.d. data (at least for me)

As far as my aggregate (and scarce) knowledge on statistics permits, I understood that if $X_1, X_2,..., X_n$ are i.i.d. random variables, then as the term implies they are independent and identically distributed.

My concern here is the former property of i.i.d. samples, which reads: $$p(X_{n}|X_{i_1},X_{i_2},...,X_{i_k}) = p(X_{n}),$$

for any collection of distinct $i_j$'s s.t. $1 \leq i_j < n$.

However one knows that the aggregate of independent samples of identical distributions provide information about the distribution structure, and as a result about $X_n$ in the above case, so indeed it should not be the case that: $$p(X_{n}|X_{i_1},X_{i_2},...,X_{i_k}) = p(X_{n}).$$

I know that I am victim of fallacy but I don't know why. Please help me out on this one.

• Do you know Bayes rule? Heard of classics. vs Bayesian statistics? Priors? Dec 13, 2015 at 6:22
• I don't follow the argument at the end of your question. Can you be more explicit? Dec 13, 2015 at 7:10
• @Glen_b what is it exactly that you don't follow? What do you mean by the end of it? I am trying to say with different logics both an equality and an inequality seems plausible which is a paradox. Dec 13, 2015 at 21:52
• There's no paradox here--merely a failure to apply the appropriate definitions. You cannot claim to have a paradox when you ignore the meaning of the words you use! In this instance, comparing the definition of independent to that of probability will reveal the error.
– whuber
Dec 13, 2015 at 21:54
• @whuber, I assume you have noticed the explicit "(at least for me)" in the title of my question and also the fact that I ask for help to find the "fallacy" of my argument, which points at the fact that this is indeed not a real paradox. Dec 13, 2015 at 21:57

I think you are confusing an estimated model of a distribution with a random variable. Let's rewrite the independence assumption as follows: $$P(X_n | \theta, X_{i_1}, X_{i_2}, \dots, X_{i_k}) = P(X_n | \theta) \tag{1}$$ which says that if you know the underlying distribution of $X_n$ (and, for example, can identify it by a set of parameters $\theta$) then the distribution does not change given that you have observed a few samples from it.

For example, think of $X_n$ as the random variable representing the outcome of the $n$-th toss of a coin. Knowing the probability of head and tail for the coin (which, btw, assume is encoded in $\theta$) is enough to know the distribution of $X_n$. In particular, the outcome of the previous tosses does not change the probability of head or tail for the $n$-th toss, and $(1)$ holds.

Note, however, that $P(\theta | X_n) \neq P(\theta | X_{i_1}, X_{i_2}, \dots, X_{i_k})$.

• Thank you very much. Quite up to the point. Quite funny that I guessed such an answer a while ago but I forgot about it....So as far as I understand the fallacy goes with implicitly assuming "a model" which can parametrize the distribution of random variable. Did I get it right? Dec 13, 2015 at 21:56
• @Cupitor: I'm glad it was useful. Yes, conditioned on the model, the independent random variables do not affect each other. But, how likely a given distribution is to have generated a sequence of outcomes changes as you see more samples from the underlying (true) distribution (regardless of the independence assumption).
– Sobi
Dec 13, 2015 at 22:09

If you take a Bayesian approach and treat parameters describing the distribution of $X$ as a random variable/vector, then the observations indeed are not independent, but they would be conditionally independent given knowledge of $\theta$ hence $P(X_n \mid X_{n-1}, \ldots X_1, \theta) = P(X_n \mid \theta)$ would hold.

In a classical statistical approach, $\theta$ is not a random variable. Calculations are done as if we know what $\theta$ is. In some sense, you're always conditioning on $\theta$ (even if you don't know the value).

When you wrote, "... provide information about the distribution structure, and as a result about $X_n$" you implicitly were adopting a Bayesian approach but not doing it precisely. You're writing a property of IID samples that a frequentist would write, but the corresponding statement in a Bayesian setup would involve conditioning on $\theta$.

### Bayesian vs. Classical statisticians

Let $x_i$ be the result of flipping a lopsided, unfair coin. We don't know the probability the coin lands heads.

• To the classical statistician, the frequentist, $P(x_i = H)$ is some parameter, let's call it $\theta$. Observe that $\theta$ here is a scalar, like the number 1/3. We may not know what the number is, but it's some number! It is not random!
• To the Bayesian statistician, $\theta$ itself is a random variable! This is extremely different!

The key idea here is that the Bayesian statistician extends the tools of probability to situations where the classical statistician doesn't. To the frequentist, $\theta$ isn't a random variable because it only has one possible value! Multiple outcomes are not possible! In the Bayesian's imagination though, multiple values of $\theta$ are possible, and the Bayesian is willing to model that uncertainty (in his own mind) using the tools of probability.

### Where is this going?

Let's say we flip the coin $n$ times. One flip does not affect the outcome of the other. The classical statistician would call these independent flips (and indeed they are). We'll have: $$P(x_n=H \mid x_{n-1}, x_{n-2}, \ldots,x_{1}) = P(x_n=H) = \theta$$ Where $\theta$ is some unknown parameter. (Remember, we don't know what it is, but it's not a random variable! It's some number.)

A Bayesian deep into subjective probability would say that what matters is the probability from her perspective!. If she sees 10 heads in a row, an 11th head is more likely because 10 heads in a row leads one to believe the coin is lopsided in favor of heads.

$$P(x_{11} = H \mid x_{10}=H, x_{9}=H, \ldots,x_{1}=H) > P(x_1 = H)$$

What has happened here? What is different?! Updating beliefs about a latent random variable $\theta$! If $\theta$ is treated as a random variable, the flips aren't independent anymore. But, the flips are conditionally independent given the value of $\theta$.

$$P(x_{11} = H \mid x_{10}=H, x_{9}=H, \ldots,x_{1}=H, \theta) = P(x_1 = H \mid \theta) = \theta$$

Conditioning on $\theta$ in a sense connects how the Bayesian and the classical statistician models the problem. Or to put it another way, the frequentist and the Bayesian statistician will agree if the Bayesian conditions on $\theta$.

### Further notes

I've tried my best to give a short intro here, but what I've done is at best quite superficial and the concepts are in some sense quite deep. If you want to take a dive into the philosophy of probability, Savage's 1954 book, Foundation of Statistics is a classic. Google for bayesian vs. frequentist and a ton of stuff will come up.

Another way to think about IID draws is de Finetti's theorem and the notion of exchangeability. In a Bayesian framework, exchangeability is equivalent to independence conditional on some latent random variable (in this case, the lopsidedness of the coin).

• In essence, the bayesian approach would treat a statement "i.i.d. random variables" not as an axiom that they must be IID but just as a very strong prior assumption that they are so - and if even stronger evidence suggests that it's extremely unlikely that the given assumptions are true, then this "disbelief in the given conditions" will be reflected in the results. Dec 13, 2015 at 11:03
• Thank you very much for your thorough answer. I have upvoted it, but I think Sobi's answer, points out more explicitly where the problem lies, i.e. implicitly assuming the model structure (or this is as far as I understood it) Dec 13, 2015 at 21:54
• @Matthew Gunn: neat, thorough, and very well explained! I learned a few things from your answer, thanks!
– Sobi
Dec 13, 2015 at 22:18