Why i.i.d. is the most conservative distribution assumption

Question

I am reading Statistical Rethinking (Section 4.3).

When talking about the i.i.d. assumption used to build a linear regression model - without knowing if distribution values are correlated, the author says:

A moment's reflection tells us that this is hardly ever true, in a physical sense. Whether measuring the same distance repeatedly or studying a population of heights, it is hard to argue that every measurement is independent of the others. For example, heights within families are correlated because of alleles shared through recent shared ancestry.

The i.i.d. assumption doesn't have to seem awkward, however, as long as you remember that probability is inside the golem [the model], not outside in the world. The i.i.d. assumption is about how the golem represents its uncertainty. It is an epistemological assumption. [...] The point isn't to say epistemology trumps reality, but rather that in ignorance of such correlations the most conservative distribution to use is i.i.d. [...]

Even furthermore, there are many types of correlation that do little or nothing to the overall shape of a distribution, but only affect the precise sequence in which values appear. For example, pairs of sisters have highly correlated heights. But the overall distribution of female height remains almost perfectly normal. In such cases, i.i.d. remains perfectly useful, despite ignoring the correlations.

Why is i.i.d. the most conservative distribution assumption? Because it does not introduce additional assumptions in the model?

Answer 1

I think the use of a word conservative here is interesting to say the least. I'm used to saying it's the strongest assumption, the one that's hardest to prove that it holds and frankly the one that's probably violated most easily.

It's the assumption that's easiest to build upon when teaching the regression theory. You don't need to worry about correlations and all the problems that they bring. You can easily apply CLT to get the asymptotic variances of parameters etc.

You'll notice how easy it is to work with i.i.d. errors the moment you start talking about time series. All of a sudden you realize that the assumptions that are somewhat reasonable in cross-sectional analysis, do not hold in time series usually. Even in the cross-sectional analysis you don't really need independence and get get away with weakened assumption, e.g. see Gauss-Markov theorem.

To me semantically it's better to use a word conservative when referencing the weakest assumption, i.e. the one that should hold true in most situations, not the strongest one, that holds rarely if ever. I would call i.i.d. assumption the most liberal, because it also liberates you from the necessity to deal with all the correlation and dependence issues, it lets you build this wonderful ideal world of independent errors. I could also call IID assumption outlandish

Answer 2

(I must note that I have not read the book, and thus may be misinterpreting this passage, or criticizing it inappropriately. That said...)

I don't think this is correct.

1. The standard regression assumption of i.i.d. errors does not pertain to the population from which the data were drawn. It is about the data that you are using to fit the model. That is, no one should ever believe that all human adult female heights that have ever existed or will exist, are independent of each other. They cannot be, due to shared genes, among other reasons. However, it is certainly possible, and often quite reasonable (IMHO) to imagine that the data in your sample are independent, e.g., when you have a set of young women all of whom are unrelated to each other. In that case, fitting a model that assumes the data are independent can be just fine.
2. The import of the assumption of independence is not for the shape of the population distribution. While it can depend on the nature of the nonindependence and the estimation of the model, it is often the case that the mean estimates are unbiased, even when the data are not independent. Instead, the concern is typically about the appropriate width of a confidence interval around that estimated mean (or in a different framing, about the correctness of the p-value from a test of that parameter).

As excerpted, the comment seems to be off-base to me. I am not primarily a Bayesian, and am considerably less sophisticated with Bayesian statistics, so it is possible there is some alternative Bayesian framing or interpretation of this such that the iid assumption is only about the whole possible (infinite) population, and specifically about its shape. But I am not aware of this.

Answer 3

One possible interpretation of "conservative" is in the context of statistical testing. Conservative tests reject the null hypothesis less often than they should. An example of a conservative test is the Fisher's Exact Test: the actual false positive error rate is less than the nominal size of the test due to the discrete distribution of the odds ratio under permutations of the table values.

In linear regression, we often test the hypothesis that one or more of the regression parameters is 0. If the errors are not in fact IID, the optimal solution due to the Gauss Markov Theorem, as @Aksakal mentioned, is the inverse variance weighted least squares. Naively using unweighted least squares does not bias estimates when the mean model is true. The lack of weighting does affect the level of the test of the regression parameters. The test with unweighted least squares may be conservative or anticonservative.

If there are unmeasured sources of dependence or heteroscedasticity in observations, the robust sandwich variance estimator from generalized estimating equations produces standard errors that are consistent and produce tests of the correct level. I would argue that if we are discussing violations of model assumptions, the GEE should be mentioned. The GEE has nothing to do with being conservative, but producing correct inference.