5
$\begingroup$

I am reading Statistical Rethinking (Section 4.3).

When talking about the i.i.d. assumption used to build a linear regression model - without knowing if distribution values are correlated, the author says:

A moment's reflection tells us that this is hardly ever true, in a physical sense. Whether measuring the same distance repeatedly or studying a population of heights, it is hard to argue that every measurement is independent of the others. For example, heights within families are correlated because of alleles shared through recent shared ancestry.

The i.i.d. assumption doesn't have to seem awkward, however, as long as you remember that probability is inside the golem [the model], not outside in the world. The i.i.d. assumption is about how the golem represents its uncertainty. It is an epistemological assumption. [...] The point isn't to say epistemology trumps reality, but rather that in ignorance of such correlations the most conservative distribution to use is i.i.d. [...]

Even furthermore, there are many types of correlation that do little or nothing to the overall shape of a distribution, but only affect the precise sequence in which values appear. For example, pairs of sisters have highly correlated heights. But the overall distribution of female height remains almost perfectly normal. In such cases, i.i.d. remains perfectly useful, despite ignoring the correlations.

Why is i.i.d. the most conservative distribution assumption? Because it does not introduce additional assumptions in the model?

$\endgroup$
9
  • 5
    $\begingroup$ "Conservative" rarely is a well-defined term, depends on context, and usually is subjective. Its meaning can be the logical negation of what you might think, depending on the perspective of the speaker. $\endgroup$
    – whuber
    Feb 20, 2018 at 4:01
  • 1
    $\begingroup$ @whuber, thanks. I updated the question hoping to clarify your doubts. $\endgroup$
    – gc5
    Feb 21, 2018 at 2:28
  • 2
    $\begingroup$ Another point about nomenclature: "i.i.d." is not a distribution, it is an aspect of the joint distribution of a sequence of random variables. $\endgroup$
    – AdamO
    Feb 21, 2018 at 17:43
  • 1
    $\begingroup$ Lastly, I think most (all?) statisticians would agree that, if there is some known dependence structure in the data, the most appropriate analysis controls for that dependence in some manner. And if that variance structure is unknown, the model for independent data has a high degree of relevance but that ignoring the dependence is a limitation. $\endgroup$
    – AdamO
    Feb 21, 2018 at 17:46
  • 1
    $\begingroup$ @whuber: totally agree that "conservative" is not straightforward. Even under my interpretation of what I believe they mean, I still don't think the statement is 100% true either. Rather, it's often the case that the iid assumption ends up with a posterior that over states certainty than if we had considered a more complex reasonable dependence structure. But surely there are also some dependency structures that lead to more certainty than the iid assumption. $\endgroup$
    – Cliff AB
    Feb 21, 2018 at 21:12

3 Answers 3

10
$\begingroup$

I think the use of a word conservative here is interesting to say the least. I'm used to saying it's the strongest assumption, the one that's hardest to prove that it holds and frankly the one that's probably violated most easily.

It's the assumption that's easiest to build upon when teaching the regression theory. You don't need to worry about correlations and all the problems that they bring. You can easily apply CLT to get the asymptotic variances of parameters etc.

You'll notice how easy it is to work with i.i.d. errors the moment you start talking about time series. All of a sudden you realize that the assumptions that are somewhat reasonable in cross-sectional analysis, do not hold in time series usually. Even in the cross-sectional analysis you don't really need independence and get get away with weakened assumption, e.g. see Gauss-Markov theorem.

To me semantically it's better to use a word conservative when referencing the weakest assumption, i.e. the one that should hold true in most situations, not the strongest one, that holds rarely if ever. I would call i.i.d. assumption the most liberal, because it also liberates you from the necessity to deal with all the correlation and dependence issues, it lets you build this wonderful ideal world of independent errors. I could also call IID assumption outlandish

$\endgroup$
1
  • 3
    $\begingroup$ +1, I view "conservative" as models has least number of parameter / least flexible. $\endgroup$
    – Haitao Du
    Feb 21, 2018 at 15:11
5
$\begingroup$

(I must note that I have not read the book, and thus may be misinterpreting this passage, or criticizing it inappropriately. That said...)

I don't think this is correct.

  1. The standard regression assumption of i.i.d. errors does not pertain to the population from which the data were drawn. It is about the data that you are using to fit the model. That is, no one should ever believe that all human adult female heights that have ever existed or will exist, are independent of each other. They cannot be, due to shared genes, among other reasons. However, it is certainly possible, and often quite reasonable (IMHO) to imagine that the data in your sample are independent, e.g., when you have a set of young women all of whom are unrelated to each other. In that case, fitting a model that assumes the data are independent can be just fine.
  2. The import of the assumption of independence is not for the shape of the population distribution. While it can depend on the nature of the nonindependence and the estimation of the model, it is often the case that the mean estimates are unbiased, even when the data are not independent. Instead, the concern is typically about the appropriate width of a confidence interval around that estimated mean (or in a different framing, about the correctness of the p-value from a test of that parameter).

As excerpted, the comment seems to be off-base to me. I am not primarily a Bayesian, and am considerably less sophisticated with Bayesian statistics, so it is possible there is some alternative Bayesian framing or interpretation of this such that the iid assumption is only about the whole possible (infinite) population, and specifically about its shape. But I am not aware of this.

$\endgroup$
2
$\begingroup$

One possible interpretation of "conservative" is in the context of statistical testing. Conservative tests reject the null hypothesis less often than they should. An example of a conservative test is the Fisher's Exact Test: the actual false positive error rate is less than the nominal size of the test due to the discrete distribution of the odds ratio under permutations of the table values.

In linear regression, we often test the hypothesis that one or more of the regression parameters is 0. If the errors are not in fact IID, the optimal solution due to the Gauss Markov Theorem, as @Aksakal mentioned, is the inverse variance weighted least squares. Naively using unweighted least squares does not bias estimates when the mean model is true. The lack of weighting does affect the level of the test of the regression parameters. The test with unweighted least squares may be conservative or anticonservative.

If there are unmeasured sources of dependence or heteroscedasticity in observations, the robust sandwich variance estimator from generalized estimating equations produces standard errors that are consistent and produce tests of the correct level. I would argue that if we are discussing violations of model assumptions, the GEE should be mentioned. The GEE has nothing to do with being conservative, but producing correct inference.

$\endgroup$
2
  • $\begingroup$ With Gauss-Markov you don't need IID errors, the spherical errors with exogeneity work just fine. In this regard I'd say that Gauss-Markov is more conservative, as it requires fewer assumptions and work in more situations than IID $\endgroup$
    – Aksakal
    Feb 21, 2018 at 20:24
  • $\begingroup$ @Aksakal indeed, as I said the Gauss Markov Theorem implies we should weight by the inverse variance matrix to obtain the best linear unbiased estimator. That has nothing to do with being conservative, but optimality. The challenge of course is you must know the covariance to weight appropriately. $\endgroup$
    – AdamO
    Feb 21, 2018 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.