2
$\begingroup$

I am using survey data with a huge amount of observations, such as the World Value Surveys. Large sample sizes are obviously very nice, but I have have encountered some downsides as well.

To give an example, in almost every econometric model I specify, about 90% of the variables is highly significant. So I will have to decide whether, in addition to an estimate being statistically significant, it is also economically significant, which is not always an easy thing to do.

The biggest issue is however, that when resorting to Instrumental Variables, the Hausman test for over identification is always very, very, very significant. See to this extent THIS POST.

How do I deal with with this consequence of large sample sizes?

The only thing I can think of is to reduce the sample size. This however seems a very arbitrary way to get the test statistic down.

$\endgroup$

1 Answer 1

1
+50
$\begingroup$

It's not fully clear what the problem is. I read the post you linked, and indeed, assuming the difference in coefficients stays the same, a larger sample size (and thus smaller standard errors) means a larger Hausman test statistic value, and thus more likely to reject the null that both tests are consistent.

Why is this problematic at all? Similarly, if you run a linear regression, and say the estimate is the exact same values for $n=10,100,1000$, you're also more likely to reject the null that the estimate is $0$ for larger observations. More data means you're more able to detect differences, and thus typically more likely to reject a null.

It's tempting to think of significant being driven by the number of observations, but that's assuming that the values (in my example, the estimate, and in yours, the difference in coefficient estimates), stay the exact same as you increase number of observations. But the whole point of these tests is that with small number of observations, it's far more likely you observe quite different estimates each time you repeat the process, whereas with a large number of observations, this is less likely. This underlies much of statistical testing. Furthermore, insignificance in small samples doesn't mean there is no difference, but rather that you fail to reject the null: you lack the precision to observe a difference, but that doesn't mean it's not there.

If the difference between OLS and IV were truly $0$ under the data generating process, then no matter how much data you observe, you would still fail to reject the null hypothesis (sometimes you will reject, but at a low frequency.. this concept is called power). If there is a difference in the data generating process, no matter how small, with enough data, you will eventually be able to distinguish and pick up this effect. In your case, assuming the assumptions for a Hausman test hold, you are able to detect very small differences between IV and OLS, and you also reject the null that they are the same. The relative difference might be small, but you still reject the null. This suggests there truly is a difference, and if you were to reduce the sample size (btw, I can't think of a good reason to ever do this unless you're training data or something), you would just not be able to detect this difference. It doesn't mean it's not there--indeed, if you did that, just fudged your data until you failed to detect it, basically hiding it.

The effect is not coming from your number of observations, it's a property of the data generating process, but the ability to observe a difference does depend on the number of observations, and that's exactly what you are seeing.

Finally, failing to reject/rejecting a null is one interesting statement you can make given data, but it's not the only one. The magnitude of the effect also matters. This is why confidence intervals are important. Consider a regression coefficient: rejecting the null at $.05$ but having confidence interval of, say, [1,50000] could be interesting, but even though you reject the null, you know little about the effects. In contrast, for that same coefficient, failing to reject the null at $.05$ but having a confidence interval of, say, [-.0001,.0001]$ for the same effect could be far more interesting, as you can basically pinpoint the effect.

$\endgroup$
4
  • $\begingroup$ Thank you very much for a great answer! This helps me a lot! $\endgroup$
    – Tom
    Jun 17, 2020 at 18:57
  • $\begingroup$ Glad it helped :) $\endgroup$
    – doubled
    Jun 17, 2020 at 20:12
  • $\begingroup$ Are you in the last paragraph referring to regression coefficients in general (as I mentioned this in my question as well)? It seems that the Hausman test for over identification does not (or cannot) provide a confidence interval in any case. If you are talking about coefficients in general, I do see one problem however. If my instruments fail the test for over identification, I assume this will also create bias in my confidence interval for the variable of interest. But since I don't know how big this bias is, because the hausman does not specify that, how do I deal with this? $\endgroup$
    – Tom
    Jun 18, 2020 at 9:27
  • 1
    $\begingroup$ @TomKisters Yep I was just giving a more general statement that pvalues are not the only important thing to look at, and illustrated in the case of coefficients. Something you could still maybe look at it the actual difference between the two estimate using IV and OLS. $\endgroup$
    – doubled
    Jun 18, 2020 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.