1
$\begingroup$

I am looking at a paper which uses a large panel data, 1 million observations, a dozen variables. I recall that in a discussion another one has the following comments:

In structural models like this with lots of data, is it a good idea to use such a large set of controls? e.g. do concerns about overfitting matter in situations like this?

How to understand this? So if the data is rich enough, we do not need to worry about overfitting (this make sense), but how does control many variables help with overfitting?

I guess this question is super naive so I do not dare to bother the discussants. Any help pointing me in the right direction of understanding this? Thanks in advance!

$\endgroup$
8
  • 1
    $\begingroup$ Could you elaborate on what "rich enough" might mean? It is very well established that merely having lots of numbers can be completely misleading--having more biased observations of a phenomenon doesn't remove the bias, for instance--so "rich" has to mean more than just "big." $\endgroup$
    – whuber
    Jan 26, 2019 at 22:36
  • $\begingroup$ @whuber Hi thanks! I think I want to say the data is representative, many individuals across many states, good variation in two dozen variables? $\endgroup$
    – Bob
    Jan 26, 2019 at 22:43
  • 1
    $\begingroup$ That might help, but the same potential problem applies. Indeed, the Literary Digest Poll of 1936 had the "rich" properties you describe but was grossly wrong. (It had over two million observations.) One can imagine a situation where that poll collected additional information, supplying data about "many individuals across many states" with "good variation in" many variables--but that wouldn't have made it any better and the results would have been just as wrong. $\endgroup$
    – whuber
    Jan 26, 2019 at 23:20
  • $\begingroup$ @whuber Thanks, in what way would having a lot of controls help in this case? $\endgroup$
    – Bob
    Jan 26, 2019 at 23:40
  • $\begingroup$ Could you explain what you mean by "control"? Your phrase "control many variables" is at odds with the usual meaning of a control group in an experiment, which makes me suspect you have a different meaning in mind. $\endgroup$
    – whuber
    Jan 27, 2019 at 14:11

1 Answer 1

1
$\begingroup$

This is a crude way of saying that a large sample size allows you to use more parameters without sacrificing estimation accuracy of those parameters. That is, when you have a large sample size, it is "safe" to use many features. The risk of overfitting to those features is less severe than it would be with a small sample size.

Since it is "safe" to use many features, that allows you to control for many factors that might influence the outcome. You can get a smaller residual variance by controlling for these variables, allowing you to have greater power to detect effects by your variables of interest.

This is all a bit crude, but I do believe that to be the idea behind the quote, and I agree with the quote to a limited extent.

However, I do not believe the quote to suggest that including many features helps with overfitting. The quote seems to allude to the fact that, due to the large sample size, overfitting is less of a concern if you want to include, say, twenty features than it would be if you had a small sample size.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.