3
$\begingroup$

What are the effects of over-specificating a linear regression model?

Could it be said that overfitting is less important when the goal is to only estimate the relationship between dependent and independent variables, and not making any predictions for the observations outside the given data set?

$\endgroup$
7
  • 2
    $\begingroup$ I don't believe so. Suppose you include one predictor $x$ that has a real effect on $y$, but also a bunch of others that are proxies for $x$. The true effect might be masked due to overfitting and your estimate would be poor. $\endgroup$ – dsaxton Mar 18 '16 at 15:43
  • $\begingroup$ @dsaxton In other words, you are saying that coefficients would be biased? $\endgroup$ – Quirik Mar 18 '16 at 15:47
  • $\begingroup$ Yes, pretty much. $\endgroup$ – dsaxton Mar 18 '16 at 16:00
  • 2
    $\begingroup$ I agree with @dsaxton, but I wouldn't call the result a bias. I would say that the resulting estimates would have such a high variance to be essentially untrustable and useless. Generally (but not always) bias results from undercutting, large variance from overfitting. $\endgroup$ – Matthew Drury Mar 18 '16 at 16:50
  • 2
    $\begingroup$ @MatthewDrury Right, I was reluctant to attribute it just to variance because that seems to suggest it could be cured with a large enough sample size, but I'm not sure if that's the case in these situations. $\endgroup$ – dsaxton Mar 18 '16 at 18:42
1
$\begingroup$

I would argue that if a model is not useful for prediction (on data that comes from the same generating process/distribution), it is not a useful model for explaining the relationship between variables either.

$\endgroup$
0
$\begingroup$

I guess no:

  1. Overfitting means that the relationship between dependent and independent variables is not correctly estimated. It means that you are not capturing only the signal, but also the noise in the data.
  2. Even if you don't see overfitting in your model, it doesn't means that you are correctly estimating the relationship that you want. Because it can happen that you equally give the wrong relationship in both traing and testing set.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.