3
$\begingroup$

What are the effects of over-specificating a linear regression model?

Could it be said that overfitting is less important when the goal is to only estimate the relationship between dependent and independent variables, and not making any predictions for the observations outside the given data set?

$\endgroup$
7
  • 2
    $\begingroup$ I don't believe so. Suppose you include one predictor $x$ that has a real effect on $y$, but also a bunch of others that are proxies for $x$. The true effect might be masked due to overfitting and your estimate would be poor. $\endgroup$
    – dsaxton
    Mar 18, 2016 at 15:43
  • $\begingroup$ @dsaxton In other words, you are saying that coefficients would be biased? $\endgroup$
    – Quirik
    Mar 18, 2016 at 15:47
  • $\begingroup$ Yes, pretty much. $\endgroup$
    – dsaxton
    Mar 18, 2016 at 16:00
  • 2
    $\begingroup$ I agree with @dsaxton, but I wouldn't call the result a bias. I would say that the resulting estimates would have such a high variance to be essentially untrustable and useless. Generally (but not always) bias results from undercutting, large variance from overfitting. $\endgroup$ Mar 18, 2016 at 16:50
  • 2
    $\begingroup$ @MatthewDrury Right, I was reluctant to attribute it just to variance because that seems to suggest it could be cured with a large enough sample size, but I'm not sure if that's the case in these situations. $\endgroup$
    – dsaxton
    Mar 18, 2016 at 18:42

2 Answers 2

1
$\begingroup$

I would argue that if a model is not useful for prediction (on data that comes from the same generating process/distribution), it is not a useful model for explaining the relationship between variables either.

$\endgroup$
0
$\begingroup$

I guess no:

  1. Overfitting means that the relationship between dependent and independent variables is not correctly estimated. It means that you are not capturing only the signal, but also the noise in the data.
  2. Even if you don't see overfitting in your model, it doesn't means that you are correctly estimating the relationship that you want. Because it can happen that you equally give the wrong relationship in both traing and testing set.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.