4
$\begingroup$

I have 2 variables which have a correlation of about 0.51. In the test dataset the estimate of one is changing from +ve to -ve. But keeping both of them in model is giving me good predictive power. Is it fine to keep both of them in model? or it might to lead to some wrong results?

$\endgroup$
  • 4
    $\begingroup$ "In the test dataset the sign of one is changing from +ve to -ve" - what do you mean exactly? A little more detail about the process you're following might be helpful. $\endgroup$ – Scortchi - Reinstate Monica Mar 21 '16 at 11:03
  • 8
    $\begingroup$ How do you know that your model is overfitted? If an "overfitted" model is giving you truly better predictive power (no cheating, honest out of sample assessment) than some benchmark "non-overfitted" model, I would suspect that the "overfitted" model is actually non-overfitted but rather the "non-overfitted" benchmark model is actually underfitted. $\endgroup$ – Richard Hardy Mar 21 '16 at 11:03
  • $\begingroup$ overfitted, because the estimate sign is becoming -ve with other variable, while actually in partial regression with only that variable it holds a +ve sign. or in other words, with that variable, it holds a -ve estimate, but if removed holds a +ve estimate! $\endgroup$ – user2542275 Mar 21 '16 at 11:06
  • 1
    $\begingroup$ Colinearity is your problem not overfitting? $\endgroup$ – user1320502 Mar 21 '16 at 11:08
  • 2
    $\begingroup$ If it's improving your predictive power then it's not "over"fitting, by definition... $\endgroup$ – Mehrdad Mar 22 '16 at 0:50
14
$\begingroup$

How do you know that your model is overfitted? If an "overfitted" model (let us call it model A) is giving you truly better predictive power (no cheating, honest out of sample assessment) than some benchmark model that you think is non-overfitted (call it model B), I would suspect that model B is actually underfitted while model A is

  1. less-underfitted than model B or
  2. non-overfitted or perhaps
  3. slightly overfitted (but not as severely as model B is underfitted).

So I would say it is fine to use model A in place of model B if you have to choose one of the two.

Regarding whether to keep both predictors or drop one, I would suggest making the choice based on out-of-sample performance assessment. If a model containing both of them gives better forecasts, choose it.

$\endgroup$
5
$\begingroup$

Sounds like your issue is collinearity rather than overfitting, as user1320502 suggested in a comment.

Do you know where these two variables came from? For example, if one is $x$ and one is $x^2$, centering the variables may help.

If all you care about is predicting, collinearity is not a direct problem. But if you look at other things (confidence intervals, etc), collinearity will affect things.

You might look at similar questions like:

How to solve collinearity problems in OLS regression?

How to prevent collinearity? (the question is about preventing collinearity, but see Aleksandr Blekh's answer which talks about dealing with collinearity if you didn't prevent it in the first place.

Importance of multiple linear regression assumptions when building predictive regression models (see the OP's comment on Michael Bishop's answer: nested CV)

$\endgroup$
  • $\begingroup$ so the one variable is x and the other is (x+y),where y is not a constant. The VIF is around 1.6, when only those 2 variables considered, but goes up around 2.1 when other model variables are considered. $\endgroup$ – user2542275 Mar 22 '16 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.