I have 2 variables which have a correlation of about 0.51. In the test dataset the estimate of one is changing from +ve to -ve. But keeping both of them in model is giving me good predictive power. Is it fine to keep both of them in model? or it might to lead to some wrong results?
$\begingroup$
$\endgroup$
7
-
4$\begingroup$ "In the test dataset the sign of one is changing from +ve to -ve" - what do you mean exactly? A little more detail about the process you're following might be helpful. $\endgroup$– Scortchi ♦Commented Mar 21, 2016 at 11:03
-
8$\begingroup$ How do you know that your model is overfitted? If an "overfitted" model is giving you truly better predictive power (no cheating, honest out of sample assessment) than some benchmark "non-overfitted" model, I would suspect that the "overfitted" model is actually non-overfitted but rather the "non-overfitted" benchmark model is actually underfitted. $\endgroup$– Richard HardyCommented Mar 21, 2016 at 11:03
-
$\begingroup$ overfitted, because the estimate sign is becoming -ve with other variable, while actually in partial regression with only that variable it holds a +ve sign. or in other words, with that variable, it holds a -ve estimate, but if removed holds a +ve estimate! $\endgroup$– user2542275Commented Mar 21, 2016 at 11:06
-
1$\begingroup$ Colinearity is your problem not overfitting? $\endgroup$– user1320502Commented Mar 21, 2016 at 11:08
-
2$\begingroup$ If it's improving your predictive power then it's not "over"fitting, by definition... $\endgroup$– user541686Commented Mar 22, 2016 at 0:50
|
Show 2 more comments
2 Answers
$\begingroup$
$\endgroup$
How do you know that your model is overfitted? If an "overfitted" model (let us call it model A) is giving you truly better predictive power (no cheating, honest out of sample assessment) than some benchmark model that you think is non-overfitted (call it model B), I would suspect that model B is actually underfitted while model A is
- less-underfitted than model B or
- non-overfitted or perhaps
- slightly overfitted (but not as severely as model B is underfitted).
So I would say it is fine to use model A in place of model B if you have to choose one of the two.
Regarding whether to keep both predictors or drop one, I would suggest making the choice based on out-of-sample performance assessment. If a model containing both of them gives better forecasts, choose it.
answered Mar 21, 2016 at 11:11
$\begingroup$
$\endgroup$
1
Sounds like your issue is collinearity rather than overfitting, as user1320502 suggested in a comment.
Do you know where these two variables came from? For example, if one is $x$ and one is $x^2$, centering the variables may help.
If all you care about is predicting, collinearity is not a direct problem. But if you look at other things (confidence intervals, etc), collinearity will affect things.
You might look at similar questions like:
How to solve collinearity problems in OLS regression?
How to prevent collinearity? (the question is about preventing collinearity, but see Aleksandr Blekh's answer which talks about dealing with collinearity if you didn't prevent it in the first place.
Importance of multiple linear regression assumptions when building predictive regression models (see the OP's comment on Michael Bishop's answer: nested CV)
-
$\begingroup$ so the one variable is x and the other is (x+y),where y is not a constant. The VIF is around 1.6, when only those 2 variables considered, but goes up around 2.1 when other model variables are considered. $\endgroup$ Commented Mar 22, 2016 at 8:21