Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have 2 variables which have a correlation of about 0.51. In the test dataset the estimate of one is changing from +ve to -ve. But keeping both of them in model is giving me good predictive power. Is it fine to keep both of them in model? or it might to lead to some wrong results?

share|improve this question
4  
"In the test dataset the sign of one is changing from +ve to -ve" - what do you mean exactly? A little more detail about the process you're following might be helpful. – Scortchi Mar 21 at 11:03
8  
How do you know that your model is overfitted? If an "overfitted" model is giving you truly better predictive power (no cheating, honest out of sample assessment) than some benchmark "non-overfitted" model, I would suspect that the "overfitted" model is actually non-overfitted but rather the "non-overfitted" benchmark model is actually underfitted. – Richard Hardy Mar 21 at 11:03
    
overfitted, because the estimate sign is becoming -ve with other variable, while actually in partial regression with only that variable it holds a +ve sign. or in other words, with that variable, it holds a -ve estimate, but if removed holds a +ve estimate! – user2542275 Mar 21 at 11:06
1  
Colinearity is your problem not overfitting? – user1320502 Mar 21 at 11:08
1  
If it's improving your predictive power then it's not "over"fitting, by definition... – Mehrdad Mar 22 at 0:50

How do you know that your model is overfitted? If an "overfitted" model (let us call it model A) is giving you truly better predictive power (no cheating, honest out of sample assessment) than some benchmark model that you think is non-overfitted (call it model B), I would suspect that model B is actually underfitted while model A is

  1. less-underfitted than model B or
  2. non-overfitted or perhaps
  3. slightly overfitted (but not as severely as model B is underfitted).

So I would say it is fine to use model A in place of model B if you have to choose one of the two.

Regarding whether to keep both predictors or drop one, I would suggest making the choice based on out-of-sample performance assessment. If a model containing both of them gives better forecasts, choose it.

share|improve this answer

Sounds like your issue is collinearity rather than overfitting, as user1320502 suggested in a comment.

Do you know where these two variables came from? For example, if one is $x$ and one is $x^2$, centering the variables may help.

If all you care about is predicting, collinearity is not a direct problem. But if you look at other things (confidence intervals, etc), collinearity will affect things.

You might look at similar questions like:

How to solve collinearity problems in OLS regression?

How to prevent collinearity? (the question is about preventing collinearity, but see Aleksandr Blekh's answer which talks about dealing with collinearity if you didn't prevent it in the first place.

Importance of multiple linear regression assumptions when building predictive regression models (see the OP's comment on Michael Bishop's answer: nested CV)

share|improve this answer
    
so the one variable is x and the other is (x+y),where y is not a constant. The VIF is around 1.6, when only those 2 variables considered, but goes up around 2.1 when other model variables are considered. – user2542275 Mar 22 at 8:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.