3
$\begingroup$

I build a prediction modeling using both regression and random forest.

testmodel2<-lm(y~as.matrix(xtest))
summary(testmodel2)

rf2<-randomForest(y~.,data=df,importance=TRUE)
varImpPlot(rf2)

The regression model result shows that t1, t10 and t11 are not significant. However, the varImpPlot show that they are pretty important. On the other side, t3,t5 and t6 are significant in terms of P-value in the regression result, but they are not important in the Random forest result.

Is there any reason that linear regression result is different with random forest? Which one should be more reliable? The correlation matrix is also attached for the reference. The result of backward step-wise variable selection is also attached.

enter image description here

enter image description here

enter image description here

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

This can happen if your explanatory variables are correlated (multicollinearity problem). So, for a start I would suggest:

  1. check your correlation matrix (and post it here)

  2. p-scores in linear regression can change as you eliminate redundant variables one-by-one

    step(lm(y~as.matrix(xtest)), direction="backward", trace=1)

    (trace=1 shows your results step-by-step)

  3. before building any model you can try variable selection (first of all, elimination of the redundant ones). A good method is to use Variance Inflation Factors (VIF). For a nice explanation and example see "Collinearity and stepwise VIF selection" post here

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi Lanenok, I have added the correlation matrix in the original post. Any suggestions will be highly appreciated. $\endgroup$
    – user3269
    Nov 16, 2014 at 20:13
  • $\begingroup$ I also run the step wise variable selection. The result was attached as well. Looks like it just stop at the fist step. No step was moving forward. $\endgroup$
    – user3269
    Nov 16, 2014 at 20:18
  • $\begingroup$ You can try adding the argument "k" to the step function (see step {stats} documentation). k=2 is default, use larger integer to force elimination of variables. $\endgroup$
    – lanenok
    Nov 17, 2014 at 18:53
  • $\begingroup$ There are some correlated variables (t1-t12, t2-t8, t8-t9). Another idea relies on your domain knowledge. What are the dimensions of the correlated variables? Can it be that their ratios or differences are better predictors than each variable itself? $\endgroup$
    – lanenok
    Nov 17, 2014 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.