4
$\begingroup$

I have a training dataset with a binary response variable, 6 independent variables, and 21,000 observations.

I've fit both an ordinary regression tree and a random forest (mtry = 2, ntree = 2000) and there is almost no difference between the two when each model is validated, using RMSE and predicted to actual ratio as goodness of fit metrics.

Is this to be expected with a small number of independent variables, or am I not using the right metrics to measure goodness of fit?

Improve this question
$\endgroup$
7
  • 4
    $\begingroup$ Just a technical point but if your response is binary then it is performing classification not regression. $\endgroup$ – Meadowlark Bradsher Jul 3 '14 at 3:54
  • 1
    $\begingroup$ Are you evaluating your models using hold-out data or cross-validation/bootstrap? $\endgroup$ – David Marx Jul 3 '14 at 4:01
  • $\begingroup$ Meadowlark- true, but the output I need is probabilities to be coded into lookup tables for actuarial software. $\endgroup$ – JenSCDC Jul 3 '14 at 8:59
  • $\begingroup$ David- the final evaluation is done on holdout data. $\endgroup$ – JenSCDC Jul 3 '14 at 9:02
  • 3
    $\begingroup$ RFs were introduced to deal with issues of overfitting in normal decision trees but this doesn't happen on every data set. More then the number of independent variables in depends on the number of possible splits that can be made with each variable and other things that can bias greedy decision tree learning and cause it to overfit. $\endgroup$ – Ryan Bressler Jul 10 '14 at 4:00
2
$\begingroup$

Answered in a comment by Ryan Bressler.

RFs were introduced to deal with issues of overfitting in normal decision trees but this doesn't happen on every data set. More then the number of independent variables in depends on the number of possible splits that can be made with each variable and other things that can bias greedy decision tree learning and cause it to overfit. – Ryan Bressler Jul 10 '14 at 4:00. Distributed under a Creative Commons CC BY-SA 3.0 license.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.