2
$\begingroup$

I want to build a prediction model on a dataset with ~1.6M rows and with the following structure:

enter image description here

And here is my code to make a random forest out of it:

fitFactor = randomForest(as.factor(classLabel)~.,data=d,ntree=300, importance=TRUE)

and summary of my data:

  fromCluster       start_day        start_time        gender       age          classLabel    
 Min.   : 1.000   Min.   :0.0000   Min.   :0.000   Min.   :1   Min.   :0.000   Min.   : 1.000  
 1st Qu.: 4.000   1st Qu.:1.0000   1st Qu.:1.000   1st Qu.:1   1st Qu.:0.000   1st Qu.: 4.000  
 Median : 6.000   Median :1.0000   Median :3.000   Median :1   Median :1.000   Median : 6.000  
 Mean   : 6.544   Mean   :0.7979   Mean   :2.485   Mean   :1   Mean   :1.183   Mean   : 6.537  
 3rd Qu.:10.000   3rd Qu.:1.0000   3rd Qu.:4.000   3rd Qu.:1   3rd Qu.:2.000   3rd Qu.:10.000  
 Max.   :10.000   Max.   :1.0000   Max.   :6.000   Max.   :1   Max.   :6.000   Max.   :10.000

But I don't understand why my error rate is so high!

enter image description here

What am I doing wrong?

$\endgroup$

migrated from stackoverflow.com May 5 '15 at 15:01

This question came from our site for professional and enthusiast programmers.

  • 1
    $\begingroup$ Shouldn't fromCluster be converted to a factor first? What do the different lines in your plot represent? $\endgroup$ – andrew May 5 '15 at 18:51
2
$\begingroup$

Random forest has several hyperparameters that need to be tuned. To do this correctly, you need to implement a nested cross validation structure. The inner CV will measure out-of-sample performance over a sequence of hyperparameters. The outer CV will characterize performance of the procedure used to select hyperparameters, and can be used to get unbiased estimates of AUC and so forth.

The hyperparameters that you may tune include ntree, mtry and tree depth (either maxnodes or nodesize or both). By far, the most important is mtry. The default mtry for $p$ features is $\sqrt{p}$. Increasing mtry may improve performance. I recommend trying a grid over the range $\sqrt{p}/2$ to $3\sqrt{p}$ by increments of $\sqrt{p}/2$.

Tuning ntree is basically an exercise in selecting a large enough number of trees so that the error rate stabilizes. Because each tree is i.i.d., you can just train a large number of trees and pick the smallest $n$ such that the OOB error rate is basically flat.

By default, randomForest will build trees with a minimum node size of 1. This can be computationally expensive for many observations. Tuning node size/tree depth might be useful for you, if only to reduce training time. In Elements of Statistical Learning, the authors write that they have only observed modest gains in performance to be had by tuning trees in this way.

$\endgroup$
  • $\begingroup$ I see.. Thanks for your explanations. I tried again with different values for 'mtry' but the gain was very little. How much 'error rate' is an acceptable value in a prediction model? I mean what can be an approximate maximum value for 'error rate' to say that we are actually predicting something? @user777 $\endgroup$ – Rojin May 5 '15 at 17:03
  • $\begingroup$ What kind of error rate is acceptable depends on your application. If you're building a spam filter, you only really care about making sure that real messages aren't discarded. If you're making a decision about amputating a limb, you'd better be certain that doing so will improve the patient's health! Keep in mind that just because you have data doesn't mean that there is signal. And better features usually beats a better algorithm. If you found my post helpful, please consider upvoting it by clicking the "up" arrow. $\endgroup$ – Sycorax May 5 '15 at 17:08
0
$\begingroup$

Just to add to the earlier answer, your start_day seems to be creating a class imbalance (it is mostly 1 and a few 0, right?), you could try removing that variable (just to confirm whether that's the case) or combining start_day with start_time, or using weighted RF to mitigate that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.