I am using randomForest (the R package) to train a multi-factor, binary- classification model. In trying to dissect performance, I started feeding in individual factors to see how the RF treated them.

The first one was a continuous variable, with a range of -1.0 : 1.0. The factor is such that for 55% of the data (both training and OOS), values above 0.0 corresponds to class 1 and below 0.0 to class 2.

When I run this single-factor model through the randomForest however, it fails to learn this.

Here's an example:

f1   f1_signal rf_pred y_class
0.6    1       -1        1
-1.0  -1       -1       -1
0.6    1       -1        1      
-0.9  -1       -1        1     
-0.9  -1        1       -1      
0.9    1       -1        1      
-0.7  -1       -1       -1     
0.7    1        1       -1       
0.9    1        1       -1       
0.6    1        1       -1       
1.0    1       -1       -1

Just separating the prediction at +/-f1 (f1_signal) gives an accuracy of approximately 54%.

Putting this through the randomForest framework:

rf_pred <- randomForest(as.factor(y) ~ f1, data)

gives an accuracy of approximately 27% - substantially worse.

I understand that subsequent nodes may cause it to 'overlearn' the data, and have tried to address this by limiting maxnodes=2, and nodesize to 0.5*(training_sample_size), but still nothing.

My questions are:

1) Is there something I should be doing differently to allow the randomForest to learn this separation, or is randomForest simply ill-suited as a tool in a single-factor model?

2) Given its failure in learning a simple boundary, how do I view its learning in the larger, multi-factor space? Even running repeated OOS tests, can there be confidence (i.e. empirical measures) that actual 'learning' has happened? Is there a way to extract (for example), high-level factor splits, and apply them manually? I get that this last point is not really relevant in the RF world, but something along those lines.....

3) Is there another tool that might be better suited to building a model that does multi-factor, single-level splitting?

  • 1
    $\begingroup$ could you please post some mock data and code showing the model(s) you are trying to fit? $\endgroup$ – Ankur Chakravarthy Feb 28 '16 at 1:28
  • $\begingroup$ @AnkurChakravarthy - updated post with some sample data. $\endgroup$ – BrauHaus Feb 29 '16 at 15:21

So now that the situation is clearer; feeding factors in one by one is not a reliable way of quantifying what variables are driving performance; look up the "importance" function to estimate how useful a factor/variable is when you keep the rest of the model constant ; and if your data are somewhat correlated, a better approach to quantifying predictor performance is to use false discovery rates relative to a null distribution, which you can do using an R package I developed, called pRF (which is on CRAN).



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.