RandomForest failing to learn single-factor, 'linearly separable' model

Question

I am using randomForest (the R package) to train a multi-factor, binary- classification model. In trying to dissect performance, I started feeding in individual factors to see how the RF treated them.

The first one was a continuous variable, with a range of -1.0 : 1.0. The factor is such that for 55% of the data (both training and OOS), values above 0.0 corresponds to class 1 and below 0.0 to class 2.

When I run this single-factor model through the randomForest however, it fails to learn this.

Here's an example:

f1   f1_signal rf_pred y_class
0.6    1       -1        1
-1.0  -1       -1       -1
0.6    1       -1        1      
-0.9  -1       -1        1     
-0.9  -1        1       -1      
0.9    1       -1        1      
-0.7  -1       -1       -1     
0.7    1        1       -1       
0.9    1        1       -1       
0.6    1        1       -1       
1.0    1       -1       -1

Just separating the prediction at +/-f1 (f1_signal) gives an accuracy of approximately 54%.

Putting this through the randomForest framework:

rf_pred <- randomForest(as.factor(y) ~ f1, data)

gives an accuracy of approximately 27% - substantially worse.

I understand that subsequent nodes may cause it to 'overlearn' the data, and have tried to address this by limiting maxnodes=2, and nodesize to 0.5*(training_sample_size), but still nothing.

My questions are:

1) Is there something I should be doing differently to allow the randomForest to learn this separation, or is randomForest simply ill-suited as a tool in a single-factor model?

2) Given its failure in learning a simple boundary, how do I view its learning in the larger, multi-factor space? Even running repeated OOS tests, can there be confidence (i.e. empirical measures) that actual 'learning' has happened? Is there a way to extract (for example), high-level factor splits, and apply them manually? I get that this last point is not really relevant in the RF world, but something along those lines.....

3) Is there another tool that might be better suited to building a model that does multi-factor, single-level splitting?

Answer 1

So now that the situation is clearer; feeding factors in one by one is not a reliable way of quantifying what variables are driving performance; look up the "importance" function to estimate how useful a factor/variable is when you keep the rest of the model constant ; and if your data are somewhat correlated, a better approach to quantifying predictor performance is to use false discovery rates relative to a null distribution, which you can do using an R package I developed, called pRF (which is on CRAN).

HTH.