I have a dataset that I'm trying to classify into 2 groups, A and B, using a random forest model. I know the true grouping and I'm trying to see how well I can model it using the other available variables. I've tried 2 different approaches that I thought would be equivalent, but which are actually giving me quite different results:
- Reading in the grouping as a (non-numeric) factor in R, growing a classification forest, and taking the proportion of trees that vote for group A as my prediction.
- Constructing an indicator variable for membership of group A, growing a regression forest, and taking the ensemble prediction as usual.
The split between the 2 groups is roughly 90-10 A vs. B. I'm growing 240 trees from ~200k observations of the same variables. I've left most of the settings at the defaults for the R randomForest package, but to keep the processing time down to a manageable level I've increased the node size to 200. The results are as follows:
- In the vast majority of cases, all 240 trees vote for A. The average predicted chance of any one observation being in A is about 99.9%. Worse still, not a single member of group B gets a majority of votes for group B!
- I get a wide range of predictions, with the mean prediction lying close to the observed mean of ~90%.
How can two apparently similar methods give such different results?
As for how I ended up trying this - I was initially trying to classify my dataset into a larger number of groups, of which B was one, but I noticed that B was being classified almost 100% incorrectly. The other groups are all much better behaved, even though most of them make up a far smaller proportion of my data.