Is it possible to do random forest with multiple responses or combine such ensembles for multi-label classification?

Question

I have a dataset which has numerous nominal responses, and many predictors. Each response is basically a pass/fail check of a certain test applied to the data. Multiple methods are applied to each sample, so some methods pass and some fail for each sample. We have measurements of each sample that are the predictors.

 X1    X2    X3    X4  ...  Y1   Y2   Y3
0.3   0.7   0.2   1.3  ...   0    1    1
0.1   0.3   0.4   1.5  ...   1    1    0
0.4   0.5   0.6   1.0  ...   0    0    1
                       ...
0.3   0.3   0,7   1.2  ...   1    1    1

The idea is to create a decision tree which will identify which method is best under each condition (best being the method which will most pass with highest accuracy given the measurements). I've tried using randomForest in R and using the combine function but it cannot combine different responses. Am I going about this the wrong way? Random forests is the considered method because you can test the significance of each predictor. SVM is another idea but it's a somewhat black box method and no insight into the effect of each predictor on the outcome. Is there a way to chain the binary classifiers? Merge the binary classifiers for each method into one?

Answer 1

"Am I going about this the wrong way?" - yes combine() is intended to combine identical trained trees on multiple cpu's. Anything else would be off-label use.

"Random forests is the considered method because you can test the significance of each predictor" - Beware, significance testing of variable importance for RF is not directly compatible with the significance testing paradigme of Fischer statistics and typical linear regression. So do not take the p-values literally and rather look on the raw or scaled variable importance to decide which variables are the most useful. It is possible to implement variable importance for SVM or any regressor/classifier, it is just a little slow. You have to bag you estimator(bootstrap aggregated ensemble), thus retrain some 50 times or more.

"SVM is another idea but it's a somewhat black box method and no insight into the effect of each predictor on the outcome." - Does SVM have multiple outputs? If so, I missed that point. But anyway SVM is not much more black-box then RF. You can investigate either model structures with partial dependence plots, check out cran packages rMiner or iceBOX.

"Is there a way to chain the binary classifiers? Merge the binary classifiers for each method into one?"

Overall classic RF does not support multi target prediction. Don't think it is impossible, but I don't know of any elegant frame work to do so. I'd like to hear I'm wrong. You may wan't to consider a neural networks model instead, with three soft-max output neurons.

That said you could make a wrap-hack. Train model on features X, RF-y1, RF-y2 and RF-y3 only predicting the respective responses. Train new models RF2-y1, RF2-y2 and RF2-y3 to predict the same reponse but with the prediction of the other responses included as features. Thus for RF2-y1 the features are X1 ...Xp + $\hat{y_2}$ + $\hat{y_3}$. You should be careful how samples are recycled between models, as the out-of-bag cross validation easily may be violated.

Suggested solution In your case i would rephrase the problem to a 3 level classification. As you want to find the most likely suitable test option, the test options choices are mutual exclusive. So you won't predict if all test options will work great or not, but instead which one is most likely to work. The answer could be 30% y1, 50% y2 and 20% y3. There's gonna be an issue on how to include training samples with less or more than one passed test option. But a quick fix is just to copy the samples and include once for each positive test. Again emphasize how not to screw up the out-of-bag cross validation.