5
$\begingroup$

I have a dataset which consists of roughly 40,000 observations grouped into 12 classes.

I did a quick random forest run on a single split of 80% training and 20% testing (Dont bring up the discussion on wether or not it is necessary to split data in random forests, this is just a quick experiment) and got the following precision on the test set:

enter image description here

I then train a RF model for each class, creating a dichotomus variable 'belongs to class': yes/no.

I get, for example, these precisions on the test set for class number 1 and class number 5:

enter image description here

Which would seem more precise. Is training several models a viable option? It may be cumbersome but if you predict each observation with all the models and choose maybe the highest membership probability a la linear discriminant analysis could you get a better precision than a single random forest? I hope this question isnt dumb, I dont know why I think about such things.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I think you are describing the procedure known as one-versus all (ova). I outlined the steps in this question. It is usually a way of using binary classification algorithms for multiclass problems. It may be worthwhile to note that fitting separate binary classifiers may not be what you want if your goal is to predict future observations that fall into any class. Therefore, a binary classification must be constructed at each iteration of cross-validation. Also note that random forests can be sensitive to class imbalance, which you may have with 12 classes.

Improve this answer
$\endgroup$
7
  • $\begingroup$ If that would be exatly my goal (predict future observations), why would this be a bad idea? In which cases is OVA a good procedure to follow? $\endgroup$
    – JEquihua
    Mar 13, 2013 at 0:11
  • $\begingroup$ I meant to write that you may not want to a series of separate binary classifications. Rather, you may want one algorithm in which all pairwise classifiers are fit at each iteration of cross-validation. In sum, I was emphasizing the idea that all pairwise classifications are done at each iteration so that you end up with a multiclass rule. So yes, ova works. There are algorithms that handle multiclass naturally. But I have not seen any comparisons that one strategy is better. $\endgroup$
    – julieth
    Mar 13, 2013 at 0:18
  • $\begingroup$ I'm just bery interested in the idea that the precision could be better with the OVA procedure than a single model in multiclass form. RandomForests handle multiclass naturally as can be seen in my example but the precision from the binary version would seem to be higher for a single class. I can easily code an algorithm in which all pairwise classifiers are fit at each iteration of cross-validation but how would one go about on using all these pairwise classifiers to predict the class of a new observation? Thank you for your attention on this by the way. $\endgroup$
    – JEquihua
    Mar 13, 2013 at 0:36
  • $\begingroup$ In OVA you pick the class which has the highest probability according to its predictor. Interesting point: you are generating 12X the number of forests in the OVA algorithm. Won't increasing the number of forests generated by the multiclass algorithm give the same improvement? $\endgroup$
    – Guy Adini
    Mar 13, 2013 at 5:18
  • $\begingroup$ I dont know. That's exactly my question. Did you see my example? Everything was done with 1000 trees (the multiclass version and the single class versions). Are you suggesting that estimating the model with 12,000 tress would give an improvement of the magnitude seen in the single class versions? I really dont think that to be so. $\endgroup$
    – JEquihua
    Mar 13, 2013 at 5:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.