I am looking to train a classifier that will discriminate between
Type A and
Type B objects with a reasonably large training set of approximately 10,000 objects, about half of which are
Type A and half of which are
Type B. The dataset consists of 100 continuous features detailing physical properties of the cells (size, mean radius, etc). Visualizing the data in pairwise scatterplots and density plots tells us that there is significant overlap in the distributions of the cancerous and normal cells in many of the features.
I am currently exploring random forests as a classification method for this dataset, and I have been seeing some good results. Using R, random forests is able to correctly classify about 90% of the objects.
One of the things we want to try and do is create a sort of "certainty score" that will quantify how confident we are of the classification of the objects. We know that our classifier will never be 100% accurate, and even if high accuracy in predictions is achieved, we will want trained technicians to identify which objects are truly
Type A and
Type B. So instead of providing uncompromising predictions of
Type A or
Type B, we want to present a score for each object that will describe how
B an object is. For example, if we devise a score that ranges from 0 to 10, a score of 0 may indicate an object is very similar to
Type A objects, while a score of 10 will indicate an object is very much like
I was thinking that I could use the votes within the random forests to devise such a score. Since classification in random forests is done by majority voting within the forest of generated trees, I would assume that objects that were voted by 100% of the trees to be
Type A would differ from objects that were voted by, say, 51% of the trees to be
Currently, I have tried setting an arbitrary threshold for the proportion of votes that an object must receive to be classified as
Type A or
Type B, and if the threshold is not passed it will be classified as
Uncertain. For example, if I force the condition that 80% or more of the trees must agree on a decision for a classification to pass, I find that 99% of the class predictions are correct, but about 40% of the objects are binned as
Would it make sense, then, to take advantage of the voting information to score the certainty of the predictions? Or am I heading in the wrong direction with my thoughts?