Creating a "certainty score" from the votes in random forests?

Question

I am looking to train a classifier that will discriminate between Type A and Type B objects with a reasonably large training set of approximately 10,000 objects, about half of which are Type A and half of which are Type B. The dataset consists of 100 continuous features detailing physical properties of the cells (size, mean radius, etc). Visualizing the data in pairwise scatterplots and density plots tells us that there is significant overlap in the distributions of the cancerous and normal cells in many of the features.

I am currently exploring random forests as a classification method for this dataset, and I have been seeing some good results. Using R, random forests is able to correctly classify about 90% of the objects.

One of the things we want to try and do is create a sort of "certainty score" that will quantify how confident we are of the classification of the objects. We know that our classifier will never be 100% accurate, and even if high accuracy in predictions is achieved, we will want trained technicians to identify which objects are truly Type A and Type B. So instead of providing uncompromising predictions of Type A or Type B, we want to present a score for each object that will describe how A or B an object is. For example, if we devise a score that ranges from 0 to 10, a score of 0 may indicate an object is very similar to Type A objects, while a score of 10 will indicate an object is very much like Type B.

I was thinking that I could use the votes within the random forests to devise such a score. Since classification in random forests is done by majority voting within the forest of generated trees, I would assume that objects that were voted by 100% of the trees to be Type A would differ from objects that were voted by, say, 51% of the trees to be Type A.

Currently, I have tried setting an arbitrary threshold for the proportion of votes that an object must receive to be classified as Type A or Type B, and if the threshold is not passed it will be classified as Uncertain. For example, if I force the condition that 80% or more of the trees must agree on a decision for a classification to pass, I find that 99% of the class predictions are correct, but about 40% of the objects are binned as Uncertain.

Would it make sense, then, to take advantage of the voting information to score the certainty of the predictions? Or am I heading in the wrong direction with my thoughts?

Answer 1

It makes perfect sense, and all implementations of random forests I've worked with (such as MATLAB's) provide probabilistic outputs as well to do just that.

I've not worked with the R implementation, but I'd be shocked if there wasn't a simple way to obtain soft outputs from the votes as well as the hard decision.

Edit: Just glanced at R, and predict.randomForest does output probabilities as well.

Answer 2

If you are using R, the caret package will save you from re-inventing the wheel. For example, the following code uses cross-validation to choose the tuning parameters for a random forest model, and then outputs the mean and standard deviation of accuracy for each cross-validation fold. Finally, it calculates class probabilities for the model.

library(caret)
library(PerformanceAnalytics)
data(iris)

#Make a yes/no dataset
Dataset <- iris
Dataset$Class <- ifelse(Dataset$Species=='versicolor','Yes','No')
Dataset$Class  <- as.factor(Dataset$Class)
Dataset$Species<- NULL
chart.Correlation(Dataset[-5], col= Dataset$Class)

#Fit an RF model
model <- train(Class~.,Dataset,
                method='rf',TuneLength=3,
                trControl=trainControl(
                    method='cv',number=10,
                    classProbs = TRUE))
model$results

#Predict class probabilities (i.e. 'certainty' scores)
pred <- predict(model,iris,"prob")
head(pred)

The nice thing about caret is that it makes it very easy to compare different predictive models. For example, if you want to try an SVM, you can replace the text method='rf' with method='svmLinear' or method='svmRadial'. You can also choose your tuning parameters based on AUC rather than accuracy by adding a line to the trainControl parameter: summaryFunction=twoClassSummary.

Finnally, there's a bit of code in there from the PerformanceAnalytics package, chart.Correlation(Dataset[-5], col= Dataset$Class), which is not needed to build the model, but provides a nice visualization of your dataset.

Answer 3

The randomForest package in R is a pretty decent package for getting into greater details about your analysis. It provides you with the votes (either as a fraction or raw counts) and it offers built in capacity for tuning and cross validation and can even give you more information about your features as well (if you wanted to know which out of your 100 are the most important in analysis).

If you're already using that package, then maybe you want to give it a closer look and if you aren't then perhaps check it out.