36
$\begingroup$

I am looking to train a classifier that will discriminate between Type A and Type B objects with a reasonably large training set of approximately 10,000 objects, about half of which are Type A and half of which are Type B. The dataset consists of 100 continuous features detailing physical properties of the cells (size, mean radius, etc). Visualizing the data in pairwise scatterplots and density plots tells us that there is significant overlap in the distributions of the cancerous and normal cells in many of the features.

I am currently exploring random forests as a classification method for this dataset, and I have been seeing some good results. Using R, random forests is able to correctly classify about 90% of the objects.

One of the things we want to try and do is create a sort of "certainty score" that will quantify how confident we are of the classification of the objects. We know that our classifier will never be 100% accurate, and even if high accuracy in predictions is achieved, we will want trained technicians to identify which objects are truly Type A and Type B. So instead of providing uncompromising predictions of Type A or Type B, we want to present a score for each object that will describe how A or B an object is. For example, if we devise a score that ranges from 0 to 10, a score of 0 may indicate an object is very similar to Type A objects, while a score of 10 will indicate an object is very much like Type B.

I was thinking that I could use the votes within the random forests to devise such a score. Since classification in random forests is done by majority voting within the forest of generated trees, I would assume that objects that were voted by 100% of the trees to be Type A would differ from objects that were voted by, say, 51% of the trees to be Type A.

Currently, I have tried setting an arbitrary threshold for the proportion of votes that an object must receive to be classified as Type A or Type B, and if the threshold is not passed it will be classified as Uncertain. For example, if I force the condition that 80% or more of the trees must agree on a decision for a classification to pass, I find that 99% of the class predictions are correct, but about 40% of the objects are binned as Uncertain.

Would it make sense, then, to take advantage of the voting information to score the certainty of the predictions? Or am I heading in the wrong direction with my thoughts?

$\endgroup$
  • $\begingroup$ Make sure you have separate training and testing data sets. Make sure you use your testing set as few as possible (ideally only once). $\endgroup$ – Boris Gorelik Jun 28 '11 at 6:05
  • $\begingroup$ @bgbg I am using a training/validation/test scheme, in a 70/20/10 split. I am training a model with 70% of the data and tuning the parameters based on the results on the validation set of 20%. After I tune the parameters on these two sets, I assess the model on the 10% test set. Though Breiman claims that the built in OOB error rate makes making a separate test set redundant, I am wary of his claim. $\endgroup$ – ialm Jun 28 '11 at 17:22
19
$\begingroup$

It makes perfect sense, and all implementations of random forests I've worked with (such as MATLAB's) provide probabilistic outputs as well to do just that.

I've not worked with the R implementation, but I'd be shocked if there wasn't a simple way to obtain soft outputs from the votes as well as the hard decision.

Edit: Just glanced at R, and predict.randomForest does output probabilities as well.

$\endgroup$
  • 1
    $\begingroup$ Thank you for your reply. You are right, I have made a script that will output the proportion of votes each object receives. I wonder, though, how useful these votes will be? Are there any next steps that you recommend? Should I look at the variability of the voting proportions through additional runs of RF? I understand that RF has a stochastic element to it. Are there any diagnostics that I should look at? $\endgroup$ – ialm Jun 27 '11 at 23:22
  • 3
    $\begingroup$ @Jonathan You may try looking at a plot featuring the fraction of real object form some class as a function of a fraction of votes for this class from the forest. I was working on one problem which required confidence score and it turned out that I managed to obtain a very nice logistic curve. $\endgroup$ – user88 Jun 27 '11 at 23:35
  • $\begingroup$ @mbq Sorry, can you clarify what you mean by that? I am interested in your suggestion and will follow up on it tomorrow! $\endgroup$ – ialm Jun 27 '11 at 23:47
  • 1
    $\begingroup$ Also, the variability in a RF comes in the training stage, so running test samples multiple times through the RF won't change the results. I would look at the AUC vs. number of trees in RF, to ensure you have enough trees, and then optimize the minimum leaf parameter according to AUC. $\endgroup$ – benhamner Jun 28 '11 at 1:19
  • 2
    $\begingroup$ @Jonathan I think mbq is referring to something named calibration (although there may be other names). See for example this questions: Calibrating a multi-class boosted classifier, What do “real values” refer to in supervised classification? $\endgroup$ – steffen Jun 28 '11 at 6:17
16
$\begingroup$

If you are using R, the caret package will save you from re-inventing the wheel. For example, the following code uses cross-validation to choose the tuning parameters for a random forest model, and then outputs the mean and standard deviation of accuracy for each cross-validation fold. Finally, it calculates class probabilities for the model.

library(caret)
library(PerformanceAnalytics)
data(iris)

#Make a yes/no dataset
Dataset <- iris
Dataset$Class <- ifelse(Dataset$Species=='versicolor','Yes','No')
Dataset$Class  <- as.factor(Dataset$Class)
Dataset$Species<- NULL
chart.Correlation(Dataset[-5], col= Dataset$Class)

#Fit an RF model
model <- train(Class~.,Dataset,
                method='rf',TuneLength=3,
                trControl=trainControl(
                    method='cv',number=10,
                    classProbs = TRUE))
model$results

#Predict class probabilities (i.e. 'certainty' scores)
pred <- predict(model,iris,"prob")
head(pred)

The nice thing about caret is that it makes it very easy to compare different predictive models. For example, if you want to try an SVM, you can replace the text method='rf' with method='svmLinear' or method='svmRadial'. You can also choose your tuning parameters based on AUC rather than accuracy by adding a line to the trainControl parameter: summaryFunction=twoClassSummary.

Finnally, there's a bit of code in there from the PerformanceAnalytics package, chart.Correlation(Dataset[-5], col= Dataset$Class), which is not needed to build the model, but provides a nice visualization of your dataset.

$\endgroup$
  • $\begingroup$ You're making me a fan of the caret package. I think I will be keeping a second R thread open just to try different classification methods and have caret do CV and parameter tuning by itself and see if I get any comparable results. $\endgroup$ – ialm Jun 28 '11 at 17:09
  • $\begingroup$ @Jonathan glad to hear it! It's a wonderful package. Try the modelLookup() command for a list of what's possible. $\endgroup$ – Zach Jun 28 '11 at 17:17
  • $\begingroup$ The prob type of predict is also available by using randomForest to train your model (with or without the use of other packages like caret or PerformanceAnalytics). $\endgroup$ – Hack-R Mar 12 '15 at 17:49
2
$\begingroup$

The randomForest package in R is a pretty decent package for getting into greater details about your analysis. It provides you with the votes (either as a fraction or raw counts) and it offers built in capacity for tuning and cross validation and can even give you more information about your features as well (if you wanted to know which out of your 100 are the most important in analysis).

If you're already using that package, then maybe you want to give it a closer look and if you aren't then perhaps check it out.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.