7
$\begingroup$

I want to evaluate the calibration of the random forest using val.prob (rms package, R). I have no problems using it and getting an output, but I feel the results may not be accurate because I don't believe that the class membership probabilities outputted by the random forest (predict.randomForest, also in R) are proper predicted probabilities for use in calibration.

I don't believe it is a true probability for calibration in the following sense: I think a random forest score is NOT equivalent that event taking place (i.e. score of 30% doesn't mean you have 30% chance of having a disease).

My main question is : what is the appropriate way to use class membership probabilities when trying to assess model calibration? If it is possible to use the class membership probabilities, what methods allow one to improve the calibration?

Improve this question
$\endgroup$
3
  • $\begingroup$ Have you considered the eponymous method of Cross Validation ? $\endgroup$
    – image_doctor
    Commented Dec 11, 2012 at 10:35
  • $\begingroup$ Can you describe a little why cross-validation would be a solution? Would cross-validation allows certainty with which some item belongs to class become the probability of an event taking place? $\endgroup$
    – user4673
    Commented Dec 12, 2012 at 19:09
  • 1
    $\begingroup$ The class probabilities from a random forest relate only to the likelihood of an individual tree labelling an instance of data as belonging to a particular class. They have an unspecified relationship to the true underlying class densities. As the size of the forest increases it usually the case that these probabilities move closer to the sample densities. Which may or may not be closer to the true underlying densities.Techniques such as cross validation allow the forest to be optimsed against some metric, often the 0/1 loss function, which generally leads to a more accurate model. $\endgroup$
    – image_doctor
    Commented Dec 13, 2012 at 11:41

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.