1
$\begingroup$

It's not clear to me, how the different summary metrics for caret's train function are defined, when I predict probabilities of a multiclass problem.

Without a loss of generality, assume that I use a random forest to produce probabilities:

library(caret)
data(iris)

control = trainControl(method="CV", number=5,verboseIter = TRUE,classProbs=TRUE)
# iv) tuning parameter

grid = expand.grid(mtry = 1:3)
rf_gridsearch = train(y=iris[,5],x=iris[-5],method="ranger", num.trees=2000, tuneGrid=grid, trControl=control)
rf_gridsearch

# Output:
....
  mtry  Accuracy   Kappa
  1     0.9600000  0.94 
  2     0.9666667  0.95 
  3     0.9666667  0.95 
Accuracy was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 2.

For the binary case I know how to compute the AUC: https://en.wikipedia.org/wiki/Receiver_operating_characteristic

So my questions are:

  1. How is the Accuracy computed for mutliclass prediction? (How is it condensed to a single value)

  2. Why is mtry=2 chosen, even if mtry=3 is equivalent according to the Accuracy?

  3. Why isn't it possible to compute the RMSE for a classification problem? (Throwing the error "Error: Metric RMSE not applicable for classification models" if metric="RMSE" is used. Furthermore, for multiclass prediction the RMSE is still defined - other then the AUC.) Since, the Brier Score is simply the MSE (for 2 classes), why not allowing to use the RMSE for probability predictions?

Many thanks in advance!

$\endgroup$
6
  • $\begingroup$ Thanks for downvoting without giving any reason - if the question is too obvious, share the link. If the question is unclear, give feedback. Otherwise, downvoting is not constructive! $\endgroup$
    – Jogi
    Commented Sep 28, 2017 at 17:04
  • $\begingroup$ Accuracy is the one performance metric that doesn't need to make any additional assumptions when there are multiple classes. Just count the proportion of correctly classified records. Of course, accuracy is inadequate when there is imbalance of misclassification costs. $\endgroup$ Commented Sep 28, 2017 at 17:45
  • $\begingroup$ @ David: So Accuracy is simply $\frac{\text{number of correct classified}}{N}$ where N is the sample size? But how on earth is the Accuracy equal amonth the different mtry? $\endgroup$
    – Jogi
    Commented Sep 28, 2017 at 17:57
  • $\begingroup$ Yes that's how it is defined $\endgroup$ Commented Sep 28, 2017 at 17:59
  • 1
    $\begingroup$ It would probably be best if you first read a chapter about classification performance metrics and ask remaining questions afterwards. There are clearly deeper issues worth questioning with those metrics, most of your questions would be answered in an introductory chapter though and have already been answered individually on this site as well. Since you like caret, maybe you could have a look at the book "Applied predictive modelling" which is written by the same author Max Kuhn. $\endgroup$ Commented Sep 28, 2017 at 20:59

1 Answer 1

1
$\begingroup$

The optimal model uses mtry=2 because it has found the inflection point in Accuracy (0.9666667), and at mtry=2 it requires less effort and achieves the same accuracy as mtry=3.

Also, you cannot compute Root Mean Squared Error on classification problems, because it doesn't make any sense to do so. RMSE is computed for regression problems, as they produce numerical predictions that have a distance metric defined for them -- you can compute the Error between a target data value and its prediction. No distance metric -- no RMSE (unless you define an error metric for your classification problem, in which case, this will not be a standard RMSE).

$\endgroup$
5
  • $\begingroup$ -Thanks for the answer. The code is completely reproducible (just plug the code into your R console) - except for the missing library, which I now added. The question requires the knowledge of the caret package, which then clears the confusion what trainControl does. Since I'm primarily interested in the formulas and mathematical background, I posted the question not in stackoverflow. Regarding "How is Area Under the Curve relevant to your questions?" - I thought that the Accuracy in caret gives the AUC. $\endgroup$
    – Jogi
    Commented Sep 28, 2017 at 17:54
  • 1
    $\begingroup$ This seems to be more like criticism of the clarity of the question which is fine for a comment or a vote to close but is not an answer. Maybe you don't have enough privilege yet but you should wait until you get it. $\endgroup$ Commented Sep 28, 2017 at 18:40
  • $\begingroup$ @ Taratango: Thanks for editing! I fully agree on your argument: since a regular random Forest outputs categories - no distance can be computed. But recall that I use a probability prediction tree. Therefore, I do have no longer nominal outputs but numericals. For these kind of problems - the Brier score is well defined. Also note that the Brier score is simply the mean squared error. So my point is - why not allowing to compute the RMSE for probability prediction trees? $\endgroup$
    – Jogi
    Commented Sep 28, 2017 at 19:40
  • $\begingroup$ @Jogi Although the score is found using the same procedure, the underlying assumptions are different. Regression values are not necessarily bounded from [0,1] like probabilities are. As long as the proper caveats are made, you should (theoretically) be able to use Brier score. You may have to use an external procedure to evaluate whether your mtry=2 or 3 model is best based on Brier score. It is likely that the authors of caret decided to keep classification and regression scoring separate. $\endgroup$
    – Tarantoga
    Commented Sep 28, 2017 at 20:09
  • $\begingroup$ Is there no stand alone accuracy function in caret?? $\endgroup$
    – user209249
    Commented Jan 8, 2019 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.