Model <- train(Movement~., tuneLength = 5, data = Database, method = 
                   trControl = trainControl(method = "cv",
                                            number = 10,
                                            savePredictions = "final",
                                            classProbs = T))
confusionMatrix(Model$pred[order(Model $pred$rowIndex),2], Database$Movement)

After running my code for my multiclass database (11 different classes), I receive the following results for Overall Statistics:

               Accuracy : 0.5808          
                 95% CI : (0.5649, 0.5965)
    No Information Rate : 0.1681          
    P-Value [Acc > NIR] : < 2.2e-16                                                
                  Kappa : 0.5301                                                 
 Mcnemar's Test P-Value : NA

but I do have more performance measures for each class

> Statistics by Class:
>                         Class AccelerationAg      Class: AccelerationNon   Class: BackGround
> Sensitivity                     0.91929825             0.00000000        0.83685801
> Specificity                     0.91368541             1.00000000        0.98676640
> Pos Pred Value                  0.46289753                    NaN        0.85758514
> Neg Pred Value                  0.99290342             0.92907801        0.98450057
> Precision                       0.46289753                     NA        0.85758514
> Recall                          0.91929825             0.00000000        0.83685801
> F1                              0.61574618                     NA        0.84709480
> Prevalence                      0.07486210             0.07092199        0.08694510
> Detection Rate                  0.06882059             0.00000000        0.07276070
> Detection Prevalence            0.14867350             0.00000000        0.08484371
> Balanced Accuracy               0.91649183             0.50000000        0.91181220

I am wondering how can I calculate other performance measures such as precision, recall, F1, sensitivity, etc for the overall database not by each class such as the accuracy and Kappa that I have already calculated for in the overall statistics. I mean, the confusionmatrix function is calculating everything that I need but by each class and I want those statistics for the overall (whole) database. For instance, as you can see above, the accuracy of my model is 0.5808 which indicates the overall accuracy of my database for prediction. I want to calculate the overall Recall, F1, Precision, Sensitivity, Specificity, etc. of my entire database not by class. I do appreciate your help in advance,

  • 1
    $\begingroup$ Unlike kappa or Accuracy, those are binary (aka class-specific aka one-hot) measures. There can be various ways to "pool" or combine them in one (to obtain the multiclass, or "overall", as you call it, measure), all ways are more or less artificial, I'd say. One way is to simply average the already computed class-wise values. Another way is to average every term in the formula of the measure and then to compute the measure by its formula. $\endgroup$
    – ttnphns
    Sep 6, 2020 at 0:33
  • $\begingroup$ @ttnphns Thank you for your answer. I am not sure if I have got it right, could please provide a code about what you mean?. Do you mean that I should calculate the average of for example Sensitivity among all of the classes?. $\endgroup$ Sep 6, 2020 at 0:49
  • $\begingroup$ Yes, for measures like Precision and Recall (Sensitivity) and Specificity it is natural just to average the class-wise values. F1 formula is based on Presision and Recall as its terms, so one would first average for the terms and then calculate F1. When averaging, class sizes or inverse class sizes could potentially be used as weights. $\endgroup$
    – ttnphns
    Sep 6, 2020 at 1:03
  • $\begingroup$ (This is my way. If you wish, you may want to open the "Compare partitions" zip on my web page, go to macro !Clasagree description in the .docx, to read about that in the "Binar" subcommand paragraph.) $\endgroup$
    – ttnphns
    Sep 6, 2020 at 1:08
  • $\begingroup$ Weighted average may be more informative than average if the classes are imbalanced, though it depends a bit on what you want. As @ttnphns mentioned, pooling these metrics is not really intuitive $\endgroup$
    – Elenchus
    Sep 6, 2020 at 1:28


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy