Confidence interval for a macro/micro average

Question

I'm working on a supervised multi class classifier that labels texts according to three possible classes. I calculated the one-vs-all precision, recall and F1 score for each class and the macro averaged version of those metrics. How can I calculate the 95% confidence interval for the macro averages?

Below is the confusion matrix.

           refClass1    refClass2 refClass3  
predClass1   1070         33        2  
predClass2   54           937       18
predClass3   13           14        802

            macro_avg
precision   0.95479955
recall      0.956325957
F1          0.955562144

Thank you very much for the input!

Answer 1

The standard answer on this one is to bootstrap. I.e. repeatedly draw your total number of records with replacement and then calculate the metric of interest on each of these bootstrapped datasets. Then you calculate a bootstrap confidence interval (either the simplistic percentile one such as taking the 10% and 90% percentile of the metrics across the datasets, or something fancier like the BCa confidence interval).

Answer 2

Just to update this question with some asymptotic results. There has been some relatively recent papers on the subject if someone wants to look into this further. Two prominent ones are: Estimating the Uncertainty of Average $F_1$ Scores by Zhang et al. (2015) and Confidence interval for micro-averaged $F_1$ and macro-averaged $F_1$ scores by Takahashi et al. (2021). In general, these papers are based on Goutte & Gaussier (2005) A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation where we view the whole confusion matrix as the sample realisation from a multinomial distribution. Especially the Takahashi et al. paper does what you describe and provide R code for it. The exact formula for the variance calculations (and subsequently for the CIs) are somewhat lenghty but relatively straightforward if we want to re-implement in another language.