# Confidence interval for Accuracy, Precision and Recall

Classification accuracy or classification error is a proportion or a ratio. It describes the proportion of correct or incorrect predictions made by the model. Each prediction is a binary decision that could be correct or incorrect which is a Bernoulli trial.

The proportions in a Bernoulli trial have a binomial distribution. With large sample sizes (e.g. more than 30), we can approximate the distribution with a Gaussian.

We can use the assumption of a Gaussian distribution of the proportion to calculate the 95% confidence interval for accuracy as:

radius = z * sqrt( (accuracy * (1 - accuracy)) / n)

where n is the size of the dataset, and z is the number of standard deviations (z = 1.96 for 95% interval) from the Gaussian distribution.

Can we use the same approach to calculate precision and recall, that are also eventually based on proportion of correct predictions ?

• I doubt that the Gaussian interval you give for the accuracy is valid. Classification accuracy is based on a certain classification method applied to all training observations at once. Arguably the involved indicators are all dependent (even in a test sample, as you could expect similar misclassification probability for close observations in data space), and this is ignored by standard Bernoulli or Gaussian confidence intervals. A similar issue will apply to precision and recall. Commented Feb 7, 2023 at 17:13
• Yes. In principle the classification of any observation may depend (probabilistically, not perfectly) on the classification of any other observation. Commented Feb 7, 2023 at 17:36
• @ Christian Hennig So what you say is that predicting True or False for the class is not a Bernoulli trial? Commented Feb 7, 2023 at 17:41
• In general this depends on how you exactly do that. I'm saying it's not a Bernoulli trial if the prediction is done by a method that also relies on other observations, as this introduces dependence. Commented Feb 7, 2023 at 17:50
• @ Christian Hennig What about precision and recall? The same as with accuracy, I think. Or any additional restrictions? Commented Feb 7, 2023 at 20:17