0
$\begingroup$

Classification accuracy or classification error is a proportion or a ratio. It describes the proportion of correct or incorrect predictions made by the model. Each prediction is a binary decision that could be correct or incorrect which is a Bernoulli trial.

The proportions in a Bernoulli trial have a binomial distribution. With large sample sizes (e.g. more than 30), we can approximate the distribution with a Gaussian.

We can use the assumption of a Gaussian distribution of the proportion to calculate the 95% confidence interval for accuracy as:

radius = z * sqrt( (accuracy * (1 - accuracy)) / n)

where n is the size of the dataset, and z is the number of standard deviations (z = 1.96 for 95% interval) from the Gaussian distribution.

Can we use the same approach to calculate precision and recall, that are also eventually based on proportion of correct predictions ?

$\endgroup$
6
  • 1
    $\begingroup$ I doubt that the Gaussian interval you give for the accuracy is valid. Classification accuracy is based on a certain classification method applied to all training observations at once. Arguably the involved indicators are all dependent (even in a test sample, as you could expect similar misclassification probability for close observations in data space), and this is ignored by standard Bernoulli or Gaussian confidence intervals. A similar issue will apply to precision and recall. $\endgroup$ Commented Feb 7, 2023 at 17:13
  • $\begingroup$ Yes. In principle the classification of any observation may depend (probabilistically, not perfectly) on the classification of any other observation. $\endgroup$ Commented Feb 7, 2023 at 17:36
  • $\begingroup$ @ Christian Hennig So what you say is that predicting True or False for the class is not a Bernoulli trial? $\endgroup$
    – dokondr
    Commented Feb 7, 2023 at 17:41
  • 2
    $\begingroup$ In general this depends on how you exactly do that. I'm saying it's not a Bernoulli trial if the prediction is done by a method that also relies on other observations, as this introduces dependence. $\endgroup$ Commented Feb 7, 2023 at 17:50
  • $\begingroup$ @ Christian Hennig What about precision and recall? The same as with accuracy, I think. Or any additional restrictions? $\endgroup$
    – dokondr
    Commented Feb 7, 2023 at 20:17

1 Answer 1

0
$\begingroup$

You can use the Gaussian interval for calculating the confidence interval if you have a large enough N.

This is the suggested equation for accuracy: enter image description here

I don't see why the same shouldn't apply to recall, precision etc.

equation source: "Introduction to Data Mining" by Pang-Nin Tan, Michael Steinbach... pg 190. This is specifically for classification confidence intervals

$\endgroup$
1
  • $\begingroup$ Thanks for the answer. It may be edit your answer to include LaTeX formatting. $\endgroup$ Commented Nov 1, 2023 at 3:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.