6
$\begingroup$

I built a detector to detect a binary outcome and then took a random sample from the population. From this, I can create a signal detection/confusion matrix (hit, miss, false alarm, correct rejection) [aka: TP, FP, FN, TN] and then calculate metrics such as Recall and Precision.

My question: How do you calculate confidence intervals for the Recall and Precision of the population from which I sampled?

I was thinking of this formula: p - z * sqrt(p*(1-p)/n) < p < p + z * sqrt(p*(1-p)/n)

where p = the statistic (e.g., Recall) and z = z-score for the desired confidence

$\endgroup$
1
$\begingroup$

An answer here suggests using bootstrapped statistics; we've done this at my place of employment and it seems to do the right thing.

Confidence interval for precision and recall in classification

$\endgroup$
  • 2
    $\begingroup$ Please include the details of the link in the answer to help avoid reads from having to click on the link. As it stands, this looks more like a comment. $\endgroup$ – Greenparker May 11 '16 at 0:48
  • 1
    $\begingroup$ The link that you posted doesn't have any answer now. $\endgroup$ – Aaron May 22 '18 at 19:00
1
$\begingroup$

The following approach might be more accurate and more efficient.

Goutte, C., & Gaussier, E. (2005, March). A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In European Conference on Information Retrieval (pp. 345-359). Springer Berlin Heidelberg.

$\endgroup$
  • 2
    $\begingroup$ in addition to providing a reference, please briefly explain why this approach would be better $\endgroup$ – Antoine Feb 13 '17 at 15:37
1
$\begingroup$

I'll summarise the approaches which are sketched by the other two answers.

  1. Suggest by @marsu. Assume that your confusion matrix $C$ has a multinomial distribution $M(n; π)$, then the distribution of the $TP$ is binomial. Assume a symmetric beta prior for precision $p$ and and recall $r$, that is $p,r ∼ Beta(λ, λ)$. Then given your data $D$ the posterior for $p$ is $p|D ∼ Beta(TP + \lambda, FP + \lambda)$ and $r$ is $r|D ∼ Beta(TP + \lambda, FN + \lambda)$. You can then use software to calculate the appropriate confidence interval as outlined here: Calculate the confidence interval for the mean of a beta distribution.

  2. Suggest by @fred. Generate $D_n$ datasets by sampling with replacement from your underlying dataset $D$. For each $D_n$ fit your classifier, and calculate the confusion matrix $C_n$. For each $C_n$ calculate precision $p_n$ and $r_n$, the confidence interval for these quantities can be calculated directly from the bootstrap distribution.

$\endgroup$
  • $\begingroup$ An alternative to the bootstrap approach in option (2) is to use the jackknife. For jackknife estimates of variance and standard errors of a given statistic (e.g. mean), see e.g. Efron, B. and Tibshirani, R.J., 1994. "An Introduction to the Bootstrap", p 141. Note that the jackknife should not be used for non-smooth statistics where a small change to the data could cause a large change in the statistic like for example in the median (or any other quantile). $\endgroup$ – mloning Oct 16 '18 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.