0
$\begingroup$

I have a very small sample (N=12) of results from a classification problem, and would like to compute the confidence interval around the precision and recall. The results of the classification are (with FNs omitted):

TP | 2  
FN | 9 
FP | 1

I'm wondering if for this problem it's better to use bootstrap resampling or an empirical CI (see also this question here). I have some questions on both approaches that I've outlined below:

(1) Bootstrap resampling, as suggested in e.g. 1

Resampling 10,000 times and computing precision leads to the distribution below:

bootstrapped precision

And then applying the percentile method (which I recently read should not be used generally), a 95% confidence interval comes out to 0-100%. Thus my question is, should I use a different (empirical) method for computing the CI, or should I forget the bootstrap altogether? See also this discussion on a similar issue here.

(2) Empirical, as suggested in e.g. 1 2. Based on this article it appears that Wilson's score interval with continuity correction is most appropriate for the small sample size, but I'm wondering if that's indeed the best choice.

Improve this question
$\endgroup$
4
  • $\begingroup$ Not only do you merely have 12 observations, only 1 observation represented a "control" meaning a true lack of disease, and they were a FP at that. So it's literally impossible to even get a point estimate of specificity because there is no TN. $\endgroup$
    – AdamO
    Jan 5 at 17:24
  • $\begingroup$ @AdamO the sample actually contains several TNs, but since I'm only interested in precision and recall I did not include them. $\endgroup$
    – turnerm
    Jan 6 at 8:44
  • 2
    $\begingroup$ If bootstrapping tells you that your CI is 0-100%, you should at least consider the possibility that this is the possibility that this is the right answer and that it's impossible to accurately estimate precision from this data. $\endgroup$
    – Eoin
    Jan 9 at 16:58
  • $\begingroup$ This sounds like it might benefit from a permutation test, but for that you would need to start with all cases including those that are true negatives, then evaluate all combinations of mappings of classes to cases. How many true negatives were there? Please edit the question to add that information, as comments are easy to overlook and can be deleted. Also, see this page for why evaluating your model via precision/recall might not be such a good choice. $\endgroup$
    – EdM
    Jan 10 at 19:29

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy