# Uncertainty in categorical statistics such as accuracy, sensitivity, specificity

The samples I've drawn (from a larger population) can be categorised into several classes (false positive, false negative, true positive, true negative). I'm interested in inferring various statistics concerning the population (such as accuracy, sensitivity, specificity and false discovery rate). However, I also want to know whether the results are statistically significant.

My population statistics can be estimated from class proportions in the sample set. For example:

$$\textrm{sensitivity} = \frac{\textrm{true positives}} {\textrm{true positives } + \textrm{ false negatives}}$$

The problem is that this value may be uninformative if some of the classes have very few samples.

This seems reminiscent of the hypothetical example of tossing a coin several times to estimate the toss bias (i.e. the population ratio of two classes, heads and tails). A Bayesian approach, assuming a uniform prior over all possible values for the bias, allows for calculating the posterior likelihood distribution for the bias, based on the number of previous samples identified to each class. In that case, the distribution mode is the simple estimate (heads $$\div$$ total) but one can also calculate a mean and a variance (or even confidence intervals) in order to express a result with error bars, e.g.:

$$\frac h {h+t} \rightarrow \frac {h+1} {h+t+2} \pm \sqrt{\frac {h+1} {h+t+2} \left( \frac {h+2} {h+t+3} - \frac {h+1} {h+t+2} \right) }$$

How should the problem (of uncertain significance) be addressed in the actual case, given that there are not two but four different classes? Is it valid to simply group together or exclude classes (however necessary according to the particular statistic) to reduce to the two-class case? (For example: to compute accuracy, replace "heads" with the union of true positives and true negatives, and replace "tails" with the union of falses; and to compute sensitivity then discard false positives and true negatives entirely?)

Also, what if there are actually more categories (e.g. not just positive and negative, but say seven different base classes, resulting in $$7^2$$ subtypes accounting for miss-classification) each with a different prevalence in the population. Does some form of multiple-comparison correction need to be incorporated?