# How to assess statistical significance of the accuracy of a classifier?

I have the classifier accuracy output in percentage and the number of input samples. Is there any test that can tell how statistically significant is the result based on this information.

Thanks

-
Can you give an example? – Max Gordon Oct 22 '11 at 10:07
It's not clear to me what you have and what you are asking for. There are tests of whether a proportion is 0, but that's not a meaningful test for accuracy - accuracy of 0 would be perfect in a way - always wrong! – Peter Flom Oct 22 '11 at 10:28

You want to define the distribution of the accuracy of just guessing. Perhaps this is like $X/n$ where $X \sim$ binomial($n$, $p$) for some known $p$ (say 50%).

Then calculate the chance of observing the results you did, if this null model were true. In R, you could use binom.test or calculate it directly with pbinom.

Usually you'd want to compare accuracy not to "guessing" but to some alternative method, in which case you might use McNemar's test; in R, mcnemar.test.

-

I don't see where testing against complete randomness is that helpful. A classifier that can only beat pure random guesses is not very useful. A bigger problem is your use of proportion classified correctly as your accuracy score. This is a discontinuous improper scoring rule that can be easily manipulated because it is arbitrary and insensitive. One (of many) ways to see its deficiencies is to compute the proportion classified correctly if you have a model with only an intercept. It will be high if the outcomes are not close to 0.5 in prevalence.

Once you choose a more proper rule it would be valuable to compute a confidence interval for the index. Statistical significance is of little value.

-
 About of the proportion of classified correctly, do you mean the standard classification accuracy? thanks – Simone Oct 26 '11 at 0:15 Yes; a highly problematic measure. – Frank Harrell Oct 26 '11 at 1:27 Yes, it is an highly problematic measure. I agree with you. – Simone Oct 26 '11 at 8:55

For sure you can computer a confidence interval. If $\mbox{acc}$ is your accuracy estimated on a test set of $N$ elements, it holds that $$\frac{acc-p}{\sqrt{p(1-p)/N}} \sim \mathcal{N}(0,1)$$ Thus $$P\bigg( \frac{acc-p}{\sqrt{p(1-p)/N}} \in [-z_{\alpha/2},+z_{\alpha/2}]\bigg) \approx 1 - \alpha$$ So you can say that: $$P(p \in [l,u]) \approx 1 - \alpha$$ For example you can calculate the Wilson interval. $$l = \frac{2 \ N \ \mbox{acc} + z_{\alpha/2}^2 - z_{\alpha/2} \sqrt{z_{\alpha/2}^2+4 \ N \ \mbox{acc}-4 \ N \ \mbox{acc}^2}}{2(N+z_{\alpha/2}^2)}$$ $$u = \frac{2 \ N \ \mbox{acc} + z_{\alpha/2}^2 + z_{\alpha/2} \sqrt{z_{\alpha/2}^2+4 \ N \ \mbox{acc}-4 \ N \ \mbox{acc}^2}}{2(N+z_{\alpha/2}^2)}$$

I think you can calculate how much your performance differs from a random one computing the gain. The accuracy of a random classifier is: $$\mbox{acc}_r = \sum_{i=1}^{c} p_i^2$$ where $p_i$ is the empirical frequency of the class $i$ estimated on the test set, and $c$ is the number of different classes. On average a random classifier, which classifies random guessing the class $i$ relying on the priors probability of the test set, classifies $p_i\cdot n_i = \frac{n_i}{N} \cdot n_i$ examples of class $i$ correctly. Where $n_i$ is the number of records of class $i$ in the test set. Thus $$\mbox{acc}_r = \frac{p_1 \cdot n_1 + \dots + p_c \cdot n_c}{n_1 + \dots + n_c} = \frac{p_1\cdot n_1}{N} + \dots + \frac{p_c\cdot n_c}{N} = \sum_{i}^{c} p_i^2$$ You might have a look to a question of mine.

The gain is: $$\mbox{gain} = \frac{\mbox{acc}}{\mbox{acc}_r}$$

I actually think a statistical test can be sketched. The numerator could be seen as a Normal random variable, $\mathcal{N}(\mbox{acc},p(1-p)/N)$, but you should figure out what kind of random variable the denominator $\mbox{acc}_r$ could be.

-
 Again I'm not convinced that a statistical test against absolutely no predictive value is of value. – Frank Harrell Oct 26 '11 at 1:27

I think that one thing you could try out would be a permutation test. Simply put just randomly permute the input-desired output pairs you feed to your classifier over a number of times. If it fails to reproduce anything at the same level over 100 different permutations than it's significant at the 99% interval and so on. This is basically the same process used to obtain p-values (which correspond to the probability of obtaining a linear correlation of the same mangnitude after randomly permuting the data) and so on.

-
 Could you elaborate further what you meant for input/desired output pairs? – Simone Oct 26 '11 at 0:12