How to assess statistical significance of the accuracy of a classifier?

I have the classifier accuracy output in percentage and the number of input samples. Is there any test that can tell how statistically significant is the result based on this information.

Thanks

• Can you give an example? Commented Oct 22, 2011 at 10:07
• It's not clear to me what you have and what you are asking for. There are tests of whether a proportion is 0, but that's not a meaningful test for accuracy - accuracy of 0 would be perfect in a way - always wrong! Commented Oct 22, 2011 at 10:28

You want to define the distribution of the accuracy of just guessing. Perhaps this is like $X/n$ where $X \sim$ binomial($n$, $p$) for some known $p$ (say 50%).

Then calculate the chance of observing the results you did, if this null model were true. In R, you could use binom.test or calculate it directly with pbinom.

Usually you'd want to compare accuracy not to "guessing" but to some alternative method, in which case you might use McNemar's test; in R, mcnemar.test.

I don't see where testing against complete randomness is that helpful. A classifier that can only beat pure random guesses is not very useful. A bigger problem is your use of proportion classified correctly as your accuracy score. This is a discontinuous improper scoring rule that can be easily manipulated because it is arbitrary and insensitive. One (of many) ways to see its deficiencies is to compute the proportion classified correctly if you have a model with only an intercept. It will be high if the outcomes are not close to 0.5 in prevalence.

Once you choose a more proper rule it would be valuable to compute a confidence interval for the index. Statistical significance is of little value.

• About of the proportion of classified correctly, do you mean the standard classification accuracy? thanks Commented Oct 26, 2011 at 0:15
• Yes; a highly problematic measure. Commented Oct 26, 2011 at 1:27
• Yes, it is an highly problematic measure. I agree with you. Commented Oct 26, 2011 at 8:55
• Classifiers that just barely beat random guessing can be extremely useful in some situations. Thus, having some test that quantifies confidence in a classifier being better than chance is also useful.
– ely
Commented Dec 12, 2013 at 18:05
• Classifiers are sometimes used to assess whether a certain variable (or set of variables) contains information about another variable. For instance, in the context of analysis of neuroimaging data, a classifier might be used to determine whether the brain signals from a certain cortical region contain information about some aspect of a visual stimulus that the person was exposed to, which might suggest that this cortical region is involved in processing such information. In this context, a significance test against chance-level performance is very relevant. Commented Jun 13 at 11:48

For sure you can computer a confidence interval. If $\mbox{acc}$ is your accuracy estimated on a test set of $N$ elements, it holds that $$\frac{acc-p}{\sqrt{p(1-p)/N}} \sim \mathcal{N}(0,1)$$ Thus $$P\bigg( \frac{acc-p}{\sqrt{p(1-p)/N}} \in [-z_{\alpha/2},+z_{\alpha/2}]\bigg) \approx 1 - \alpha$$ So you can say that: $$P(p \in [l,u]) \approx 1 - \alpha$$ For example you can calculate the Wilson interval. $$l = \frac{2 \ N \ \mbox{acc} + z_{\alpha/2}^2 - z_{\alpha/2} \sqrt{z_{\alpha/2}^2+4 \ N \ \mbox{acc}-4 \ N \ \mbox{acc}^2}}{2(N+z_{\alpha/2}^2)}$$ $$u = \frac{2 \ N \ \mbox{acc} + z_{\alpha/2}^2 + z_{\alpha/2} \sqrt{z_{\alpha/2}^2+4 \ N \ \mbox{acc}-4 \ N \ \mbox{acc}^2}}{2(N+z_{\alpha/2}^2)}$$

I think you can calculate how much your performance differs from a random one computing the gain. The accuracy of a random classifier is: $$\mbox{acc}_r = \sum_{i=1}^{c} p_i^2$$ where $p_i$ is the empirical frequency of the class $i$ estimated on the test set, and $c$ is the number of different classes. On average a random classifier, which classifies random guessing the class $i$ relying on the priors probability of the test set, classifies $p_i\cdot n_i = \frac{n_i}{N} \cdot n_i$ examples of class $i$ correctly. Where $n_i$ is the number of records of class $i$ in the test set. Thus $$\mbox{acc}_r = \frac{p_1 \cdot n_1 + \dots + p_c \cdot n_c}{n_1 + \dots + n_c} = \frac{p_1\cdot n_1}{N} + \dots + \frac{p_c\cdot n_c}{N} = \sum_{i}^{c} p_i^2$$ You might have a look to a question of mine.

The gain is: $$\mbox{gain} = \frac{\mbox{acc}}{\mbox{acc}_r}$$

I actually think a statistical test can be sketched. The numerator could be seen as a Normal random variable, $\mathcal{N}(\mbox{acc},p(1-p)/N)$, but you should figure out what kind of random variable the denominator $\mbox{acc}_r$ could be.

• Again I'm not convinced that a statistical test against absolutely no predictive value is of value. Commented Oct 26, 2011 at 1:27
• Classifiers that just barely beat random guessing can be extremely useful in some situations. Thus, having some test that quantifies confidence in a classifier being better than chance is also useful.
– ely
Commented Dec 12, 2013 at 18:09
• In the vast majority of situations we want to know how well a prediction discriminates, not just whether it discriminates better than random chance. Commented Dec 12, 2013 at 18:29
• Not if you are boosting a bunch of weak classifiers, which is a very common activity. You may care about discrimination once you reach the fully boosted final classifier, but there's a lot of work between the start and the finish, and demonstrating that a complicated classifier empirically performs better than chance is important.
– ely
Commented Dec 12, 2013 at 19:17
• And some application domains, say financial markets, where you get to use the classifier in many many roughly independent cases, just being a bit better than chance (R-squared's of like 11% or 12% are considered great) can mean a lot. In those cases, if even the boosted classifier has R-squared of 15% that might be considered very good -- in which case it really matters if you can statistically resolve whether the weak classifiers are definitely better than guessing.
– ely
Commented Dec 12, 2013 at 19:20

You may be interested in the following papers:

• Eric W. Noreen, Computer-intensive Methods for Testing Hypotheses: An Introduction, John Wiley & Sons, New York, NY, USA, 1989.
• Alexander Yeh, More accurate tests for the statistical significance of result differences, in: Proceedings of the 18th International Conference on Computational Linguistics, Volume 2, pages 947-953, 2000.

I think they cover what Dimitrios Athanasakis talks about.

I implemented one option of Yeh in the manner that I understand it:

http://www.clips.uantwerpen.be/~vincent/software#art

I think that one thing you could try out would be a permutation test. Simply put just randomly permute the input-desired output pairs you feed to your classifier over a number of times. If it fails to reproduce anything at the same level over 100 different permutations than it's significant at the 99% interval and so on. This is basically the same process used to obtain p-values (which correspond to the probability of obtaining a linear correlation of the same mangnitude after randomly permuting the data) and so on.

• Could you elaborate further what you meant for input/desired output pairs? Commented Oct 26, 2011 at 0:12