How to assess statistical significance of the accuracy of a classifier?

Question

I have the classifier accuracy output in percentage and the number of input samples. Is there any test that can tell how statistically significant is the result based on this information.

Thanks

Answer 1

You want to define the distribution of the accuracy of just guessing. Perhaps this is like $X/n$ where $X \sim $ binomial($n$, $p$) for some known $p$ (say 50%).

Then calculate the chance of observing the results you did, if this null model were true. In R, you could use binom.test or calculate it directly with pbinom.

Usually you'd want to compare accuracy not to "guessing" but to some alternative method, in which case you might use McNemar's test; in R, mcnemar.test.

Answer 2

I don't see where testing against complete randomness is that helpful. A classifier that can only beat pure random guesses is not very useful. A bigger problem is your use of proportion classified correctly as your accuracy score. This is a discontinuous improper scoring rule that can be easily manipulated because it is arbitrary and insensitive. One (of many) ways to see its deficiencies is to compute the proportion classified correctly if you have a model with only an intercept. It will be high if the outcomes are not close to 0.5 in prevalence.

Once you choose a more proper rule it would be valuable to compute a confidence interval for the index. Statistical significance is of little value.

Answer 3

For sure you can computer a confidence interval. If $\mbox{acc}$ is your accuracy estimated on a test set of $N$ elements, it holds that $$\frac{acc-p}{\sqrt{p(1-p)/N}} \sim \mathcal{N}(0,1)$$ Thus $$ P\bigg( \frac{acc-p}{\sqrt{p(1-p)/N}} \in [-z_{\alpha/2},+z_{\alpha/2}]\bigg) \approx 1 - \alpha$$ So you can say that: $$P(p \in [l,u]) \approx 1 - \alpha$$ For example you can calculate the Wilson interval. $$l = \frac{2 \ N \ \mbox{acc} + z_{\alpha/2}^2 - z_{\alpha/2} \sqrt{z_{\alpha/2}^2+4 \ N \ \mbox{acc}-4 \ N \ \mbox{acc}^2}}{2(N+z_{\alpha/2}^2)}$$ $$u = \frac{2 \ N \ \mbox{acc} + z_{\alpha/2}^2 + z_{\alpha/2} \sqrt{z_{\alpha/2}^2+4 \ N \ \mbox{acc}-4 \ N \ \mbox{acc}^2}}{2(N+z_{\alpha/2}^2)}$$

I think you can calculate how much your performance differs from a random one computing the gain. The accuracy of a random classifier is: $$ \mbox{acc}_r = \sum_{i=1}^{c} p_i^2$$ where $p_i$ is the empirical frequency of the class $i$ estimated on the test set, and $c$ is the number of different classes. On average a random classifier, which classifies random guessing the class $i$ relying on the priors probability of the test set, classifies $p_i\cdot n_i = \frac{n_i}{N} \cdot n_i$ examples of class $i$ correctly. Where $n_i$ is the number of records of class $i$ in the test set. Thus $$ \mbox{acc}_r = \frac{p_1 \cdot n_1 + \dots + p_c \cdot n_c}{n_1 + \dots + n_c} = \frac{p_1\cdot n_1}{N} + \dots + \frac{p_c\cdot n_c}{N} = \sum_{i}^{c} p_i^2$$ You might have a look to a question of mine.

The gain is: $$\mbox{gain} = \frac{\mbox{acc}}{\mbox{acc}_r} $$

I actually think a statistical test can be sketched. The numerator could be seen as a Normal random variable, $\mathcal{N}(\mbox{acc},p(1-p)/N)$, but you should figure out what kind of random variable the denominator $\mbox{acc}_r$ could be.

Answer 4

I think that one thing you could try out would be a permutation test. Simply put just randomly permute the input-desired output pairs you feed to your classifier over a number of times. If it fails to reproduce anything at the same level over 100 different permutations than it's significant at the 99% interval and so on. This is basically the same process used to obtain p-values (which correspond to the probability of obtaining a linear correlation of the same mangnitude after randomly permuting the data) and so on.