8
$\begingroup$

I have the classifier accuracy output in percentage and the number of input samples. Is there any test that can tell how statistically significant is the result based on this information.

Thanks

$\endgroup$
2
  • $\begingroup$ Can you give an example? $\endgroup$ – Max Gordon Oct 22 '11 at 10:07
  • 3
    $\begingroup$ It's not clear to me what you have and what you are asking for. There are tests of whether a proportion is 0, but that's not a meaningful test for accuracy - accuracy of 0 would be perfect in a way - always wrong! $\endgroup$ – Peter Flom Oct 22 '11 at 10:28
10
$\begingroup$

You want to define the distribution of the accuracy of just guessing. Perhaps this is like $X/n$ where $X \sim $ binomial($n$, $p$) for some known $p$ (say 50%).

Then calculate the chance of observing the results you did, if this null model were true. In R, you could use binom.test or calculate it directly with pbinom.

Usually you'd want to compare accuracy not to "guessing" but to some alternative method, in which case you might use McNemar's test; in R, mcnemar.test.

$\endgroup$
6
$\begingroup$

I don't see where testing against complete randomness is that helpful. A classifier that can only beat pure random guesses is not very useful. A bigger problem is your use of proportion classified correctly as your accuracy score. This is a discontinuous improper scoring rule that can be easily manipulated because it is arbitrary and insensitive. One (of many) ways to see its deficiencies is to compute the proportion classified correctly if you have a model with only an intercept. It will be high if the outcomes are not close to 0.5 in prevalence.

Once you choose a more proper rule it would be valuable to compute a confidence interval for the index. Statistical significance is of little value.

$\endgroup$
4
  • $\begingroup$ About of the proportion of classified correctly, do you mean the standard classification accuracy? thanks $\endgroup$ – Simone Oct 26 '11 at 0:15
  • 1
    $\begingroup$ Yes; a highly problematic measure. $\endgroup$ – Frank Harrell Oct 26 '11 at 1:27
  • $\begingroup$ Yes, it is an highly problematic measure. I agree with you. $\endgroup$ – Simone Oct 26 '11 at 8:55
  • 2
    $\begingroup$ Classifiers that just barely beat random guessing can be extremely useful in some situations. Thus, having some test that quantifies confidence in a classifier being better than chance is also useful. $\endgroup$ – ely Dec 12 '13 at 18:05
3
$\begingroup$

For sure you can computer a confidence interval. If $\mbox{acc}$ is your accuracy estimated on a test set of $N$ elements, it holds that $$\frac{acc-p}{\sqrt{p(1-p)/N}} \sim \mathcal{N}(0,1)$$ Thus $$ P\bigg( \frac{acc-p}{\sqrt{p(1-p)/N}} \in [-z_{\alpha/2},+z_{\alpha/2}]\bigg) \approx 1 - \alpha$$ So you can say that: $$P(p \in [l,u]) \approx 1 - \alpha$$ For example you can calculate the Wilson interval. $$l = \frac{2 \ N \ \mbox{acc} + z_{\alpha/2}^2 - z_{\alpha/2} \sqrt{z_{\alpha/2}^2+4 \ N \ \mbox{acc}-4 \ N \ \mbox{acc}^2}}{2(N+z_{\alpha/2}^2)}$$ $$u = \frac{2 \ N \ \mbox{acc} + z_{\alpha/2}^2 + z_{\alpha/2} \sqrt{z_{\alpha/2}^2+4 \ N \ \mbox{acc}-4 \ N \ \mbox{acc}^2}}{2(N+z_{\alpha/2}^2)}$$

I think you can calculate how much your performance differs from a random one computing the gain. The accuracy of a random classifier is: $$ \mbox{acc}_r = \sum_{i=1}^{c} p_i^2$$ where $p_i$ is the empirical frequency of the class $i$ estimated on the test set, and $c$ is the number of different classes. On average a random classifier, which classifies random guessing the class $i$ relying on the priors probability of the test set, classifies $p_i\cdot n_i = \frac{n_i}{N} \cdot n_i$ examples of class $i$ correctly. Where $n_i$ is the number of records of class $i$ in the test set. Thus $$ \mbox{acc}_r = \frac{p_1 \cdot n_1 + \dots + p_c \cdot n_c}{n_1 + \dots + n_c} = \frac{p_1\cdot n_1}{N} + \dots + \frac{p_c\cdot n_c}{N} = \sum_{i}^{c} p_i^2$$ You might have a look to a question of mine.

The gain is: $$\mbox{gain} = \frac{\mbox{acc}}{\mbox{acc}_r} $$

I actually think a statistical test can be sketched. The numerator could be seen as a Normal random variable, $\mathcal{N}(\mbox{acc},p(1-p)/N)$, but you should figure out what kind of random variable the denominator $\mbox{acc}_r$ could be.

$\endgroup$
5
  • 3
    $\begingroup$ Again I'm not convinced that a statistical test against absolutely no predictive value is of value. $\endgroup$ – Frank Harrell Oct 26 '11 at 1:27
  • 3
    $\begingroup$ Classifiers that just barely beat random guessing can be extremely useful in some situations. Thus, having some test that quantifies confidence in a classifier being better than chance is also useful. $\endgroup$ – ely Dec 12 '13 at 18:09
  • 1
    $\begingroup$ In the vast majority of situations we want to know how well a prediction discriminates, not just whether it discriminates better than random chance. $\endgroup$ – Frank Harrell Dec 12 '13 at 18:29
  • 1
    $\begingroup$ Not if you are boosting a bunch of weak classifiers, which is a very common activity. You may care about discrimination once you reach the fully boosted final classifier, but there's a lot of work between the start and the finish, and demonstrating that a complicated classifier empirically performs better than chance is important. $\endgroup$ – ely Dec 12 '13 at 19:17
  • 2
    $\begingroup$ And some application domains, say financial markets, where you get to use the classifier in many many roughly independent cases, just being a bit better than chance (R-squared's of like 11% or 12% are considered great) can mean a lot. In those cases, if even the boosted classifier has R-squared of 15% that might be considered very good -- in which case it really matters if you can statistically resolve whether the weak classifiers are definitely better than guessing. $\endgroup$ – ely Dec 12 '13 at 19:20
1
$\begingroup$

You may be interested in the following papers:

  • Eric W. Noreen, Computer-intensive Methods for Testing Hypotheses: An Introduction, John Wiley & Sons, New York, NY, USA, 1989.
  • Alexander Yeh, More accurate tests for the statistical significance of result differences, in: Proceedings of the 18th International Conference on Computational Linguistics, Volume 2, pages 947-953, 2000.

I think they cover what Dimitrios Athanasakis talks about.

I implemented one option of Yeh in the manner that I understand it:

http://www.clips.uantwerpen.be/~vincent/software#art

$\endgroup$
0
$\begingroup$

I think that one thing you could try out would be a permutation test. Simply put just randomly permute the input-desired output pairs you feed to your classifier over a number of times. If it fails to reproduce anything at the same level over 100 different permutations than it's significant at the 99% interval and so on. This is basically the same process used to obtain p-values (which correspond to the probability of obtaining a linear correlation of the same mangnitude after randomly permuting the data) and so on.

$\endgroup$
1
  • $\begingroup$ Could you elaborate further what you meant for input/desired output pairs? $\endgroup$ – Simone Oct 26 '11 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.