2
$\begingroup$

In computer science literature, we always see different algorithms are trained with a lot of data (n=100,000), and then they are tested on a test set (n=10,000). Then, often,if one algorithm NUMERICALLY is slightly bigger than another algorithm in term of accuracy (e.g., 79.1% vs 78.1%), then the author will claim his/her method is better (beat the state of the art). However, how do we know the difference in term of accuracy is significant enough?

In computer science literature, they often don't do statistical test. So, if I am in this situation, what should I do to compare the significance of the difference between two algorithms and what tests should be used?

Improve this question
$\endgroup$
4
  • $\begingroup$ Some papers, particularly pure ML papers do paired statistical hypothesis testing to test whether mean accuracy differs between the algorithms. An example would be paired t-test or Wilcoxon signed-rank test. You are certainly correct though that it is not a common practice for applied ML papers. $\endgroup$
    – Andrzej Pronobis
    Commented Oct 29, 2015 at 23:48
  • $\begingroup$ So, I need to do bootrapping to get multiple accuracies for same algorithm with different subset? $\endgroup$
    – RockTheStar
    Commented Oct 30, 2015 at 0:04
  • $\begingroup$ You might find this paper useful: dl.acm.org/citation.cfm?id=1248548 $\endgroup$
    – Andrzej Pronobis
    Commented Oct 30, 2015 at 0:23
  • $\begingroup$ Consider using Diebold-Mariano test if you can define a loss function (e.g. loss is proportional to absolute errors or squared error). See the original paper from 1990s and a recent reflection by Diebold here. $\endgroup$
    – Richard Hardy
    Commented Oct 30, 2015 at 11:09

1 Answer 1

Reset to default
2
$\begingroup$

You're using two different meanings of significance here.

how do we know the difference in term of accuracy is significant enough?

Here you're talking about practical impact, or effect size. What effect size is important? That really depends. If your financial model gives better results to the extent that you're earning an extra 100.000€/year, but switching models costs 1.000.000.000€, the 100.000€/year is not sufficiently better. If your machine learning system saves the lives of 1% of the treated people (by detecting the disease early) and the disease affects 1.000.000 people/year, it probably is an important effect.

Note that you're talking here about degrees of significance; something can be more or less significant than something else.

statistical test

Now this is a completely different question. Typically, statistical tests give only binary results, they're only compared to a pre-defined threshold (e.g. the famous p < 0.05). And crossing that threshold does not mean that surely, practical significance will be achieved. Instead, it just means that the signal you are observing is so much stronger than the background noise that you can with confidence assume there is a signal.

This signal may have zero practical importance, but it seems to be there.

Now you can do significance tests to compare classifiers and machine learning models. This depends on the type of model, and by what measure you want to evaluate them. Very often, bootstrapping is indeed used here. However, make sure you're truly interested in significance in the statistical sense, not the practical, effect size sense.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the feedback! I see, so if I am interested in the statistical significance, then I should do the test, right? Then, isn't it all the CS machine learning algorithm should do statistical significance test too because we are not sure if the performance of one classifier is really better than another? $\endgroup$
    – RockTheStar
    Commented Oct 30, 2015 at 7:30
  • $\begingroup$ Not necessarily, because again, a classifier could be much better than another without being statistically significantly better, or it could be statistically significantly better without being truly better. $\endgroup$
    – jona
    Commented Oct 30, 2015 at 9:52
  • $\begingroup$ Hmm... that is a problem....then does it mean we don't know the truth? And If a classifier could be much better than another, then it is high likely that it should be statistically significantly better? $\endgroup$
    – RockTheStar
    Commented Oct 30, 2015 at 16:54
  • $\begingroup$ What statistical significance testing can tell you is estimate how confident you can be of at least a certain distance (whatever you choose) between two models. If this difference has practical significance is a different question. $\endgroup$
    – jona
    Commented Oct 30, 2015 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.