4
$\begingroup$

I have what I imagine is an elementary question about evaluating statistical significance, but while I know a lot about probability I can't t-test my way out of a paper bag. From here I'm hoping to get a pointer to where I should look for the answer.

I have a machine learning system and a test set of N questions. I run the test and the system gets r questions right and w questions wrong where r + w = N. I measure the performance of my system with its accuracy, which I define to be r/N.

I make a change to my machine learning system and rerun the same test. Now I get a different accuracy. Is is possible to ask whether or not the change in accuracy is statistically significant?

I suppose the null hypothesis is that the change in accuracy is just due to chance. The system is deterministic, so in theory there is no chance involved. However, certain questions will be hard ones that lie on its decision boundary, making them so sensitive to the system's configuration their correctness is essentially random. Is it meaningful to talk about statistical significance in this instance? If so, what test should I use?

Following up on the cross-validation suggestions in the comments below, would the following work?

I have a test data set of size N. It is divided into M disjoint partitions. To test my system, I calculate its accuracy on each of the M partitions. Then I can take the mean and standard deviation of this accuracy.

To compare the performance of two systems, I run them both on the same set of M partitions, and then see if the difference between the mean accuracies is statistically significant. Would I use Welch's Test for this?

Here the "randomness" in the accuracies arises from overfitting.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ doesn't this depend on how you determine accuracy? If you're using your training set to determine accuracy this is an issue. Similarly with split sample. But if you're using some form of cross-validation you might well obtain a distribution of OOB estimates from which to generate some sense of the reliability of estimate. (The key being that the training set accuracy is deterministic, but you're interested in the OOB accuracy that can only be estimated.) $\endgroup$
    – charles
    Commented Dec 12, 2013 at 3:40
  • $\begingroup$ Usual data hygiene rules apply. The training and evaluation sets are separate. I have enough data so that I can have a dedicated evaluation set instead of doing cross-validation. $\endgroup$
    – W.P. McNeill
    Commented Dec 12, 2013 at 16:59
  • $\begingroup$ Or are you saying that if I randomly chose multiple evaluation sets (e.g. via cross validation) I could get a distribution over accuracy numbers which I could then compare with a different distribution using standard techniques? $\endgroup$
    – W.P. McNeill
    Commented Dec 12, 2013 at 17:01
  • $\begingroup$ I might misunderstand. Sorry. But the type of graph/output you want might be (stanford.edu/~hastie/local.ftp/Springer/OLD/ESLII_print4.pdf) page 244, 314, 378,... I believe Hastie et al have even popularized a "one-standard deviation" (?) rule of thumb such that CV values within 1-SD are not considered different. How you practically obtain this information from CV I have no idea. $\endgroup$
    – charles
    Commented Dec 12, 2013 at 17:11
  • $\begingroup$ If you have a test set (rather than use CV of some sort), I have not idea. Since you'll just obtain a point estimate without any sort of idea of the distribution. (My assumption, but would be interested if someone knows more, is that this is part of the popularity of CV. Usually these researches have datasets that are large enough that split sampling could reasonably be used if they wanted to) $\endgroup$
    – charles
    Commented Dec 12, 2013 at 17:14

1 Answer 1

Reset to default
-1
$\begingroup$

You should look in to Area Under the ROC Curve as a method. You can use that as a metric for how well each method separates the datasets.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.