I have what I imagine is an elementary question about evaluating statistical significance, but while I know a lot about probability I can't t-test my way out of a paper bag. From here I'm hoping to get a pointer to where I should look for the answer.
I have a machine learning system and a test set of N questions. I run the test and the system gets r questions right and w questions wrong where r + w = N. I measure the performance of my system with its accuracy, which I define to be r/N.
I make a change to my machine learning system and rerun the same test. Now I get a different accuracy. Is is possible to ask whether or not the change in accuracy is statistically significant?
I suppose the null hypothesis is that the change in accuracy is just due to chance. The system is deterministic, so in theory there is no chance involved. However, certain questions will be hard ones that lie on its decision boundary, making them so sensitive to the system's configuration their correctness is essentially random. Is it meaningful to talk about statistical significance in this instance? If so, what test should I use?
Following up on the cross-validation suggestions in the comments below, would the following work?
I have a test data set of size N. It is divided into M disjoint partitions. To test my system, I calculate its accuracy on each of the M partitions. Then I can take the mean and standard deviation of this accuracy.
To compare the performance of two systems, I run them both on the same set of M partitions, and then see if the difference between the mean accuracies is statistically significant. Would I use Welch's Test for this?
Here the "randomness" in the accuracies arises from overfitting.