I iterated my 10-fold cross validation 100 times for several methods. Now I want to use a t-test to test if the results are significant. However, I'm not sure what the sample size is. Is the sample size the original amount of samples, or is it the original amount of samples x 100?


For university we need to classify 3 cancer types and give an estimation of how well our model will perform. We received a dataset with 100 samples. We split the data up into a training and test set using stratified sampling with a ratio of 0.3 and 0.7. The resulting training set consists of 69 samples, and the test set out of 31 samples.

We used repeated cross validation because of this paper: http://www.cse.iitb.ac.in/~tarung/smt/papers_ppt/ency-cross-validation.pdf

The repeated cross-validation is done on the same training set, but with the folds are randomly chosen every time, so they should be different every time.

The significance we want to test is if the accuracy of one model is significantly better than the accuracy of a different model.

  • $\begingroup$ I think you need to provide more detail for us to be able to answer properly. You repeated cross-validation 100 times? Why? Is this on simulated data, or did you reuse the same dataset each time? Then you want to test if "the results" are significant. What results? And given answers to the above: are you sure your observations are independent? $\endgroup$ – Nick Sabbe Apr 25 '13 at 7:57
  • $\begingroup$ I added the extra info under the edit $\endgroup$ – Niek Apr 25 '13 at 8:33

I think you can make such estimation. since different model use the same dataset, so the accuracy can be used to be compared. However, one important question, maybe, you need take the parameters of the models. whether these parameter will influence your conclusion.

  • $\begingroup$ Welcome to the site, @ShichengGuo. Can you clarify your answer? For example, what does it mean that the OP needs to "take the parameters of the models. whether these parameter will influence your conclusion"? How should the OP try to do this? $\endgroup$ – gung Feb 18 '14 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.