I iterated my 10-fold cross validation 100 times for several methods. Now I want to use a t-test to test if the results are significant. However, I'm not sure what the sample size is. Is the sample size the original amount of samples, or is it the original amount of samples x 100?


For university we need to classify 3 cancer types and give an estimation of how well our model will perform. We received a dataset with 100 samples. We split the data up into a training and test set using stratified sampling with a ratio of 0.3 and 0.7. The resulting training set consists of 69 samples, and the test set out of 31 samples.

We used repeated cross validation because of this paper: http://www.cse.iitb.ac.in/~tarung/smt/papers_ppt/ency-cross-validation.pdf

The repeated cross-validation is done on the same training set, but with the folds are randomly chosen every time, so they should be different every time.

The significance we want to test is if the accuracy of one model is significantly better than the accuracy of a different model.

  • $\begingroup$ I think you need to provide more detail for us to be able to answer properly. You repeated cross-validation 100 times? Why? Is this on simulated data, or did you reuse the same dataset each time? Then you want to test if "the results" are significant. What results? And given answers to the above: are you sure your observations are independent? $\endgroup$
    – Nick Sabbe
    Apr 25, 2013 at 7:57
  • $\begingroup$ I added the extra info under the edit $\endgroup$
    – Niek
    Apr 25, 2013 at 8:33

1 Answer 1


I think you can make such estimation. since different model use the same dataset, so the accuracy can be used to be compared. However, one important question, maybe, you need take the parameters of the models. whether these parameter will influence your conclusion.

  • $\begingroup$ Welcome to the site, @ShichengGuo. Can you clarify your answer? For example, what does it mean that the OP needs to "take the parameters of the models. whether these parameter will influence your conclusion"? How should the OP try to do this? $\endgroup$ Feb 18, 2014 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.