I have a 12% holdout cross validation and I've done a bunch of repetitions which gives me a distribution of scores for each method.

I'd like to do some sort of hypothesis testing for comparing methods. Each score I have is unitary value between 0 and 1. A view at a histogram yields an approximately normal distribution.

Could a t-test be valid in this scenario?

  • $\begingroup$ Can you clarify what you did a bit more? What do you mean by a bunch of repetitions. How many methods do you want to compare? What histogram is approximately normal? $\endgroup$ – Joel W. Jul 18 '12 at 0:50
  • $\begingroup$ Depending on the setup, there may be more suitable comparisons. For example, if the score was a predicted probability, you can construct AUC and confidence intervals and compare. At the same time, if we test several different models and pick the best, there is a risk of overfitting. $\endgroup$ – julieth Jul 18 '12 at 1:51

I would say no for several reasons: 1. the individual scores are not independent because of sample reuse 2. the distribution is confined to [0,1] so it is truncated and not normal (could be approximately normal though if truncation is not too great) 3, Saying "some sort of hypothesis test" doesn't tell us what you want to do. How many methods are you comparing ? If it is more than 2 are you comparing them pairwise? If one method has a higher average score what does that tell you? Maybe a nonparametric ANOVA is really what you need.


If you have two different models and the generalized classification performance of each model is being estimated using m repetitions of k-fold validation. Assuming you are computing a performance metric (e.g. classification accuracy) for each fold of cross-validation, the classification accuracy is then the mean value across all test set folds and repetitions while the 95% confidence interval can be computed from the set of performance metrics obtained from each fold (m*n values).

You can then use a hypothesis test (such as a t-test) to determine if the two sets of performance metrics obtained from each fold of cross-validation are significantly different.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.