I benchmark three algrothims that construct regression models. To bechmark them I use cross-validation, so I obtain a matrix of errors (mean squared errors for example) with rows is number of cross-validation split of the data to training and test, and columns refer to different algorithms.

I want to find if difference between algorithms errors are statistically significant. What is the correct statistical test to use? Should I use Z-score as provided in Comparing different classifiers (using ski-kit cross validation values) or there is a better option for regression?

Anecdotical evidence for usage of such tests in top journal articles is also of great interest.


First of all, if you want to compare the training algorithms as opposed to the predictive ability of the models you got for the particular dataset you used (i.e. you want to extrapolate the training algorithms' performance to unknown training data of the same kind), you need to be aware that there are at least 2 different sources of uncertainty in the values you measure:

  • Uncertainty on measuring a given surrogate model's performance: the variance here is due to the finite number of test cases you evaluate for each of the surrogate models
  • The actual models you train are subject to variation as well: model instability wrt. to the actual training data.
    For cross validation (resampling validation), you can further divide this into one part that you can measure (e.g. via repeated/iterated cross validation) and a part you cannot measure because your surrogate models share most of their training data. See Bengio, Y. and Grandvalet, Y.: No Unbiased Estimator of the Variance of K-Fold Cross-Validation Journal of Machine Learning Research, 2004, 5, 1089-1105 for this.
    This latter part is of course important if you want to compare the training algorithms in general, but it is of no concern if you want to compare them only for the task at hand, i.e. the data set you actually have.

For regression, you may get away with pooling the two variances you can actually measure into a single pooled variance. However, you should be aware (and state) that for comparison of the algorithms, you can only measure part of the variance uncertainty this way and that this does limit the conclusions you can draw. I'd probably conclude that this is such a big uncertainty that testing (as in calculate some kind of number that quantitates the conclusion) doesn't make sense and rather report on observed differences.

That being said, I'd go for a paired evaluation as you can test all splits with each algorithm.

| cite | improve this answer | |
  • $\begingroup$ To sum up, you suggest to take into account, that we can not handle all uncertainty using cross-validation, and for uncertainty we can handle we can use something like Wilcoxon signed-rank test as also suggested in jmlr.org/papers/v17/benavoli16a.html? $\endgroup$ – Alexey Zaytsev Mar 19 '16 at 18:23
  • 1
    $\begingroup$ Yes, as far as I can see, the Wilcoxon signed rank should work for the part of the uncertainty that you can measure by cross validation. There may be more powerful tests, though. So maybe wait a bit longer whether someone else comes up with something. $\endgroup$ – cbeleites unhappy with SX Mar 20 '16 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.