How do I know if the difference in performance between two machine learning algorithms is significant?

Question

I built two machine learning models, A and B, and performed 10 times repeated 10-fold cross validation (CV).

The mean RMSE and $R^2$ (over all folds) for the models is:

Model A: mean RMSE=0.780, mean $R^2$=0.340

Model B: mean RMSE=0.748. mean $R^2$=0.390

So, is the difference in performance between the algorithms significant?

To check this I used the Wilcoxon rank sum test to compare the RMSE of the 100 folds. The p-value is 0.104, so probably not significant.

However, intiuitively, a difference in $R^2$ of 0.05 over 100 folds "feels" to me noteworthy.

How can I deal with this situation? Do you have any suggestions? Instead of using 10 repeated CV, I could repeat CV 100 times, but this feels to me too much like p-hacking. What do you think?

Answer 1

is the difference in performance between the algorithms significant?

$H_0$: Model A and B have the same generalization error

$H_1$: Model B has a lower generalization error than model A

In cross-validation, the out of sample error approximates the generalization error. When you're doing $n$ repeated $k$-fold cross validation, you're producing $n$ different models (assuming different random seed and training sets due to different $k$-fold split). Therefore you don't compare the models, you're comparing the algorithms and hyperparameters.

Rather, let models A and B be produced once by a $k'$-fold cross validation with $k'=nk$. You obtain $k'$ out of sample errors for model A and B. You can then apply Mann-Whitney U test to know how significant the difference in generalization error is.

Assuming you don't set an alpha level to reject the null hypothesis $H_0$ and you get $p=0.104$, the roughs guidelines are:

• If $p > .10$ → “not significant”
• If $p ≤ .10$ → “marginally significant”
• If $p ≤ .05$ → “significant”
• If $p ≤ .01$ → “highly significant.”

Now what does it means in practice? If you were to choose between the two models, rather have the one with the lowest estimated generalization error. But should you spend time changing an algorithm in production for such difference? Not if you don't expect a lot of business revenue generated by this difference.

Answer 2

My take on statistical testing o regression problems is different than statistical tests on classification problems. As you may know, statistical tests need sets of measures and not just single measures. So if you have a single test set, and you calculate the RMSE or accuracy, you have a single number, and therefore no way of performing a statistical test. So you "create" many test sets by using some form of cross-validation. (k-folds, repeated k-folds and so on).

But for regression (I will discuss classification later) you do have a set of numbers - the error (or squared error) for each data in the test set. Just perform a paired t-test or paired Wilcoxon test!

You may be tense on making this decision using only one test set, which may be "unlucky" or "non-representative". But notice that the set itself is not that important, since you are using each measure of error on the data on the test set for the test. But why not use more data for the statistical test? I would suggest at most a k-fold (whatever k) - in this case, all data will contribute only one measure of error to the statistical test.

Clearly, doing a repeated k-fold (10 times a 10-fold) is p-haking since you are adding repeated data which will only decrease the p-value of the test.

TLDR: do not use RMSE on repeated k-folds - use the SE (squared error) of each data on a single k-fold.

Why not do this for classification? The problem is that the result of the classification for each data is a binary variable - the result is correct or the result is not correct. The measure for each data is a binary variable and the correct test for paired binary variables is the McNemar test which I believe (I do not have a reference for this) is a weak test. So tradition, or research end up suggesting an aggregate measure (such as accuracy, or F1, or so on) on a set of test data, and using multiple tests data to be able to have a set of numbers to perform statistical test.

Finally, I really dislike things like a 10 times repeated 10-fold against for instance F. Harrell who, for example, suggests 100-times 10-fold https://www.fharrell.com/post/split-val/. My criticism of this many repetitions of k-fold is the p-haking aspect of it - you are generating more and more data (up to $2^N$ data points where N is the size of your dataset) and that will clearly result in a low p-value for any significance test you perform (I forget the name of this theorem - infinite samples results only in p-value = 0 or p-value = 1 and this last case if the two infinite samples come from the same distribution).

Answer 3

Using cross validated estimates of out of sample performance is not advisable, since results fold to fold are correlated due to the overlapping data in each fold. Not only that, but any tests are woefully under powered with 10 samples unless the difference in performance is extremely large, in which case there is no need for a statistical test.

To test if performance is different model to model, you may want to proceed as follows:

1. Hold out additional data that the model has not seen.

2. For each observation in this data, make a prediction with each model and compute a loss value (for RMSE, we would compute $\ell_i = \vert y_i - \hat{y}_i \vert ^2$).

3. You now have samples of loss values for each model, and you can test for a difference in expected loss using a t test (note, the distribution of these loss values is irrelevant when the validation set is large enough).