What is an appropriate way to apply a statistical hypothesis test for evaluating model performance metrics (e.g. MAE, MAPE, MSE) between two regression models based on a single holdout (validation) dataset?
Possible approaches
What I'd like to do is just apply a paired t-test on the observations directly (at least for measures like absolute error (AE) or absolute percent error (APE) where there are meaningful corresponding observation level measures). The problem with this though seems like we would be at risk of violating some of the assumptions of a paired t-test, e.g.
- dependent variable might not be normally distributed (as they would be when comparing the average of the metrics of cross-validaon splits, see [Context], which we can trust in many metrics due to central limit theorem)
- will not be safe from outliers
I guess though how much do I need to worry about these (isn't the t-test pretty robust)? Or what adjustments can we make to the test to still make it valid? Would using a Wilcoxan signed-rank test be a better way to go... (tweet about this)?
Preference is for approaches that are simple, reasonably appropriate / generalizable and use performance on holdout data that do not depend on building the model multiple times...
Context
A common model comparison technique based on predictions on hold-out data is to build multiple models each using k-fold cross validation, calculate the average performance for each on the k holdout sets for each model, and then compare performance between the model's using a paired t-test (e.g. from the {tidymodels} team at Rstudio see Feature Engineering..., 3.7 or Tidy Models with R, 11.3).
I use this procedure frequently and love how simple it is, however a few concerns I have with it are:
- cannot be used when doing a simple training-validation split for evaluation (as only have one hold-out set)
- the power of the test seems like it could be dependent on the number of splits, k
- Post on Statistical Significance Tests for Comparing Machine Learning Algorithms critiques the methodology for a lack of independence (due to the training sets being largely similar between k models) though still notes the approach is fine and may just have a little high type 1 errors -- this point doesn't bother me too much as I primarily am using the test as an indicator.
This question is in regards to point 1. where we are doing evaluation using a single validation set. I found a lot of cross-validated posts on statistical tests in model comparisons, but most are in the context of cross-validation or other resampling procedures.
Related / follow-up questions
- Similarly, I recently tweeted about adding confidence intervals on model performance estimates in a validation set using simple methods (based on calculating the standard error), is this approach fine or problematic?
- In the case when doing k-fold cross-validation would it be OK to do a paired t-test on the errors of the individual hold-out predictions from the training set (i.e. run the test on the sample of n observations, rather than the sample of k average errors of each split). However the point in 3 and the concerns from the bullet points give me even greater pause here... (Also, similar to above note, some metrics cannot be calculated at the observation level, e.g. in the classification context the AUC under ROC curve).