How to tell whether one regression model is significantly better than another? Bootstrapping? CV?

Question

As part of my PhD research I'm trying to find a good predictor of $y$ using $x1$, $x2$ and $x3$.

(Out of interest, I'm in psychoacoustics. $y$ is the strength of a perceptual attribute elicited by an audio signal; $x1$, $x2$ and $x3$ are physical measurements taken on an audio signal. I have about 300 data points.)

I've built three regression models and have looked at their AIC values.

As shown, $x1 + x2$ is lower than $x1$, which in turn is lower than $x3$. What (little) I know of the statistics literature seems to suggest that I should simply report these values and declare that $x1 + x2$ is the best fitting model. However, this approach doesn't feel quite right to me. Intuitively, I'd like to get some estimate of how much the performance of these models varies depending on the sample I take. If the performance varies a lot, and the distributions of model performance overlap considerably, a more sensible conclusion from the research might be that none of the three models differ meaningfully in their predictive ability.

In the case of $x1$ vs $x1 + x2$ I know that I can test for a better fit using a likelihood ratio test, as the models are nested. My question is: how can I test whether the fit of these two models is significantly better than the fit of $x3$?

While researching how to test for differences between model performance, I came upon this answer, which suggests using the bootstrap to estimate the variability of a correlation coefficient. Following this line of thought, would either of the following approaches be reasonable? Which would you recommend, if either?

1. I use the bootstrap to estimate the sampling distributions of the AIC for each of my three models, and then use hypothesis tests (e.g. t-tests) to check for significant differences between them. Alternatively, I build confidence intervals around each AIC and check whether they overlap.

2. I use (10 fold) cross validation to estimate the test error (MSE) for each model. Again, I check for significant differences in estimated test MSE using hypothesis tests or confidence intervals.

Thoughts?

Answer 1

In addition to @DemetriPananos' very good advise to look into ESL:

• I find fig. 1 in Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895–1923. extremely helpful in clarifying what one is talking about (including $\mathrm Err_\tau$ vs. $\mathrm Err$).

• If you are concerned with the performance of your modeling approach for new (not yet existing) data sets of the same type as the one you have ("analyse algorithms" branch in Dietterich) rather than with producing a good model from the data at hand (that's usually my task), you'll also want to have a look at Bengio, Y. and Grandvalet, Y.: No Unbiased Estimator of the Variance of K-Fold Cross-Validation Journal of Machine Learning Research, 2004, 5, 1089-1105 as that paper explains that you won't be able to measure all the relevant variance with the cross validation procedure.

• The situation is different (better) for the "analyze classifiers" branch in Dietterich's figure (because the variance uncertainty of what happens if a new dataset of size $n$ is drawn from the population doesn't matter for the classifier that can be obtained from the data at hand.). But you still have at least two sources of variance: uncertainty due to the limited number of tested cases, and uncertainty due to model instability. In addition, there may be further confounders (that e.g. cause clustering in your data and thus correlation between cases).
  These can be assessed with bootstrapping (and also iterated/repeated CV/jackknifing)

  One important point that can be extremely helpful in this context is that you may be able to say why the so far apparently better model isn't significantly better: because you've had too few test case

• One advantage of bootstrapping or iterated/repeated CV here is that you can set up the resampling procedure to take such confounders into account (e.g. if the measurements come from fewer subjects, you'd resample subjects).

• You can then construct a pairwise test by running the same splits for all formulae.

Answer 2

Intuitively, I'd like to get some estimate of how much the performance of these models varies depending on the sample I take.

Take a look at cross validation schemes. There are a couple interpretations of error in Elements of statistical learning which I think would be beneficial for you to know.

The first is Generalization or Test Error. This is the expected error your model would make conditioned on training on the data you have presently. Again, to be clear, this is the error your model would make when fit on this training set. Let's call this $\operatorname{Err}_{\mathcal{T}}$.

Contrast this to the Expected Test Error, which is the error you would make averaging over new test and training data. Call this, $\operatorname{Err}$.

To me, it sounds like you are interested in $\operatorname{Err}$ rather than $\operatorname{Err}_{\mathcal{T}}$. The authors of ESL write in chapter 7 "Estimation of $\operatorname{Err}_{\mathcal{T}}$ [through cross validation] will be our goal, although we will see that $\operatorname{Err}$ is more amenable to statistical analysis", and so I think chapter 7 is really what you need and I would recommend you read that chapter.