As part of my PhD research I'm trying to find a good predictor of $y$ using $x1$, $x2$ and $x3$.
(Out of interest, I'm in psychoacoustics. $y$ is the strength of a perceptual attribute elicited by an audio signal; $x1$, $x2$ and $x3$ are physical measurements taken on an audio signal. I have about 300 data points.)
I've built three regression models and have looked at their AIC values.
As shown, $x1 + x2$ is lower than $x1$, which in turn is lower than $x3$. What (little) I know of the statistics literature seems to suggest that I should simply report these values and declare that $x1 + x2$ is the best fitting model. However, this approach doesn't feel quite right to me. Intuitively, I'd like to get some estimate of how much the performance of these models varies depending on the sample I take. If the performance varies a lot, and the distributions of model performance overlap considerably, a more sensible conclusion from the research might be that none of the three models differ meaningfully in their predictive ability.
In the case of $x1$ vs $x1 + x2$ I know that I can test for a better fit using a likelihood ratio test, as the models are nested. My question is: how can I test whether the fit of these two models is significantly better than the fit of $x3$?
While researching how to test for differences between model performance, I came upon this answer, which suggests using the bootstrap to estimate the variability of a correlation coefficient. Following this line of thought, would either of the following approaches be reasonable? Which would you recommend, if either?
I use the bootstrap to estimate the sampling distributions of the AIC for each of my three models, and then use hypothesis tests (e.g. t-tests) to check for significant differences between them. Alternatively, I build confidence intervals around each AIC and check whether they overlap.
I use (10 fold) cross validation to estimate the test error (MSE) for each model. Again, I check for significant differences in estimated test MSE using hypothesis tests or confidence intervals.
Thoughts?