How to test whether there is a significant difference in mean squared error between two datasets?

Question

I want to test whether Alzheimer's disease causes a change in brain aging compared to healthy patients.

Therefore I have constructed a linear regression model of spectral parameters of brain recordings(the features, or dependent variables) versus age (age is the independent variable).

Now I wish to fit the model on the healthy patients, and then use the coefficients to calculate the expected age of the Alzheimer's patients - comparing the mean squared error of the healthy dataset and the Alzheimer's dataset should help show whether or not there is a difference to the aging due to the disease. (i.e. if the model that works well for healthy patients fails miserably for Alzheimer's patients then there is probably a difference)

I guess I will fit the linear regression model on 80% of the healthy patients (the training set), holding back 20% (as a test set) to calculate the MSE with.

I would use cross-validation but then I will end up with as many different sets of coefficients as I have folds and how would I know which to fit the Alzheimer's patients with? The mean of the coefficients perhaps? An advantage to cross validation though is that I would have a mean and standard deviation of the MSE estimates from the sets of healthy patients so I could use that to determine the significance of deviation between the healthy and diseased MSE's, which is handy.

I guess I could also sample subsets many times from the Alzheimer's patients and create a set of MSE estimates which I could then calculate the standard deviation and mean of to get some idea of the variance there as well so I have an idea about how sensitive it is to that particular dataset. (Should I do this with replacement i.e. bootstrapping? Why or why not?)

Any advice is greatly appreciated.

Answer 1

MSE Significance testing:

You can test for significantly different MSEs if your samples are large enough for the Central Limit Theorem to apply (say, larger than 30).

You want to test whether the means of your two samples are significantly different, where the samples in this case are the squared errors from your healthy and Alzheimer's sets. This becomes a standard case of hypothesis testing. To test if the true difference is zero, assume:

$MSE_1 - MSE_2 \sim N(0, \frac{s_1^2}{N_1} + \frac{s_2^2}{N_2})$

which is the same as the textbook* case:

$\overline X_1 - \overline X_2 \sim N(0, \frac{s_1^2}{N_1} + \frac{s_2^2}{N_2})$

where s₁ is the sample standard deviation of your first group of squared errors, N₁ is the sample size of the first group, and so forth. From there you can find the one-sided p-value with

$Pvalue = P(Z \le \frac{MSE_1 - MSE_2}{\sqrt{\frac{s_1^2}{N_1} + \frac{s_2^2}{N_2}}})$

Another easy way of doing this is to run a 2 sample t-test of the squared errors, which should provide practically identical results.

*Assuming your textbook is Modern Mathematical Statistics with Applications, Second Edition by Devore and Berk, especially pages 490-491

Alternative option– test regression coefficients for significance:

In your regression, you could add a dummy variable for Alzheimer's, and use the standard error of the estimated coefficient to test if it's significantly different from zero. Or, if you think age and Alzheimer's interact, add an age*Alzheimers interaction variable and test that coefficient for significance.

Answer 2

Sounds like an interesting set-up.

Just to re-state your plan, you're creating two populations -- healthy and Alzheimer's. You ten plan to use a 5-fold cross-validation method on only the healthy group. You then plan to fit a model to the training portion of the fold and finally compare the error when applied to the testing portion of the fold versus the Alzheimer's.

If that's correct, I think the simplest approach would be to train your 5 models, apply them against the respective testing sets as well as the Alzheimer's pool. At least initially I would look at the comparison between populations on a per-fold basis. If the results are consistent, I think you're in fine shape. Otherwise, you might do a comparison of the coefficients of the various models you've trained.

I would only follow-up on the penultimate paragraph if your results are very strange.