Model prediction: test for difference in MSE

Question

I have made two regression models. They were made on a training set of 80% of the data. And 20% of the data is the validation set. No test set is made.

The models tell me how much premium a supermarket has to pay in insurance, in case something goes wrong and the food goes bad. So the premium is low, but the claims are high, because of low claim frequency.

Now I wish to test how the two models perform on the $N$ observations in the validation set. Model 1 has many variables, and Model 2 has only a few. I had the idea of calculating the Mean Squared Error per combination of the covariates. So all customers with one specific combination of the covariates is one group $k$ (of a total of $K$ combinations). So if you have two categorical variables with 2 categories each, you would have $2*2=4$ groups:

$MSE_1= \frac{\sum_{k=1}^K [\sum_{j=1}(\hat{g}(x_{kj})-g(x_{kj}))]^2}{N}$

The regular Mean Squared Error is:

$MSE_2 = \frac{\sum_{i=1}^N (\hat{g}(x_{i})-g(x_{i}))^2}{N}$

Question 1: The first method favours Model 2 (the parsimonious model) over Model 1 (complex model) with $MSE_1$ of 5 < 10000. Am I correct in assuming that this method does not work when the difference in groups is very large? Model 2 has 1000 times more groups because it has so many covariates (hence many combinations).

Question 2: $MSE_2$ favours Model 1 (complex model) with MSE of 45053 < 45061. This difference seems small to me. Is it possible to test for this difference?

Question 3: I'm afraid of the sensitivity to outliers in $MSE_2$, so that is why I made $MSE_1$. Would MAE (Mean Absolute Error) in general be better than MSE?

Answer 1

A quick warning on not having test data (going straight from training to validation): if you use the validation data-set to do model selection you are still at risk of overfitting (as in the real prediction error will be higher than what the validation data will suggest). This is because, in essence, you are using the validation dataset for fitting, if only obliquely so.
This is what drives the nested cross-validation approach, in a nutshell. Split your training data further into training and testing if at all possible.

1-3) There are common prediction errors formulas you could grab. Besides the "mean absolute error" the cvTool package suggests using trimmed error (trimming out the largest errors). However keep in mind that this will fix the reliance of the model selection step to the outliers but not the "fitting step".
In other words, if you are afraid that model selection is driven by outliers then try out trimmed errors but if you are afraid outliers are affecting the overall shape of your regressions then you should switch to robust regressions.

Also it is possible that your domain may provide you a clue on how expensive it is to make bad predictions. If you can put a dollar amount to the error (one that is perhaps not even symmetric) that makes sense to your domain then that will clearly be better than coming up with strange error definitions.

2) Caution in using the DM test here; as the reference provided in the comments warn: "The DM test was not intended for comparing models" (emphasis not mine).
K-fold Cross-validation would give you a vector of $k$ prediction errors and you could potentially build a small confidence interval around them to compare the prediction difference between your models.

Answer 2

This is merely a suggestion how to test the difference between the MSE of two models: why not bootstrap? Resample the 20% validation set, and calculate MSE for each re-sample, and obtain a distribution.