1
$\begingroup$

I have made two regression models. They were made on a training set of 80% of the data. And 20% of the data is the validation set. No test set is made.

The models tell me how much premium a supermarket has to pay in insurance, in case something goes wrong and the food goes bad. So the premium is low, but the claims are high, because of low claim frequency.

Now I wish to test how the two models perform on the $N$ observations in the validation set. Model 1 has many variables, and Model 2 has only a few. I had the idea of calculating the Mean Squared Error per combination of the covariates. So all customers with one specific combination of the covariates is one group $k$ (of a total of $K$ combinations). So if you have two categorical variables with 2 categories each, you would have $2*2=4$ groups:

$MSE_1= \frac{\sum_{k=1}^K [\sum_{j=1}(\hat{g}(x_{kj})-g(x_{kj}))]^2}{N}$

The regular Mean Squared Error is:

$MSE_2 = \frac{\sum_{i=1}^N (\hat{g}(x_{i})-g(x_{i}))^2}{N}$

Question 1: The first method favours Model 2 (the parsimonious model) over Model 1 (complex model) with $MSE_1$ of 5 < 10000. Am I correct in assuming that this method does not work when the difference in groups is very large? Model 2 has 1000 times more groups because it has so many covariates (hence many combinations).

Question 2: $MSE_2$ favours Model 1 (complex model) with MSE of 45053 < 45061. This difference seems small to me. Is it possible to test for this difference?

Question 3: I'm afraid of the sensitivity to outliers in $MSE_2$, so that is why I made $MSE_1$. Would MAE (Mean Absolute Error) in general be better than MSE?

$\endgroup$
  • 1
    $\begingroup$ Question 2: Diebold-Mariano test? $\endgroup$ – Richard Hardy May 6 '16 at 8:42
  • $\begingroup$ @RichardHardy: do you have comment on what the answer given below says about DM test not being meant for comparing models? $\endgroup$ – Erosennin May 6 '16 at 10:26
  • 2
    $\begingroup$ In question 2, you care whether $\text{MSE}_1$ is statistically greater than $\text{MSE}_2$, a question that can be answered by the DM test. What you do further is another question. If you want to compare models rather than their forecast accuracy, DM may be inappropriate. $\endgroup$ – Richard Hardy May 6 '16 at 10:56
  • $\begingroup$ @RichardHardy: But is this not effectively comparing models in any case (comparing MSEs)? (sorry if I'm not seeing the obvious here) $\endgroup$ – Erosennin May 6 '16 at 10:58
  • 1
    $\begingroup$ I think, it is not; but this is a tricky question. DM stress the difference between comparing forecasts (which need not even be generated by a model - e.g. they may be due to a survey of professional forecasters) and comparing models. Take a closer look at their paper, the answer is there. $\endgroup$ – Richard Hardy May 6 '16 at 11:32
4
$\begingroup$

A quick warning on not having test data (going straight from training to validation): if you use the validation data-set to do model selection you are still at risk of overfitting (as in the real prediction error will be higher than what the validation data will suggest). This is because, in essence, you are using the validation dataset for fitting, if only obliquely so.
This is what drives the nested cross-validation approach, in a nutshell. Split your training data further into training and testing if at all possible.

1-3) There are common prediction errors formulas you could grab. Besides the "mean absolute error" the cvTool package suggests using trimmed error (trimming out the largest errors). However keep in mind that this will fix the reliance of the model selection step to the outliers but not the "fitting step".
In other words, if you are afraid that model selection is driven by outliers then try out trimmed errors but if you are afraid outliers are affecting the overall shape of your regressions then you should switch to robust regressions.

Also it is possible that your domain may provide you a clue on how expensive it is to make bad predictions. If you can put a dollar amount to the error (one that is perhaps not even symmetric) that makes sense to your domain then that will clearly be better than coming up with strange error definitions.

2) Caution in using the DM test here; as the reference provided in the comments warn: "The DM test was not intended for comparing models" (emphasis not mine).
K-fold Cross-validation would give you a vector of $k$ prediction errors and you could potentially build a small confidence interval around them to compare the prediction difference between your models.

$\endgroup$
  • 4
    $\begingroup$ Regarding your warning. I was under the belief (stats.stackexchange.com/questions/19048/…) that we divided the data set into three parts: training, validation and test. The point of the training set is to fit your model, the validation set gives you which of your models is the best, and the test set gives you how well your model actually predicts (ref. your comment on actually fitting on your validation set). My concern is only choosing the best model, not actually how well it predicts. So a training + validation must suffice, no? $\endgroup$ – Erosennin May 6 '16 at 10:21
  • 1
    $\begingroup$ You are absolutely right, my bad! Hastie, Tibshirani and Friedman at page 222 of "The Elements of Statistical Learning" use your terminology. $\endgroup$ – CarrKnight May 6 '16 at 11:07
0
$\begingroup$

This is merely a suggestion how to test the difference between the MSE of two models: why not bootstrap? Resample the 20% validation set, and calculate MSE for each re-sample, and obtain a distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.