# Model comparison: better AIC, but worse Mean Actual/Predicted Response

I have made two regression models using gamma regression, and fitted them to 80% of my data, with the intention of doing out-of-sample analysis on the 20%.

Model 2 is just Model 1, with one extra continous variable as regressor. Model 2 shows far better AIC and BIC:

      Model 1   Model 2
AIC   67,710    66,567
BIC   67,875    66,738


The curious thing, in my eyes, is that the Mean Predicted Response is better for Model 2:

                           Model 1   Model 2
Mean Actual Response       233       233
Mean Predicted Response    223       226


Question:

How does on interpret this?

EDIT: I messed up my AIC and BIC.

## 1 Answer

The mean predicted response isn't that meaningful of an evaluation metric. (A model that just predicted the mean would appear perfect with it...) It's usually much better to use, say, mean squared error; you should also make a scatterplot of predicted vs true responses to get a better understanding of what's going on.

• sorry, i messed up my aic and bic due to the data table... i fixed it – Erosennin Mar 15 '16 at 8:31
• @Erosennin Okay; now check that model 1 actually gives better predictions than model 2, by looking at something like root mean squared error or so on, and also by scatterplots of true label be predicted label. The mean prediction being closer doesn't tell you very much. – Dougal Mar 15 '16 at 8:34
• on a sidenote: what if both my models are complex (many regressors) and considering a model with just the mean (no regressors) are out of the question: would not checking the mean predicted response give us an indication of model bias? – Erosennin Mar 15 '16 at 9:22
• @Erosennin Yes, mean predicted response is an indication of the mean of the bias across the training set. Since bias is a signed quantity that varies across the input space, though, taking its mean is not incredibly informative; even if it were, just ignoring variance is not generally a great idea.... – Dougal Mar 15 '16 at 18:29