How to compare/choose regression models based on predictions and prediction intervals I have two regression models M1 and M2, each delivers a prediction (blue circle) together with a prediction interval as shown in blue in the picture below. The actual value is marked with a black cross.

How can I objectively compare these two models? On one hand the real value (black cross) is within the prediction interval of M1, but on the other hand the prediction (average value, blue circle) of M2 is closer to the real value. Is there a way to choose one model over the other??
Thanks in advance!
 A: I am assuming you have more than one prediction point on which to compare models M1 and M2 -- I wouldn't make a model selection based on a single prediction point. It really depends on your goal. 
I would suggest summarizing your models' predictions in the following ways on the validation dataset:


*

*Empirical coverage of the 95% prediction intervals -- that is, the proportion of your 95% prediction intervals that contain the true value. This is similar to your example, but you would repeat the process for all of your prediction dataset. Ideally the proportion would be close to 0.95 for both models.

*You can also summarize the average width of the prediction intervals. If the empirical coverage rate is similar in the two models on the validation dataset, you could use the prediction interval width and pick the model with smaller intervals (M2 in your image). 

*Bias: Calculate the average of the true values minus the predictions for both models. Ideally this will be close to 0 for both. 

*Mean-squared error (MSE): Both methods could be unbiased, but one could produce smaller mean-squared error, meaning that the squared distance is smaller (M2 in your image). 


My sense is if your goal is solely prediction accuracy (getting close to the true value), then MSE could be a good metric on which to choose the model (so M2 in your case), but bias should not be overlooked either. Supposing we had a validation dataset which pointed to two unbiased models but lower MSE and poorer prediction interval coverage for M2 vs. M1, it really depends on whether uncertainty quantification is important to you. For many prediction tasks, people are solely interested in point predictions. From a statistician's point of view, the model should be quantifying uncertainty correctly. In this scenario, probably you would want to re-evaluate and try to construct a model that has the predictive accuracy of M2 but the uncertainty intervals of M1.
