# Tradeoff between Prediction Interval Accuracy & Mean Squared Error

My goal is to quantify the prediction uncertainty in a model regressing climate covariates against GDP. I start with a model with temperature as a third degree polynomial, country fixed effects ($$\alpha_i$$}, year fixed effects ($$\theta_t$$), and incremental time trend by country ($$\gamma_i$$).

$$GDP_{it} = \beta_1 * Temp_{it} + \beta_2 * Temp^2_{it} + \beta_3 * Temp^3_{it} + \alpha_i + \theta_t + \gamma_i$$

I use some withheld data to gather out-of-sample Mean Squared Error for the above model. I also use the standard error of the out-of-sample predictions to construct 95% prediction intervals and then check the actual percentage of real Y (GDP) values that fall within those intervals as prediction interval accuracy.

Out-of-sample MSE: 0.017
Out-of-sample Prediction interval accuracy: 0.577


In an effort to get the prediction interval accuracy closer to the 95% target, I try a different model with higher-degree polynomial time trends, shown below:

$$GDP_{it} = \beta_1 * Temp_{it} + \beta_2 * Temp^2_{it} + \beta_3 * Temp^3_{it} + \alpha_i + \theta_t + \gamma_i + \gamma^2_i + \gamma^3_i$$

Out-of-sample MSE: 0.018
Out-of-sample Prediction interval accuracy: 0.722


The prediction interval accuracy is much improved as a result of the prediction intervals being wider, presumably because the model is incorporating more of the variance in the training data. However, probably due to overfitting from the additional model complexity, the MSE is higher in the second model than the first.

My question has less to do with this specific example than this phenomenon I have observed in general. I am wondering:

1. Is there a single underlying phenomenon that explains why increasing model complexity causes both the out-of-sample prediction intervals to move closer to the 95% target while at the same time increases the out-of-sample MSE of the model, or are these essentially independent observations?

2. If my goal is to quantify model uncertainty the best that I can, what is the right way to think about trading off higher quality (in this case meaning wider) out-of-sample prediction intervals for a subsequent increase in out-of-sample MSE?

1. The only mechanism I could think of that would increase both the MSE and the PI converage would be straightforward overfitting. You overfit in the training sample, then get larger MAEs in the holdout set, and that of course leads to wider PIs, which have a larger coverage. In the meantime, you also get a larger MSE, again because of the overfitting.

Third degree polynomials can and will explode for very large and very small predictor values, and the effect is exacerbated by the fact that you very likely have few observations at these predictor values.

I would recommend you use natural or other splines. These are flexible across most of your predictor range, but linear at the extremes.

In addition, it looks like your MAE approach to PIs is not working very well. You are not saying how exactly you turn your MAE into PIs, but it presumably uses some distributional assumption. It may be that this assumption is not working out well. Alternatives would be a direct quantile regression for two quantiles using a pinball loss, or possibly conformal prediction. (Also, I do wonder how you get year fixed effects for forecasting. Are you forecasting these themselves?)

2. I would argue (in this paper and also here) that it is important to first understand the context. What are you forecasting for? Which subsequent decisions will depend on the forecast? What you mean by a "best way to quantify uncertainty" can only be understood in this context.

It seems like you want both an expectation forecast and a prediction interval. One way to proceed would be to aim for full predictive densities, from which you can extract point forecasts like the expectation and quantiles. Predictive densities can be assessed using proper scoring rules, the tag wiki has more information.

This thread gives some general resources about forecasting: Resources/books for project on forecasting models.

• Could you explain what you mean by my "MAE approach to PIs"? Maybe I didn't give enough details on this but I am using the Delta method to get the standard error of the prediction. Maybe you already understood that but I am unfamiliar with how MAE could be used to create prediction intervals Commented Jul 22 at 21:07
• I'm also curious about the relationship about overfitting and PIs in general. I think that is really the crux of this question, when it comes down to it. Are PIs simply not accurate when there is an overfit model? Commented Jul 22 at 21:14
• Ah, on re-reading, you are not using the MAE, but the standard error of your predictions. I am not sure what you mean by that. A standard error is typically attached to an estimate of a parameter, not to realizations or observables. I am also unfamiliar with the Delta method. In any case, you will need some distributional assumptions, which you are not explaining. For instance, if your observations are negbin distributed, then the conditional mean plus its standard error will not be enough to give you any PI. Perhaps you can give more details on how you do your PIs? Commented Jul 22 at 21:16
• Re your second comment, I would not say that PIs are simply not accurate when there is an overfit model. I would say that PIs, being just two quantile forecasts, can suffer from overfitting just like parameter estimates or expectation forecasts: large differences between in-sample fits and out-of-sample accuracy, and high variability. In this particular case, it seems to me like the way you calculate PIs based on a holdout sample propagates the worse performance of holdout forecasts through to PIs. Commented Jul 22 at 21:18
• Thanks for the quick response! Yes, there are distributional assumptions in the calculation of standard error of the prediction. I probably can't do as good a job as describing it as other easily-found sources could, but the Delta method uses Taylor series to describe the asymptotic distribution of the estimator function. There is a Gaussian distribution assumption of the RV baked in. To create the PIs, for each prediction I generate the standard error of the prediction which combines with the prediction mean to provide a likely range of prediction values. Commented Jul 22 at 21:33