Measuring forecast accuracy of the conditional mean

Consider a dependent variable $y$, independent variables $x_1,\dotsc,x_K$, a model

$$y = X \beta + \varepsilon,$$

and an estimated coefficient $\hat\beta$. If the model is correctly specified, the true conditional mean of $y$ given $X$ is

$$\mathbb{E}(y|X) = X \beta.$$

Since we do not know $\beta$, we use its sample estimate $\hat\beta$ to get the estimated conditional mean

$$\hat{\mathbb{E}}(y|X) = X \hat\beta.$$

For a new set of observations $x_{1,i},\dotsc,x_{K,i}$, we can predict the conditional mean of the corresponding $y_i$ using the new $x$s and the estimated coefficient $\hat\beta$ as follows:

$$\hat{\mathbb{E}}(y_i|x_{1,i},\dotsc,x_{K,i}) = (x_{1,i},\dotsc,x_{K,i}) \hat\beta.$$

Now let us evaluate the forecast accuracy. We take the realized value $y_i$ and compare it to the predicted value $\hat{\mathbb{E}}(y_i|x_{1,i},\dotsc,x_{K,i})$. If the two are close, we say that the forecast is accurate.

Here is what bugs me:

• Aren't we predicting the true conditional mean $\mathbb{E}(y_i|x_{1,i},\dotsc,x_{K,i})$ rather than the actual realization $y_i$?
• If so, we are committing a measurement error when using $y_i$ in place of the unobserved $\mathbb{E}(y_i|x_{1,i},\dotsc,x_{K,i})$ when evaluating forecast accuracy. Isn't this problematic?

(One may also think about modelling higher order moments, such as conditional variance. There it is more obvious that the population moment being forecasted is unobservable, and hence measuring forecast accuracy is nontrivial.)

Aren't we predicting the true conditional mean $\mathbb{E}(y_i|x_{1,i},\dots,x_{K,i})$ rather than the actual realization $y_i$?

Yes indeed we are.

If so, we are committing a measurement error when using $y_i$ in place of the unobserved $\mathbb{E}(y_i|x_{1,i},\dots,x_{K,i})$ when evaluating forecast accuracy. Isn't this problematic?

On the one hand, you are right. We are forecasting an unobservable quantity and want to assess the accuracy of this forecast. We have a problem here.

The apparently only way out is to assess the accuracy of forecasts based on observables, and then deal with the fact that our forecast accuracy inevitably again only is one realization of a random variable, by invoking asymptotic arguments.

Now, whether and how this works for a given point forecast depends on what you want to use the point forecast for. Or, from a different perspective, it depends on your loss function.

Loss functions for forecast errors have been a topic for quite a while now (Fildes & Makridakis, 1988, IJF). In the area I am most familiar with, forecasting retail sales, the conditional mean (quadratic loss) is most useful for planning promotions, whereas store replenishment requires high quantiles.

Now, all the above relies on the fact that there always remains unexplained variance. (In some areas, like physics, we have a sufficiently good handle on the data generating process that we can explain almost all the variance and forecast extremely well, say, the trajectory of a bullet in vacuum.) People have been arguing that in non-physics situation, point forecasts alone are not overly helpful, and we should really aim for density forecasts, also known as predictive distributions. This ties into your parenthetical remark at the end of the question.

This is commonly accepted in financial and macroeconomic forecasting (in finance, driven by Value at Risk and options pricing) - not so much in supply chain and sales forecasting, where people happily calculate conditional means, estimate variances and assume a homoskedastic normal distribution in setting safety stocks. I have argued that predictive distributions make more sense in supply chains, too. The problem is that evaluating a density forecast is a bit more involved than evaluating point forecasts. I give a few pointers in this earlier answer of mine.

• Stephan, I am lucky to get you on my case. Had you ever thought of this problem before? Don't you think the formulations involving forecast accuracy in the setting above are often (always?) careless? I mean, if we acknowledge that measuring the forecast accuracy of conditional variance is nontrivial, shouldn't we acknowledge the same for the case of conditional mean, too? Should we take some care when reporting measures of forecast accuracy (do some measurement-error corrections)? – Richard Hardy Apr 7 '16 at 11:37
• I am writing an academic paper where I distinguish between the nontrivial measurement of forecast accuracy for cond. variance and "trivial" for cond. mean. Now I realized that even the cond. mean case is nontrivial, so I posted the question. – Richard Hardy Apr 7 '16 at 11:40
• I fully agree that assessing the accuracy of a forecast for the conditional mean is roughly as hard as for the conditional variance. (Somewhat less so, because the mean is easier to estimate than the variance.) As per above, that means that all forecast accuracy measures are really random variables, and we should treat them as such, leading to things like Diebold-Mariano tests... – Stephan Kolassa Apr 7 '16 at 12:25
• ... I'd be interested in any measurement-error corrections you have in mind, but I'd suppose that most things you would want to account for in correcting for measurement error in realizations you'd also already incorporate in your forecasts. – Stephan Kolassa Apr 7 '16 at 12:26
• That said, this discussion really takes us close to the concept of forecastability, which is a frequent topic in Foresight. If your supermarket sales are well described by a Poisson distribution, there is a lower limit for your forecast's MSE - although we could probably improve on it if we knew all shopping lists in a 100 mile radius. So everything depends on your information set. – Stephan Kolassa Apr 7 '16 at 12:30