Measuring forecast accuracy of the conditional mean

Question

Consider a dependent variable $y$, independent variables $x_1,\dotsc,x_K$, a model

$$ y = X \beta + \varepsilon, $$

and an estimated coefficient $\hat\beta$. If the model is correctly specified, the true conditional mean of $y$ given $X$ is

$$ \mathbb{E}(y|X) = X \beta. $$

Since we do not know $\beta$, we use its sample estimate $\hat\beta$ to get the estimated conditional mean

$$ \hat{\mathbb{E}}(y|X) = X \hat\beta. $$

For a new set of observations $x_{1,i},\dotsc,x_{K,i}$, we can predict the conditional mean of the corresponding $y_i$ using the new $x$s and the estimated coefficient $\hat\beta$ as follows:

$$ \hat{\mathbb{E}}(y_i|x_{1,i},\dotsc,x_{K,i}) = (x_{1,i},\dotsc,x_{K,i}) \hat\beta. $$

Now let us evaluate the forecast accuracy. We take the realized value $y_i$ and compare it to the predicted value $\hat{\mathbb{E}}(y_i|x_{1,i},\dotsc,x_{K,i})$. If the two are close, we say that the forecast is accurate.

Here is what bugs me:

• Aren't we predicting the true conditional mean $\mathbb{E}(y_i|x_{1,i},\dotsc,x_{K,i})$ rather than the actual realization $y_i$?
• If so, we are committing a measurement error when using $y_i$ in place of the unobserved $\mathbb{E}(y_i|x_{1,i},\dotsc,x_{K,i})$ when evaluating forecast accuracy. Isn't this problematic?

---

(One may also think about modelling higher order moments, such as conditional variance. There it is more obvious that the population moment being forecasted is unobservable, and hence measuring forecast accuracy is nontrivial.)

Answer 1

Aren't we predicting the true conditional mean $\mathbb{E}(y_i|x_{1,i},\dots,x_{K,i})$ rather than the actual realization $y_i$?

Yes indeed we are.

If so, we are committing a measurement error when using $y_i$ in place of the unobserved $\mathbb{E}(y_i|x_{1,i},\dots,x_{K,i})$ when evaluating forecast accuracy. Isn't this problematic?

On the one hand, you are right. We are forecasting an unobservable quantity and want to assess the accuracy of this forecast. We have a problem here.

The apparently only way out is to assess the accuracy of forecasts based on observables, and then deal with the fact that our forecast accuracy inevitably again only is one realization of a random variable, by invoking asymptotic arguments.

Now, whether and how this works for a given point forecast depends on what you want to use the point forecast for. Or, from a different perspective, it depends on your loss function.

• If your loss function is quadratic, then forecasting the conditional mean is the best you can do.
• If your loss function depends on the absolute error without squares involved, you want to forecast the conditional median.
• If your loss function involves "skewed" absolute errors, you want to forecast conditional quantiles.

Loss functions for forecast errors have been a topic for quite a while now (Fildes & Makridakis, 1988, IJF). In the area I am most familiar with, forecasting retail sales, the conditional mean (quadratic loss) is most useful for planning promotions, whereas store replenishment requires high quantiles.

---

Now, all the above relies on the fact that there always remains unexplained variance. (In some areas, like physics, we have a sufficiently good handle on the data generating process that we can explain almost all the variance and forecast extremely well, say, the trajectory of a bullet in vacuum.) People have been arguing that in non-physics situation, point forecasts alone are not overly helpful, and we should really aim for density forecasts, also known as predictive distributions. This ties into your parenthetical remark at the end of the question.

This is commonly accepted in financial and macroeconomic forecasting (in finance, driven by Value at Risk and options pricing) - not so much in supply chain and sales forecasting, where people happily calculate conditional means, estimate variances and assume a homoskedastic normal distribution in setting safety stocks. I have argued that predictive distributions make more sense in supply chains, too. The problem is that evaluating a density forecast is a bit more involved than evaluating point forecasts. I give a few pointers in this earlier answer of mine.