What is the relationship between minimizing prediction error versus parameter estimation error?

Question

With the advent of statistical learning techniques, people are talking a lot about prediction error, while in classical statistics, one is focusing on parameter estimation error. What is the relationship between the two concepts? Does one imply the other?

Assuming a true linear model $y = X\beta_0 + \varepsilon$, estimate $\hat\beta$ and prediction $\hat y=X\hat\beta$. One can define, with $\lVert.\rVert$ the mean square error norm for example:

• Prediction error: $\lVert y-\hat y\rVert = \lVert X(\beta -\hat\beta)\rVert$ (note this definition omits the part related to the error term )
• Estimation error: $\lVert \beta -\hat\beta\rVert$

Does one concept imply the other one? In other terms, if a model minimizes estimation error, does it necessarily minimize prediction error under the assumption of a linear model?

Intuition in the linear case seems to indicate that this only matters when $X$ has correlated component (otherwise if X is such that $X'X=I$, definitions are equivalent under L2 norm), does it hold in more general cases?

Thanks!

Answer 1

With the advent of statistical learning techniques, people are talking a lot about prediction error, while in classical statistics, one is focusing on parameter estimation error.

Exactly. This difference can be properly understood only if we realize and keep in mind that the scope of model like regression, first of all linear regression as your true model suggest, can be different (read here: Regression: Causation vs Prediction vs Description). If your goal is prediction, as usual in supervised/predictive machine learning, you have to minimize the prediction error; parameters value per se not matters therefore endogeneity is not the core issue. At the other side if your goal is description or causal inference you have to focus on parameters estimation. For example in econometrics the usual focus is (or was) in causal inference (conflated with description if we follow the argument suggested in previous link), then endogeneity is treated as the main issue. In this literature prediction is treated as secondary problem, or ad hoc one in time series context (ARMA models for example). In most case is given the impression that if endogeneity go away, as consequence, the best prediction/forecasting model is achieved too. If this thing was true, the two minimization problem that you write above would be equivalent.

However this is not true, infact in prediction/forecasting endogeneity are not the main problem while overfitting is (read here: Endogeneity in forecasting)

In order to understand this distinction the bias-variance tradeoff is the crucial point. Infact at the start of most machine learning books this topic is exhaustively treated and overfitting problem come as consequence. Indeed in most generalistic econometric books the bias-variance tradeoff is completely forget, for overfitting problem the same is true or, at best, it is vaguely treated. I started to study topic like those we treat here from econometrics side and when I realized this fact I remained badly surprised.

The article that underscore at best this problem is probably: To Explain or to Predict – Shmueli (2010). Read here (Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict"))

In other terms, if a model minimizes estimation error, does it necessarily minimize prediction error under the assumption of a linear model?

No, definitely not. For prediction scope, more precisely in term of Expected Prediction Error, the "wrong model" (incorrectly specified regression) can be better then the "right one" (correctly specified regression). Obviously this fact is irrelevant if, as in causal inference, parameters is the core of analysis. In the article is given an example that involve an underspecified model. I used this argument here (Are inconsistent estimators ever preferable?). The proof is in the appendix of the article but the main issue are write down also in this strongly related question (Paradox in model selection (AIC, BIC, to explain or to predict?)).

Warning: if the true model is noiseless or the amount of data we have go to infinity, therefore never in practice, the bias-variance tradeoff disappear and the two minimization problem become equivalent. This discussion is related: Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict")

Answer 2

Multicollinearity

You could have multicollinearity which can make the variance in the error of estimates of $y$ and $\beta$ a lot different (typically the error in $y$ will have lower relative variance). See for more background: https://stats.stackexchange.com/tags/multicollinearity and https://en.wikipedia.org/wiki/Multicollinearity

Assuming a true linear model $y = X\beta_0 + \varepsilon$, estimate $\hat\beta$ and prediction $\hat y=X\hat\beta$. One can define, with $\lVert.\rVert$ the mean square error norm for example:

• Prediction error: $\lVert y-\hat y\rVert = \lVert X(\beta -\hat\beta)\rVert$ (note this definition omits the part related to the error term )
• Estimation error: $\lVert \beta -\hat\beta\rVert$

Let's express the variation of this prediction error $y-\hat y$ in terms of the estimation error $\beta -\hat\beta$.

$$\begin{array}{} \text{Var}[{y_k}-{\hat{y_k}}] &=& \text{Var}[\mathbf{X_k}(\boldsymbol{\beta} - \boldsymbol{\hat \beta})] \\ &=& \text{Var}[\sum_{i=1}^n X_{ik}(\beta_i - \hat \beta_i) ] \\ \\ &=& \sum_{i=1}^n X_{ik}^2 \text{Var}[\beta_i - \hat \beta_i] \\ && \quad + \, 2 \sum_\limits{1 \leq i<j\leq n} X_{ik} X_{jk} \text{Cov} [\beta_i - \hat \beta_i,\beta_j - \hat \beta_j] \\ \end{array}$$

This last line has an additional term with covariances. This makes that the error (variance) of $y$ can be a lot different from the error (variance) of $\beta$.

A very common issue is that the $\beta_i$ have a negative correlation (due to a positive correlation between the $X_i$, ie multicollinearity) and the variance of the predictions/estimates of $y$ might be (relatively) much smaller than the variance of the estimates of $\beta$.

Prediction versus Estimation

In addition to the issue of multicollinearity, there might be several other issues. The terms 'prediction' and 'estimation' can be ambiguous.

In this particular question, the terms are linked to the estimation of $y$ versus the estimation of $\beta$. However I can see the estimation/prediction of $y$ in various ways. When we fit data $y_i$ with a curve $\hat y_i$ (like the typical fitting, e.g. as in regression) then the $\hat y_i$ are in my vocabulary estimates of $y_i$ and not predictions of $y_i$.

With prediction, I am thinking about issues like generating prediction intervals (which differ from confidence intervals) or issues like extrapolating curves (e.g. extending trends, predicting new values based on old values).

This prediction of values of $y$ based on estimates of $\beta$ contains the same issue as the multicollinearity explained above, but it is more than that and I feel it is not right to conflate these two. The biggest issue is often the discrepance between 'estimating $y$ versus *estimating $\beta$'. In addition you have the discrepance between 'predicting $y$ versus *estimating $\beta$', which contains 'estimating $y$ versus *estimating $\beta$', but is also more than that (e.g. optimizing different loss functions, reducing the loss of our predictions, according to some loss function, is different from reducing the error of our estimates, according to some probability model/likelihood).