When using a smoothing method for a linear model such as Ridge Regression. It is usual to choose the regularization parameter $\lambda$ as the one that minimizes the cross validation error. However this criterion is focused on minimizing prediction error for new observations. I understand that the amount of regularization needed for predicting is usually less than for estimating. So, if estimation is the objective, what differents approachs are useful for determining $\lambda$?

I belive a possible answer should be estimating the estimation error (MSE) by bootstrap. If i had to choose from two different models that is what I would do. But now I have a continium of models. Ofcourse I could boostrap in a finite grid for $\lambda$. But Is this a good approach?


1 Answer 1


Here is a half-way solution. I do not conclude with a clear answer, but hopefully the thought process can be of some use.

Take a linear model $$ y=X\beta+\varepsilon, $$ and assume it is correctly specified (i.e. the model is actually the one that generates the data).
Denote $\hat\beta:=\hat\beta^{ridge}(\lambda)$ the ridge estimate due to penalty intensity $\lambda$ of the parameter vector $\beta$.
The forecast error for a new observation will be $$ \begin{aligned} \hat\varepsilon_i &= y_i-X_i\hat\beta \\ &= (X_i\beta+\varepsilon_i)-X_i\hat\beta \\ &= X_i(\beta-\hat\beta)+\varepsilon_i. \end{aligned} $$ When aiming to minimize the expected squared forecast error $\mathbb{E}(\hat\varepsilon_i^2)$, we cannot do anything about $\varepsilon_i$ in the above expression, because $\varepsilon_i$ is totally unpredictable (after all, it is a random error).

Since $\varepsilon$ is also independent of the variables in $X$, the goal $$ \mathbb{E}(\hat\varepsilon_i^2) = \mathbb{E}( [X_i(\beta-\hat\beta)+\varepsilon_i]^2 ) \rightarrow\min_{\hat\beta} $$ becomes $$ \mathbb{E}([X_i(\beta-\hat\beta)]^2)\rightarrow\min_{\hat\beta}. $$ That is, we can focus on the part $X_i(\beta-\hat\beta)$ alone. But this is almost the same goal as estimating $\beta$ as accurately as possible in terms of mean squared error: $$ \mathbb{E}((\beta-\hat\beta)^2)\rightarrow\min_{\hat\beta}, $$ which is what you are interested in.

Clearly, the true $\beta$ is the optimal solution for both the forecasting and the estimation problems, there is no tradeoff there.

I am not sure how this works empirically, though, when $\lambda$ is selected by cross validation taking the mean squared forecast error as the loss to be minimized in the test sample. And I really do not see why less intensive regularization should be used for forecasting as compared to estimation, as you note in the question.

  • $\begingroup$ Interesting approach. It gives me a lot to think. However I belive that it doesn't work when $\beta \in \mathbb{R}^p$ . $\mathbb{E}(\sum[X_i(\beta-\hat\beta)]^2) = \mathbb{E}(\|X(\beta-\hat\beta)\|^2) = \mathbb{E}([ (\beta-\hat\beta)'X'X(\beta-\hat\beta)])$ which is diferent to $\mathbb{E}(\|\beta - \hat \beta\|^2) = \mathbb{E}([ (\beta-\hat\beta)'(\beta-\hat\beta)])$. $\endgroup$
    – Manuel
    Jan 26, 2017 at 15:54
  • $\begingroup$ I actually meant the $\mathbb{R}^p$ case. The conceptual difference is not that big, anyway. The difference that you note is just about scaling, and the question is, does this scaling matter. Or isn't it? $\endgroup$ Jan 26, 2017 at 16:32
  • $\begingroup$ I understand what you say about scaling and I belive it matters. If $X'X = \left( \begin{matrix} 1 & 0 \\ 0 & 0.1 \end{matrix} \right)$ then you would be assigning ten times more importance to estimating correctly $\beta_1$ than $\beta_2$. I should clarify that I am working in a theoretical model and not focusing in a practical case. $\endgroup$
    – Manuel
    Jan 26, 2017 at 17:08
  • $\begingroup$ @Manuel, But in ridge regression the regressors are typically scaled, so there would be all ones on the diagonal. $\endgroup$ Jan 26, 2017 at 17:42
  • $\begingroup$ How about $X'X = \left( \begin{matrix} 1 & .9 \\ .9 & 1\end{matrix}\right)$ ? This matrix is scaled and simetric. It has eigenvalues $1.9$ and $0.1$ for the eigenvectors $(1,1)$ and $(-1,1)$ So ridge would be estimating $19$ betters the projection of $\beta$ over the first eigenvector than over the second. $\endgroup$
    – Manuel
    Jan 26, 2017 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.