# How to choose smoothing parameter when aiming for estimation?

When using a smoothing method for a linear model such as Ridge Regression. It is usual to choose the regularization parameter $\lambda$ as the one that minimizes the cross validation error. However this criterion is focused on minimizing prediction error for new observations. I understand that the amount of regularization needed for predicting is usually less than for estimating. So, if estimation is the objective, what differents approachs are useful for determining $\lambda$?

I belive a possible answer should be estimating the estimation error (MSE) by bootstrap. If i had to choose from two different models that is what I would do. But now I have a continium of models. Ofcourse I could boostrap in a finite grid for $\lambda$. But Is this a good approach?

Here is a half-way solution. I do not conclude with a clear answer, but hopefully the thought process can be of some use.

Take a linear model $$y=X\beta+\varepsilon,$$ and assume it is correctly specified (i.e. the model is actually the one that generates the data).
Denote $\hat\beta:=\hat\beta^{ridge}(\lambda)$ the ridge estimate due to penalty intensity $\lambda$ of the parameter vector $\beta$.
The forecast error for a new observation will be \begin{aligned} \hat\varepsilon_i &= y_i-X_i\hat\beta \\ &= (X_i\beta+\varepsilon_i)-X_i\hat\beta \\ &= X_i(\beta-\hat\beta)+\varepsilon_i. \end{aligned} When aiming to minimize the expected squared forecast error $\mathbb{E}(\hat\varepsilon_i^2)$, we cannot do anything about $\varepsilon_i$ in the above expression, because $\varepsilon_i$ is totally unpredictable (after all, it is a random error).

Since $\varepsilon$ is also independent of the variables in $X$, the goal $$\mathbb{E}(\hat\varepsilon_i^2) = \mathbb{E}( [X_i(\beta-\hat\beta)+\varepsilon_i]^2 ) \rightarrow\min_{\hat\beta}$$ becomes $$\mathbb{E}([X_i(\beta-\hat\beta)]^2)\rightarrow\min_{\hat\beta}.$$ That is, we can focus on the part $X_i(\beta-\hat\beta)$ alone. But this is almost the same goal as estimating $\beta$ as accurately as possible in terms of mean squared error: $$\mathbb{E}((\beta-\hat\beta)^2)\rightarrow\min_{\hat\beta},$$ which is what you are interested in.

Clearly, the true $\beta$ is the optimal solution for both the forecasting and the estimation problems, there is no tradeoff there.

I am not sure how this works empirically, though, when $\lambda$ is selected by cross validation taking the mean squared forecast error as the loss to be minimized in the test sample. And I really do not see why less intensive regularization should be used for forecasting as compared to estimation, as you note in the question.

• Interesting approach. It gives me a lot to think. However I belive that it doesn't work when $\beta \in \mathbb{R}^p$ . $\mathbb{E}(\sum[X_i(\beta-\hat\beta)]^2) = \mathbb{E}(\|X(\beta-\hat\beta)\|^2) = \mathbb{E}([ (\beta-\hat\beta)'X'X(\beta-\hat\beta)])$ which is diferent to $\mathbb{E}(\|\beta - \hat \beta\|^2) = \mathbb{E}([ (\beta-\hat\beta)'(\beta-\hat\beta)])$. – Manuel Jan 26 '17 at 15:54
• I actually meant the $\mathbb{R}^p$ case. The conceptual difference is not that big, anyway. The difference that you note is just about scaling, and the question is, does this scaling matter. Or isn't it? – Richard Hardy Jan 26 '17 at 16:32
• I understand what you say about scaling and I belive it matters. If $X'X = \left( \begin{matrix} 1 & 0 \\ 0 & 0.1 \end{matrix} \right)$ then you would be assigning ten times more importance to estimating correctly $\beta_1$ than $\beta_2$. I should clarify that I am working in a theoretical model and not focusing in a practical case. – Manuel Jan 26 '17 at 17:08
• @Manuel, But in ridge regression the regressors are typically scaled, so there would be all ones on the diagonal. – Richard Hardy Jan 26 '17 at 17:42
• How about $X'X = \left( \begin{matrix} 1 & .9 \\ .9 & 1\end{matrix}\right)$ ? This matrix is scaled and simetric. It has eigenvalues $1.9$ and $0.1$ for the eigenvectors $(1,1)$ and $(-1,1)$ So ridge would be estimating $19$ betters the projection of $\beta$ over the first eigenvector than over the second. – Manuel Jan 26 '17 at 20:38