11
$\begingroup$

After reading Chapter 3 in the Elements of Statistical Learning (Hastie, Tibshrani & Friedman), I wondered if it was possible to implement the famous shrinkage methods quoted on the title of this question given a covariance structure, i.e., minimize the (perhaps more general) quantity $$(\vec{y}-X\vec{\beta})^TV^{-1}(\vec{y}-X\vec{\beta})+\lambda f(\beta),\ \ \ (1)$$

instead of the usual $$(\vec{y}-X\vec{\beta})(\vec{y}-X\vec{\beta})+\lambda f(\beta).\ \ \ \ \ \ \ \ \ \ \ \ (2)$$ This was mainly motivated by the fact that in my particular application, we have different variances for the $\vec{y}$ (and sometimes even a covariance structure that can be estimated) and I would love to include them in the regression. I did it for ridge regression: at least with my implementation of it in Python/C, I see that there are important differences in the paths that the coefficients trace, which is also notable when comparing the cross-validation curves in both cases.

I was now preparing to try to implement the LASSO via Least Angle Regression, but in order to do it I have to prove first that all its nice properties are still valid when minimizing $(1)$ instead of $(2)$. So far, I haven't seen any work that actually does all this, but some time ago I also read a quote that said something like "those who don't know statistics are doomed to rediscover it" (by Brad Efron, perhaps?), so that's why I'm asking here first (given that I'm a relative newcomer to the statistics literature): is this already done somewhere for these models? Is it implemented in R in some way? (including the solution and implementation of the ridge by minimizing $(1)$ instead of $(2)$, which is what's implemented in the lm.ridge code in R)?

Thanks in advance for your answers!

$\endgroup$
13
$\begingroup$

If we know the Cholesky decomposition $V^{-1} = L^TL$, say, then $$(y - X\beta)^T V^{-1} (y - X\beta) = (Ly - LX\beta)^T (Ly - LX\beta)$$ and we can use standard algorithms (with whatever penalization function one prefers) by replacing the response with the vector $Ly$ and the predictors with the matrix $LX$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.