4
$\begingroup$

Assuming we are working with a linear regression model, lasso penalization solves:

\begin{equation} \min_{\beta}\left\{\left\lVert y-X\beta\right\rVert_2^2+\lambda\sum_{j=1}^p \left\vert \beta_j\right\vert\right\} \end{equation}

While adaptive lasso penalization solves:

\begin{equation} \min_{\beta}\left\{\left\lVert y-X\beta\right\rVert_2^2+\lambda\sum_{j =1}^p w_j\left\vert \beta_j\right\vert\right\} \end{equation}

where $w$ defines a vector of weights previusly defined by the researcher.

This adaptive idea was initially proposed in "The adaptive Lasso and its Oracle Properties" (Journal of the American Statistical Association 101.476 (2006): 1418-1429.), and in this paper, in section 3.5, the authors state that it is possible to solve the adaptive lasso penalization using any algorithm for solving lasso penalization, just taking into account the following steps:

  1. Define $x_j^{**}=x_j/\hat{w_j}, j=1,\ldots,p$
  2. Solve the lasso problem \begin{equation} \hat{\beta}^{**}=\arg\min_{\beta}\left\{\left\lVert y- \sum_{j=1}^px_j^{**}\beta_j\right\rVert^2+\lambda\sum_{j=1}^p\left\vert \beta_j\right\vert\right\} \end{equation}
  3. Output $\hat{\beta_j}^*=\hat{\beta}_j^{**}/w_j$

So here they state that just by dividing each predictor column by the weight associated to that predictor, solving the lasso model and dividing the solution obtained here by the weights, we get the adaptive lasso solution. They say that the demonstration of this fact is very simple and it is therefore omitted, but I have been unable to mathematically check this. I would appreciate any hint on how to solve this doubt.

$\endgroup$
  • 1
    $\begingroup$ This is straightforward algebra, not statistics: it rests on the observation that $x\beta = (xw)(\beta/w)$ for any nonzero weight $w.$ $\endgroup$ – whuber Mar 22 at 12:58
5
+50
$\begingroup$

I'm going to expand on the comment from whuber, since I know how annoying it can be to be stuck on these kind of things, in case his explanation was not enough.

Let's define a new scaled $\boldsymbol{\beta}$ vector, and call it $\boldsymbol{\beta^w}$. It is an element-wise multiplication of $\mathbf{w}$ and $\boldsymbol{\beta}$, such that $\beta^w_j = w_j\cdot\beta_j$. We can represent the old $\boldsymbol{\beta}$ in terms of the new one as $\beta_j = \frac{1}{w_j}\cdot\beta^w_j$. We can with a little abuse of notation write $\boldsymbol{\beta} = \frac{1}{\mathbf{w}}\bullet \boldsymbol{\beta^w}$, where the division is element-wise and $\bullet$ is element-wise multiplication. Now let's replace this in the minimisation problem to get

\begin{align} \text{arg}\min_{\boldsymbol\beta^w}\left\{\left\lVert y-X\left(\frac{1}{\mathbf{w}}\bullet \boldsymbol{\beta^w}\right)\right\rVert_2^2+\lambda\sum_{j =1}^p w_j\left\vert \frac{1}{w_j}\cdot\beta^w_j\right\vert\right\} \end{align}

Now we can multiply into the absolute value in the latter sum to get rid of the $w_j$, so it becomes:

\begin{align} \text{arg}\min_{\boldsymbol\beta^w}\left\{\left\lVert y-X\left(\frac{1}{\mathbf{w}}\bullet \boldsymbol{\beta^w}\right)\right\rVert_2^2+\lambda\sum_{j =1}^p \left\vert \beta^w_j\right\vert\right\} \end{align}

Now you should simplify the least squares term into a sum, and then it should be obvious why the covariates need to be scaled. If that is the step that was stopping you, then I'll gladly expand this further.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.