# How to solve an adaptive lasso model?

Assuming we are working with a linear regression model, lasso penalization solves:

$$\begin{equation} \min_{\beta}\left\{\left\lVert y-X\beta\right\rVert_2^2+\lambda\sum_{j=1}^p \left\vert \beta_j\right\vert\right\} \end{equation}$$

$$\begin{equation} \min_{\beta}\left\{\left\lVert y-X\beta\right\rVert_2^2+\lambda\sum_{j =1}^p w_j\left\vert \beta_j\right\vert\right\} \end{equation}$$

where $$w$$ defines a vector of weights previusly defined by the researcher.

This adaptive idea was initially proposed in "The adaptive Lasso and its Oracle Properties" (Journal of the American Statistical Association 101.476 (2006): 1418-1429.), and in this paper, in section 3.5, the authors state that it is possible to solve the adaptive lasso penalization using any algorithm for solving lasso penalization, just taking into account the following steps:

1. Define $$x_j^{**}=x_j/\hat{w_j}, j=1,\ldots,p$$
2. Solve the lasso problem $$\begin{equation} \hat{\beta}^{**}=\arg\min_{\beta}\left\{\left\lVert y- \sum_{j=1}^px_j^{**}\beta_j\right\rVert^2+\lambda\sum_{j=1}^p\left\vert \beta_j\right\vert\right\} \end{equation}$$
3. Output $$\hat{\beta_j}^*=\hat{\beta}_j^{**}/w_j$$

So here they state that just by dividing each predictor column by the weight associated to that predictor, solving the lasso model and dividing the solution obtained here by the weights, we get the adaptive lasso solution. They say that the demonstration of this fact is very simple and it is therefore omitted, but I have been unable to mathematically check this. I would appreciate any hint on how to solve this doubt.

• This is straightforward algebra, not statistics: it rests on the observation that $x\beta = (xw)(\beta/w)$ for any nonzero weight $w.$ – whuber Mar 22 '19 at 12:58

Let's define a new scaled $$\boldsymbol{\beta}$$ vector, and call it $$\boldsymbol{\beta^w}$$. It is an element-wise multiplication of $$\mathbf{w}$$ and $$\boldsymbol{\beta}$$, such that $$\beta^w_j = w_j\cdot\beta_j$$. We can represent the old $$\boldsymbol{\beta}$$ in terms of the new one as $$\beta_j = \frac{1}{w_j}\cdot\beta^w_j$$. We can with a little abuse of notation write $$\boldsymbol{\beta} = \frac{1}{\mathbf{w}}\bullet \boldsymbol{\beta^w}$$, where the division is element-wise and $$\bullet$$ is element-wise multiplication. Now let's replace this in the minimisation problem to get
\begin{align} \text{arg}\min_{\boldsymbol\beta^w}\left\{\left\lVert y-X\left(\frac{1}{\mathbf{w}}\bullet \boldsymbol{\beta^w}\right)\right\rVert_2^2+\lambda\sum_{j =1}^p w_j\left\vert \frac{1}{w_j}\cdot\beta^w_j\right\vert\right\} \end{align}
Now we can multiply into the absolute value in the latter sum to get rid of the $$w_j$$, so it becomes:
\begin{align} \text{arg}\min_{\boldsymbol\beta^w}\left\{\left\lVert y-X\left(\frac{1}{\mathbf{w}}\bullet \boldsymbol{\beta^w}\right)\right\rVert_2^2+\lambda\sum_{j =1}^p \left\vert \beta^w_j\right\vert\right\} \end{align}