Skip to main content
added 239 characters in body
Source Link
user795305
  • 3k
  • 1
  • 25
  • 43

This can likely be solved with the elastic net. Suppose we have response $y \in \mathbb{R}^n$ and design matrix $X \in \mathbb{R}^{n \times p}$.

By lagrangian duality, the estimator $$\hat\theta_{(a)} = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 \leq 1} \|y-X\theta\|_2^2$$ is also the minimizer $$\hat\theta_{(b)} = \arg\min_{\theta \in \mathbb{R}^p} \frac{1}{2n} \|y - X \theta\|_2^2 + \lambda \|\theta\|_1 + \mu \|\theta\|_2^2$$ for some dual variables $\lambda$ and $\mu$. That is, $\hat\theta_{(a)} = \hat\theta_{(b)} \left( =: \hat\theta \right)$. That means that all you have to do is tune $\mu$ appropriately to achieve the desired $\ell_2$ norm of the estimate $\hat\theta_{(b)}$. Further, you have to be sure that $\lambda$ is not "too large" to ensure that there exists some $\mu$ so that the estimate $\hat\theta_{(b)}$ can have $\ell_2$ norm equal to one.

The only hitch here is that the estimator you are actually interested in is $$\tilde\theta = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 = 1} \|y-X\theta\|_2^2.$$ However$$\tilde\theta = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 = 1} \|y-X\theta\|_2^2,$$ and it isn't necessarily true that $\hat\beta = \tilde\beta$. However, we can see that $\tilde\theta = \hat\theta$ if $\mu \ne 0$ by complementary slackness. (Notice, of course, that this $\mu$ was specifically defined to be the dual variable corresponding to the bound of $1$--it isn't arbitrary.) In general though, it's possible that $\|\hat\theta_{(b)} \|_2 > 1$ for all tuning parameterspossible choices of $(\lambda, \mu)$. In which case, the problem you're trying to solve is nonconvex and elastic net would not work: you'd have to do a more complicated optimization method, perhaps like hxd mentions.

It's also worth pointing out that $\|\hat\theta_{(b)}\|_2$ is not necessarily monotonically decreasing with respect to $\lambda$. See, for instance, Can $\|\beta^*\|_2$ increase when $\lambda$ increases in Lasso?.

This can likely be solved with the elastic net. Suppose we have response $y \in \mathbb{R}^n$ and design matrix $X \in \mathbb{R}^{n \times p}$.

By lagrangian duality, the estimator $$\hat\theta_{(a)} = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 \leq 1} \|y-X\theta\|_2^2$$ is also the minimizer $$\hat\theta_{(b)} = \arg\min_{\theta \in \mathbb{R}^p} \frac{1}{2n} \|y - X \theta\|_2^2 + \lambda \|\theta\|_1 + \mu \|\theta\|_2^2$$ for some dual variables $\lambda$ and $\mu$. That is, $\hat\theta_{(a)} = \hat\theta_{(b)} \left( =: \hat\theta \right)$. That means that all you have to do is tune $\mu$ appropriately to achieve the desired $\ell_2$ norm of the estimate $\hat\theta_{(b)}$. Further, you have to be sure that $\lambda$ is not "too large" to ensure that there exists some $\mu$ so that the estimate $\hat\theta_{(b)}$ can have $\ell_2$ norm equal to one.

The only hitch here is that the estimator you are actually interested in is $$\tilde\theta = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 = 1} \|y-X\theta\|_2^2.$$ However, we can see that $\tilde\theta = \hat\theta$ if $\mu \ne 0$. (Notice, of course, that this $\mu$ was specifically defined to be the dual variable corresponding to the bound of $1$--it isn't arbitrary.) In general though, it's possible that $\|\hat\theta_{(b)} \|_2 > 1$ for all tuning parameters $(\lambda, \mu)$. In which case, the problem you're trying to solve is nonconvex and elastic net would not work: you'd have to do a more complicated optimization method, perhaps like hxd mentions.

This can likely be solved with the elastic net. Suppose we have response $y \in \mathbb{R}^n$ and design matrix $X \in \mathbb{R}^{n \times p}$.

By lagrangian duality, the estimator $$\hat\theta_{(a)} = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 \leq 1} \|y-X\theta\|_2^2$$ is also the minimizer $$\hat\theta_{(b)} = \arg\min_{\theta \in \mathbb{R}^p} \frac{1}{2n} \|y - X \theta\|_2^2 + \lambda \|\theta\|_1 + \mu \|\theta\|_2^2$$ for some dual variables $\lambda$ and $\mu$. That is, $\hat\theta_{(a)} = \hat\theta_{(b)} \left( =: \hat\theta \right)$. That means that all you have to do is tune $\mu$ appropriately to achieve the desired $\ell_2$ norm of the estimate $\hat\theta_{(b)}$. Further, you have to be sure that $\lambda$ is not "too large" to ensure that there exists some $\mu$ so that the estimate $\hat\theta_{(b)}$ can have $\ell_2$ norm equal to one.

The only hitch here is that the estimator you are actually interested in is $$\tilde\theta = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 = 1} \|y-X\theta\|_2^2,$$ and it isn't necessarily true that $\hat\beta = \tilde\beta$. However, we can see that $\tilde\theta = \hat\theta$ if $\mu \ne 0$ by complementary slackness. (Notice, of course, that this $\mu$ was specifically defined to be the dual variable corresponding to the bound of $1$--it isn't arbitrary.) In general though, it's possible that $\|\hat\theta_{(b)} \|_2 > 1$ for all possible choices of $(\lambda, \mu)$. In which case, the problem you're trying to solve is nonconvex and elastic net would not work: you'd have to do a more complicated optimization method, perhaps like hxd mentions.

It's also worth pointing out that $\|\hat\theta_{(b)}\|_2$ is not necessarily monotonically decreasing with respect to $\lambda$. See, for instance, Can $\|\beta^*\|_2$ increase when $\lambda$ increases in Lasso?.

Source Link
user795305
  • 3k
  • 1
  • 25
  • 43

This can likely be solved with the elastic net. Suppose we have response $y \in \mathbb{R}^n$ and design matrix $X \in \mathbb{R}^{n \times p}$.

By lagrangian duality, the estimator $$\hat\theta_{(a)} = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 \leq 1} \|y-X\theta\|_2^2$$ is also the minimizer $$\hat\theta_{(b)} = \arg\min_{\theta \in \mathbb{R}^p} \frac{1}{2n} \|y - X \theta\|_2^2 + \lambda \|\theta\|_1 + \mu \|\theta\|_2^2$$ for some dual variables $\lambda$ and $\mu$. That is, $\hat\theta_{(a)} = \hat\theta_{(b)} \left( =: \hat\theta \right)$. That means that all you have to do is tune $\mu$ appropriately to achieve the desired $\ell_2$ norm of the estimate $\hat\theta_{(b)}$. Further, you have to be sure that $\lambda$ is not "too large" to ensure that there exists some $\mu$ so that the estimate $\hat\theta_{(b)}$ can have $\ell_2$ norm equal to one.

The only hitch here is that the estimator you are actually interested in is $$\tilde\theta = \arg\min_{\theta \in \mathbb{R}^p \, : \, \|\theta\|_1 \leq C_1, \, \|\theta\|_2 = 1} \|y-X\theta\|_2^2.$$ However, we can see that $\tilde\theta = \hat\theta$ if $\mu \ne 0$. (Notice, of course, that this $\mu$ was specifically defined to be the dual variable corresponding to the bound of $1$--it isn't arbitrary.) In general though, it's possible that $\|\hat\theta_{(b)} \|_2 > 1$ for all tuning parameters $(\lambda, \mu)$. In which case, the problem you're trying to solve is nonconvex and elastic net would not work: you'd have to do a more complicated optimization method, perhaps like hxd mentions.