2
$\begingroup$

The lasso estimator is define as $argmin_{\beta}~ MSE +\lambda \parallel \beta\parallel_1$ I am wondering if there is an alternative penalty function that is $C^\infty$ and that preserves the sparsity of the lasso estimator.

$\endgroup$
  • $\begingroup$ $C$ is the regularity. It means that the penalty function can be derived infinite time. mathworld.wolfram.com/C-InfinityFunction.html $\endgroup$ – Donbeo Oct 6 '14 at 14:41
  • $\begingroup$ The problem is not the minimization. I want to train a neural network with the extreme learning machine method. In order to do that I need a $C^\infty$ functions on the nodes. And I want to use a lasso like function on the nodes. $\endgroup$ – Donbeo Oct 6 '14 at 16:57
2
$\begingroup$

This is not possible. It's exactly the nondifferentiability at zero that induces sparsity.

Indeed, by the KKT conditions, we see that any $\hat\beta = \arg\min_\beta \mathrm{MSE} + \lambda \mathrm{pen}(\beta)$ maybe rewritten as $$\hat\beta = \arg\min_{\beta \, : \, \mathrm{pen}(\beta) \leq C} \mathrm{MSE}$$ for some constant $C$. (Note that this direction holds even if the penalty $\mathrm{pen}$ is nonconvex.) Since sparsity is induced by the level sets $\{\beta \, : \, \mathrm{pen}(\beta) = C\}$ having "corners" on the coordinate axes, all we need to do is determine whether a differentiable function can have "corners" in its level sets. However, by the implicit function theorem, we know that this isn't possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.