2. What's the intuition and math behind elastic net?

#3. What if you add additional $L^q$ norms?

3. What if you add additional $L^q$ norms?

deleted 23 characters in body

Source Link

edited May 26, 2019 at 17:42

Sycorax ♦

94.1k
23
236
390

In On this threadWhy do we only see $L_1$ and $L_2$ regularization but not other norms?, @whuber offers this comment:

added 600 characters in body

Source Link

edited Jun 23, 2018 at 0:13

Sycorax ♦

94.1k
23
236
390

Comments to my post have pointed out that the advantages of elastic net are not unqualified. I persist in my belief that the generality of the elastic net regression is still preferable to either $L_1$$L^1$ or $L_2$$L^2$ regularization on its own. Specifically, I think that the points of contention between myself and others are directly tied to what assumptions we are willing to make about the modeling process. In the presence of strong knowledge about the underlying data, some methods will be preferred to others. However, my preference for elastic net is rooted in my skepticism that one will confidently know that $L_1$$L^1$ or $L_2$$L^2$ is the true model.

Claim: Modestly large data will not permit discovery of $L_1$$L^1$ or $L_2$$L^2$ solutions as preferred, even in cases when the $L_1$$L^1$ or $L_2$$L^2$ solution is the true model.

This is true, but at the step where one is contemplating which method to use, one will not know which of elastic net, ridge or LASSO is the best. If one reasons that the best solution must be LASSO or ridge regression, then we're in the domain of claim (1). If we're still uncertain which is best, then we can test LASSO, ridge and elastic net solutions, and make a choice of a final model at that point (or, if you're an academic, just write your paper about all 3three). This situation of prior uncertainty will either place us in the domain of claim (2), where the true model is LASSO/ridge but we did not know so ahead of time, and we accidentally select the wrong model due to poorly identified hyperparameters, or elastic net is actually the best solution.

#3. What if you add additional $L_q$$L^q$ norms?

Let me suggest one further argument against your point of view that elastic net is uniformly better than lasso or ridge alone. Imagine that we add another penalty to the elastic net cost function, e.g. an $L_3$$L^3$ cost, with a hyperparameter $\gamma$. I don't think there is much research on that, but I would bet you that if you do a cross-validation search on a 3d parameter grid, then you will get $\gamma\not =0$ as the optimal value. If so, would you then argue that it is always a good idea to include $L_3$$L^3$ cost too?.

$L_1$$L^1$ regularization producestends to produce sparse solutions, but also tends to select the feature most strongly correlated with the outcome and zero out the rest. Moreover, in a data set with $n$ observations, it can select at most $n$ features. $L_2$ regularization is suited to deal with ill-posed problems resulting from highly (or perfectly) correlated features. In a data set with $p$ features, $L_2$ regularization can be used to uniquely identify a model in the $p>n$ case.

But I am not aware of the statistical properties for $L_3$$L^3$ regularization. In the problems I've worked on, we generally face both problems: the inclusion of poorly correlated features (hypotheses that are not borne out by the data), and colinearco-linear features. In a situation where I understood what problem

Indeed, there are compelling reasons that $L_3$ regularization might solve$L^1$ and also suspected$L^2$ penalties on parameters are the same problems to be present in my data, perhaps it would merit inclusion. But I don't think it makes sense to add penalization terms arbitrarilyonly ones typically used.

(As an aside, I think it might be better to frame the problem as a convex sumOn this thread, so that@whuber offers this comment:

I haven't investigated this question specifically, but experience with similar situations suggests there may be a nice qualitative answer: all norms that are second differentiable at the origin will be locally equivalent to each other, of which the $L^2$ norm is the standard. All other norms will not be differentiable at the origin and $L^1$ qualitatively reproduces their behavior. That covers the gamut. In effect, a linear combination of an $L^1$ and $L^2$ norm approximates any norm to second order at the origin--and this is what matters most in regression without outlying residuals.

So we have values $\alpha_1, \alpha_2, \alpha_3$, one corresponding to each ofcan effectively cover the three coefficientrange of options which could possibly be provided by $L^q$ norms, such that as combinations of $\sum_{i=1}^3 \alpha_i=1$$L^1$ and $\alpha_i>0 \forall i.$)$L^2$ norms -- all without requiring additional hyperparameter tuning.

Comments to my post have pointed out that the advantages of elastic net are not unqualified. I persist in my belief that the generality of the elastic net regression is still preferable to either $L_1$ or $L_2$ regularization on its own. Specifically, I think that the points of contention between myself and others are directly tied to what assumptions we are willing to make about the modeling process. In the presence of strong knowledge about the underlying data, some methods will be preferred to others. However, my preference for elastic net is rooted in my skepticism that one will confidently know that $L_1$ or $L_2$ is the true model.

Claim: Modestly large data will not permit discovery of $L_1$ or $L_2$ solutions as preferred, even in cases when the $L_1$ or $L_2$ solution is the true model.

This is true, but at the step where one is contemplating which method to use, one will not know which of elastic net, ridge or LASSO is the best. If one reasons that the best solution must be LASSO or ridge regression, then we're in the domain of claim (1). If we're still uncertain which is best, then we can test LASSO, ridge and elastic net solutions, and make a choice of a final model at that point (or, if you're an academic, just write your paper about all 3). This situation of prior uncertainty will either place us in the domain of claim (2), where the true model is LASSO/ridge but we did not know so ahead of time, and we accidentally select the wrong model due to poorly identified hyperparameters, or elastic net is actually the best solution.

#3. What if you add additional $L_q$ norms?

Let me suggest one further argument against your point of view that elastic net is uniformly better than lasso or ridge alone. Imagine that we add another penalty to the elastic net cost function, e.g. an $L_3$ cost, with a hyperparameter $\gamma$. I don't think there is much research on that, but I would bet you that if you do a cross-validation search on a 3d parameter grid, then you will get $\gamma\not =0$ as the optimal value. If so, would you then argue that it is always a good idea to include $L_3$ cost too?

$L_1$ regularization produces sparse solutions, but tends to select the feature most strongly correlated with the outcome and zero out the rest. Moreover, in a data set with $n$ observations, it can select at most $n$ features. $L_2$ regularization is suited to deal with ill-posed problems resulting from highly (or perfectly) correlated features. In a data set with $p$ features, $L_2$ regularization can be used to uniquely identify a model in the $p>n$ case.

But I am not aware of the statistical properties for $L_3$ regularization. In the problems I've worked on, we generally face both problems: the inclusion of poorly correlated features (hypotheses that are not borne out by the data), and colinear features. In a situation where I understood what problem $L_3$ regularization might solve and also suspected the same problems to be present in my data, perhaps it would merit inclusion. But I don't think it makes sense to add penalization terms arbitrarily.

(As an aside, I think it might be better to frame the problem as a convex sum, so that we have values $\alpha_1, \alpha_2, \alpha_3$, one corresponding to each of the three coefficient norms, such that $\sum_{i=1}^3 \alpha_i=1$ and $\alpha_i>0 \forall i.$)

Comments to my post have pointed out that the advantages of elastic net are not unqualified. I persist in my belief that the generality of the elastic net regression is still preferable to either $L^1$ or $L^2$ regularization on its own. Specifically, I think that the points of contention between myself and others are directly tied to what assumptions we are willing to make about the modeling process. In the presence of strong knowledge about the underlying data, some methods will be preferred to others. However, my preference for elastic net is rooted in my skepticism that one will confidently know that $L^1$ or $L^2$ is the true model.

Claim: Modestly large data will not permit discovery of $L^1$ or $L^2$ solutions as preferred, even in cases when the $L^1$ or $L^2$ solution is the true model.

This is true, but at the step where one is contemplating which method to use, one will not know which of elastic net, ridge or LASSO is the best. If one reasons that the best solution must be LASSO or ridge regression, then we're in the domain of claim (1). If we're still uncertain which is best, then we can test LASSO, ridge and elastic net solutions, and make a choice of a final model at that point (or, if you're an academic, just write your paper about all three). This situation of prior uncertainty will either place us in the domain of claim (2), where the true model is LASSO/ridge but we did not know so ahead of time, and we accidentally select the wrong model due to poorly identified hyperparameters, or elastic net is actually the best solution.

#3. What if you add additional $L^q$ norms?

Let me suggest one further argument against your point of view that elastic net is uniformly better than lasso or ridge alone. Imagine that we add another penalty to the elastic net cost function, e.g. an $L^3$ cost, with a hyperparameter $\gamma$. I don't think there is much research on that, but I would bet you that if you do a cross-validation search on a 3d parameter grid, then you will get $\gamma\not =0$ as the optimal value. If so, would you then argue that it is always a good idea to include $L^3$ cost too.

$L^1$ regularization tends to produce sparse solutions, but also tends to select the feature most strongly correlated with the outcome and zero out the rest. Moreover, in a data set with $n$ observations, it can select at most $n$ features. $L_2$ regularization is suited to deal with ill-posed problems resulting from highly (or perfectly) correlated features. In a data set with $p$ features, $L_2$ regularization can be used to uniquely identify a model in the $p>n$ case.

But I am not aware of the statistical properties for $L^3$ regularization. In the problems I've worked on, we generally face both problems: the inclusion of poorly correlated features (hypotheses that are not borne out by the data), and co-linear features.

Indeed, there are compelling reasons that $L^1$ and $L^2$ penalties on parameters are the only ones typically used.

On this thread, @whuber offers this comment:

I haven't investigated this question specifically, but experience with similar situations suggests there may be a nice qualitative answer: all norms that are second differentiable at the origin will be locally equivalent to each other, of which the $L^2$ norm is the standard. All other norms will not be differentiable at the origin and $L^1$ qualitatively reproduces their behavior. That covers the gamut. In effect, a linear combination of an $L^1$ and $L^2$ norm approximates any norm to second order at the origin--and this is what matters most in regression without outlying residuals.

So we can effectively cover the range of options which could possibly be provided by $L^q$ norms as combinations of $L^1$ and $L^2$ norms -- all without requiring additional hyperparameter tuning.