2
$\begingroup$

I was wondering if Ridge and LASSO shrinkage parameters (I am referring to the lambda in the canonical loss function) are functions of the number of features if one is targeting a "constant" shrinkage level?

An example is: let's suppose I determine my optimal shrinkage parameter to be some number x for a set of 4 features that are appropriately standardized. If I then bring a 5th feature into my feature set (also appropriately standardized), would I need to revise my previously selected shrinkage parameter if I wanted to target the same level of total shrinkage as before? Does the answer differ for Ridge vs. LASSO?

$\endgroup$
0

1 Answer 1

2
$\begingroup$

To ground the discussion I will consider a problem with $n$ points and $p$ parameters. Furthermore I will assume that the data matrix $\boldsymbol{X}\in\mathbb{R}^{n\times{p}}$ has been whitened, such that $\boldsymbol{X}^T\boldsymbol{X}=n\boldsymbol{I}_{p\times{p}}$.

Then the question relates to a regularized M-estimation problem with objective function of type $$E[\boldsymbol{c}]=\tfrac{1}{k}\|\boldsymbol{Xc}-\boldsymbol{y}\|_k^k+\tfrac{\lambda}{j}\|\boldsymbol{c}\|_j^j$$ for some regularization strength $\lambda\geq{0}$ and data/penalty norms $k,j\in\mathbb{N}$, and seeks a solution $\boldsymbol{c}\in\mathbb{R}^p$.

In the case of ridge regression ($k=j=2$) we have the solution $$(n+\lambda)\boldsymbol{c}=\boldsymbol{X}^T\boldsymbol{y}$$

From this, we can see that the magnitude of a coefficient $c_i$ will be impacted by $\lambda/n$, the relative magnitude of the penalty vs. the number of data points.

In terms of $p$, the answer depends on what "shrinkage level" means. From the above solution, the magnitude of $|c_i|$ is not impacted by $p$, but the norm $\|\boldsymbol{c}\|$ may scale with $p$.

However commonly "shrinkage" is not really of direct interest per se. Instead, the goal of the regularization is to prevent overfitting, in which case generalization error is more relevant. In this context, it would be more appropriate to set the value of $\lambda$ based on something like cross-validation. Then the general expectation would be that the optimal $\lambda$ would tend to increase with $p$.

I believe these same considerations should apply to the general problem, regardless of the values of $k$ and $j$ (i.e. also for LASSO, where $k=2$, $j=1$).

$\endgroup$
4
  • $\begingroup$ I keep thinking that this question is more complicated than it sounds. My thought experiment is suppose the p variables $X_1, \ldots X_p $ explain 10% of the variance of the response variable Y. One then adds another variable which explain 89% of the variance of Y and is orthogonal to all the others. Or suppose that $X_{p+1} = Y +Z$ where Z is pure noise. I can't believe that the optimal strategy is to do anything but associate almost, if not all of the 'lambda' to this new variable. In this last case, $\endgroup$
    – meh
    Commented Dec 28, 2016 at 3:14
  • $\begingroup$ I'm almost positive that the lasso would assign all the first p variable the value of 0. There are formulas in 'Elements of Statistical Learning' that explain for a given value of lambda what happens to the coefficients of the regularized regression. The stickler from my point of view for turning this into an answer rather than a comment is, "what is the meaning of the word optimal". $\endgroup$
    – meh
    Commented Dec 28, 2016 at 3:15
  • $\begingroup$ A recent free reference for LASSO is Hastie, Tibshirani, WainWright 2015. For my problem above, this gives the LASSO solution as $c=\mathrm{sgn}[\hat{c}]\big(\hat{c}-\frac{\lambda}{n}\big)_+$, where $\hat{c}=\frac{X^Ty}{n}$ is the OLS solution. $\endgroup$
    – GeoMatt22
    Commented Dec 28, 2016 at 4:30
  • $\begingroup$ The same solution is in my reference too- as well as the formula for ridge. There are two points. How does $\hat{c} $ change when a new variable is added and how does the optimal $\lambda $ change with new variables. That is what seems very hard to me in general. $\endgroup$
    – meh
    Commented Dec 28, 2016 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.