6
$\begingroup$

Are there some special cases, where the Ridge Regression can also lead to coefficients that are zero ? It is widely known, that lasso is shrinking coefficients towards or on zero, while the ridge Regression cant shrink coefficients to zero

$\endgroup$
  • 1
    $\begingroup$ Of course! If the least squares estimates are zero, then Ridge Regression will always produce zeros. What would be of interest is to find any other situation :-). $\endgroup$ – whuber Aug 28 '18 at 18:53
  • $\begingroup$ In which cases an ols coefficient can be exactly zero ? $\endgroup$ – Vala Aug 28 '18 at 18:54
  • 2
    $\begingroup$ This will happen whenever the response variable is orthogonal to each of the explanatory variables. $\endgroup$ – whuber Aug 28 '18 at 18:56
  • $\begingroup$ Would there be also the requirement that the predictors are orthogonal to each other, or would it be enough if just the correlation to the respone is zero $\endgroup$ – Vala Aug 28 '18 at 18:57
  • 1
    $\begingroup$ See also stats.stackexchange.com/questions/74542/… for an explanation of why Ridge cannot shrink the parameters to zero (unless they start there, as @whuber observes.) $\endgroup$ – jbowman Aug 28 '18 at 20:14
13
$\begingroup$

Suppose, as in the case of least squares methods, you are trying to solve a statistical estimation problem for a (vector-valued) parameter $\beta$ by minimizing an objective function $Q(\beta)$ (such as the sum of squares of the residuals). Ridge Regression "regularizes" the problem by adding a non-negative linear combination of the squares of the parameter, $P(\beta).$ $P$ is (obviously) differentiable with a unique global minimum at $\beta=0.$

The question asks, when is it possible for the global minimum of $Q+P$ to occur at $\beta=0$? Assume, as in least squares methods, that $Q$ is differentiable in a neighborhood of $0.$ Because $0$ is a global minimum for $Q+P$ it is a local minimum, implying all its partial derivatives are $0.$ The sum rule of differentiation implies

$$\frac{\partial}{\partial \beta_i}(Q(\beta) + P(\beta)) = \frac{\partial}{\partial \beta_i}Q(\beta) + \frac{\partial}{\partial \beta_i}P(\beta) = Q_i(\beta) + P_i(\beta)$$ is zero at $\beta=0.$ But since $P_i(0)=0$ for all $i,$ this implies $Q_i(0)=0$ for all $i,$ which makes $0$ at least a local minimum for the original objective function $Q.$ In the case of any least squares technique every local minimum is also a global minimum. This compels us to conclude that

Quadratic regularization of Least Squares procedures ("Ridge Regression") has $\beta=0$ as a solution if and only if $\beta=0$ is a solution of the original unregularized problem.

$\endgroup$
  • 1
    $\begingroup$ As pointed out by Martijn Weterings, it would also shrink coefficients to zero if t=0, or lambda converges to infinity. Regarding to the latter one: Could Ridge shrink a coefficient to zero for a sufficient large Tuning Parameter or is it just a theoretical concept that if lambda converges to infinity then the coefficient will be converge also to zero $\endgroup$ – Vala Aug 28 '18 at 19:14
  • 2
    $\begingroup$ Lambda going to infinity is the equivalent of minimizing $Q/\lambda + P.$ I hope it's easy to see that for sufficiently large $\lambda$ the solutions will have to be close to $\beta=0,$ guaranteeing convergence to $\beta=0$ in the limit. $\endgroup$ – whuber Aug 28 '18 at 19:20
  • $\begingroup$ There will be also close to zero, but cant get exactly zero, Right ? $\endgroup$ – Vala Aug 28 '18 at 19:34
  • $\begingroup$ Please re-read the conclusion of my answer: I can't think of any way to make it clearer. $\endgroup$ – whuber Aug 28 '18 at 19:36
  • $\begingroup$ Regarding to your final conclusion it should be correct $\endgroup$ – Vala Aug 28 '18 at 19:39
1
$\begingroup$

I think it can, and it does it actually very frequently (shrink some coefficient near zero). If you do an internet search for "images" and search for "ridge regression coefficient path" you will see the visual output of a large quantity of ridge regression models and their respective coefficient path output.

What you will notice is that very frequently some of the variable coefficients paths show actual coefficients flipping signs. By definition this entails that in the transition between one directional sign and the other, the respective coefficient has to cross through the zero barrier (the horizontal line in the graph defining zero). And, at such time the coefficient has to be very close to zero.

This actually leads to another major point. Ridge regression may not be nearly as robust as many people think. When you look at ridge regression coefficient paths, you will most often observe the most influential coefficients (with the highest standardized regression coefficients) being shrunk the most. And, that is a direct function of the ridge regression penalizing algorithm that shrinks the sum of squared regression coefficients. As mentioned, even worse is when Ridge regression causes a shift in the directional sign of your variable coefficients. Thus, you may be starting with a model that has much explanatory power supported by logic and economic theory. And, after you regularize it using Ridge regression (or Lasso and Elastic-net), you have a model that has nothing left from your original explanatory power, logic, etc. That is just something to watch out for with all regularization models.

$\endgroup$
0
$\begingroup$

Traditional OLS ridge regression indeed will only return zero coefficients when a traditional OLS regression would also give you zero coefficients - the shrinkage for the other coefficients can approach zero but never actually become equal to zero. Likewise, if you use nonnegative least square ridge regression coefficients will also only become zero when they would be zero in a regular nonnegative least square (nnls) fit.

When adaptive penalty weights are used in ridge regression and such adaptive ridge regression is iterated things become a bit different though - in that case, some coefficients would asymptotically (after many iterations) tend to zero (or end up being smaller than the machine precision and be rounded off to zero) and by choosing appropriate adaptive weights you can use this scheme to approximate L0-pseudonorm penalized regression (which in combination with a well tuned lambda would approximate best subset selection). So in that case, iterative adaptive ridge regression could be used for variable selection, and this in fact works very well. You can find the details in this talk and in this paper. Basically, you just have to choose your lambdas with which you ridge penalize your p variables as lambda * penweights where penweights is a vector given by 1/(betahat^2+delta^2), where delta is a small number, e.g. 1E-5, and betahat are the estimated coefficients in the previous adaptive ridge iteration (in the first iteration they can be set to 1). Especially in combination with nonnegativity constraints (which can be imposed by fitting your ridge regression with nnls and row augmenting your covariate matrix with a matrix with the square root of your lambdas along the diagonal, i.e. as nnls(A=rbind(X, sqrt(lambdas) * diag(1, p)), b=c(y, rep(0,p)) )$x, this can converge quite fast, often converging to the L0-norm penalized solution in 10 or 20 iterations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.