Is there any special case where ridge regression can shrink coefficients to zero?

Question

Are there some special cases, where the Ridge Regression can also lead to coefficients that are zero ? It is widely known, that lasso is shrinking coefficients towards or on zero, while the ridge Regression cant shrink coefficients to zero

Answer 1

Suppose, as in the case of least squares methods, you are trying to solve a statistical estimation problem for a (vector-valued) parameter $\beta$ by minimizing an objective function $Q(\beta)$ (such as the sum of squares of the residuals). Ridge Regression "regularizes" the problem by adding a non-negative linear combination of the squares of the parameter, $P(\beta).$ $P$ is (obviously) differentiable with a unique global minimum at $\beta=0.$

The question asks, when is it possible for the global minimum of $Q+P$ to occur at $\beta=0$? Assume, as in least squares methods, that $Q$ is differentiable in a neighborhood of $0.$ Because $0$ is a global minimum for $Q+P$ it is a local minimum, implying all its partial derivatives are $0.$ The sum rule of differentiation implies

$$\frac{\partial}{\partial \beta_i}(Q(\beta) + P(\beta)) = \frac{\partial}{\partial \beta_i}Q(\beta) + \frac{\partial}{\partial \beta_i}P(\beta) = Q_i(\beta) + P_i(\beta)$$ is zero at $\beta=0.$ But since $P_i(0)=0$ for all $i,$ this implies $Q_i(0)=0$ for all $i,$ which makes $0$ at least a local minimum for the original objective function $Q.$ In the case of any least squares technique every local minimum is also a global minimum. This compels us to conclude that

Quadratic regularization of Least Squares procedures ("Ridge Regression") has $\beta=0$ as a solution if and only if $\beta=0$ is a solution of the original unregularized problem.

Answer 2

Traditional OLS ridge regression indeed will only return zero coefficients when a traditional OLS regression would also give you zero coefficients - the shrinkage for the other coefficients can approach zero but never actually become equal to zero. Likewise, if you use nonnegative least square ridge regression coefficients will also only become zero when they would be zero in a regular nonnegative least square (nnls) fit.

When adaptive penalty weights are used in ridge regression and such adaptive ridge regression is iterated things become a bit different though - in that case, some coefficients would asymptotically (after many iterations) tend to zero (or end up being smaller than the machine precision and be rounded off to zero) and by choosing appropriate adaptive weights you can use this scheme to approximate L0-pseudonorm penalized regression (which in combination with a well tuned lambda would approximate best subset selection). So in that case, iterative adaptive ridge regression could be used for variable selection, and this in fact works very well. You can find the details in this talk and in this paper. Basically, you just have to choose your lambdas with which you ridge penalize your p variables as lambda * penweights where penweights is a vector given by 1/(betahat^2+delta^2), where delta is a small number, e.g. 1E-5, and betahat are the estimated coefficients in the previous adaptive ridge iteration (in the first iteration they can be set to 1). Especially in combination with nonnegativity constraints (which can be imposed by fitting your ridge regression with nnls and row augmenting your covariate matrix with a matrix with the square root of your lambdas along the diagonal, i.e. as nnls(A=rbind(X, sqrt(lambdas) * diag(1, p)), b=c(y, rep(0,p)) )$x, this can converge quite fast, often converging to the L0-norm penalized solution in 10 or 20 iterations.

Answer 3

I think it can, and it does it actually very frequently (shrink some coefficient near zero). If you do an internet search for "images" and search for "ridge regression coefficient path" you will see the visual output of a large quantity of ridge regression models and their respective coefficient path output.

What you will notice is that very frequently some of the variable coefficients paths show actual coefficients flipping signs. By definition this entails that in the transition between one directional sign and the other, the respective coefficient has to cross through the zero barrier (the horizontal line in the graph defining zero). And, at such time the coefficient has to be very close to zero.

This actually leads to another major point. Ridge regression may not be nearly as robust as many people think. When you look at ridge regression coefficient paths, you will most often observe the most influential coefficients (with the highest standardized regression coefficients) being shrunk the most. And, that is a direct function of the ridge regression penalizing algorithm that shrinks the sum of squared regression coefficients. As mentioned, even worse is when Ridge regression causes a shift in the directional sign of your variable coefficients. Thus, you may be starting with a model that has much explanatory power supported by logic and economic theory. And, after you regularize it using Ridge regression (or Lasso and Elastic-net), you have a model that has nothing left from your original explanatory power, logic, etc. That is just something to watch out for with all regularization models.