I remember having read somewhere on the web a connection between ridge regression (with $\ell_2$ regularization) and PCA regression: while using $\ell_2$-regularized regression with hyperparameter $\lambda$, if $\lambda \to 0$, then the regression is equivalent to removing the PC variable with the smallest eigenvalue.

  • Why is this true?
  • Does this have anything to do with the optimization procedure? Naively, I would have expected it to be equivalent to OLS.
  • Does anybody have a reference for this?
  • 1
    $\begingroup$ Could you explain more explicitly how PCA and regression are connected in your statement? Regression distinguishes dependent from independent variables, whereas nothing of the sort occurs in PCA. So what variables are you applying PCA to? It cannot be just the independent variables, for that would have little to do with the regression. But if it's applied to all the variables, then the eigenvectors are linear combinations of them all. What could it possibly mean to remove any such component from the dataset, since it involves the dependent variable? $\endgroup$ – whuber Jan 6 '14 at 16:29
  • 1
    $\begingroup$ The connection (as I understand), is that if you use a very very small regularization penalty, an L2-regularized regression would be removing the variable that has the smallest eigenvalue. Therefore, doing SVD on the design matrix, and removing the variable with the smallest eigenvalue is equivalent to a regression with a "soft" regularization penalty... This is the closest explanation I've found to this: sites.stat.psu.edu/~jiali/course/stat597e/notes2/lreg.pdf $\endgroup$ – Jose G Jan 6 '14 at 16:39
  • 3
    $\begingroup$ Your reference appears to demonstrate the opposite of what you are saying in your comments: for small $\lambda$, there is very little change in the results. Nothing is removed at all. In fact, several slides seem aimed at pointing out the difference between $L^2$ penalized regression (in which estimates are shrunk towards $0$) and "PCA regression" (in which the smallest components are entirely removed--which can be a very bad thing in some circumstances). $\endgroup$ – whuber Jan 6 '14 at 16:59
  • 2
    $\begingroup$ Mmm.. found another reference: statweb.stanford.edu/~owen/courses/305/Rudyregularization.pdf In the slide, "$y^{ridge}$ and principal components", it says that ridge regression projects y onto these components with large dj * sigh * $\endgroup$ – Jose G Jan 6 '14 at 17:11
  • 3
    $\begingroup$ Did you notice that p. 14 of that latest reference explicitly answers your question? $\endgroup$ – whuber Jan 6 '14 at 17:22

Let $\mathbf X$ be the centered $n \times p$ predictor matrix and consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a diagonal matrix with diagonal elements $s_i$.

The fitted values of ordinary least squares (OLS) regression are given by $$\hat {\mathbf y}_\mathrm{OLS} = \mathbf X \beta_\mathrm{OLS} = \mathbf X (\mathbf X^\top \mathbf X)^{-1} \mathbf X^\top \mathbf y = \mathbf U \mathbf U^\top \mathbf y.$$ The fitted values of the ridge regression are given by $$\hat {\mathbf y}_\mathrm{ridge} = \mathbf X \beta_\mathrm{ridge} = \mathbf X (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf y = \mathbf U\: \mathrm{diag}\left\{\frac{s_i^2}{s_i^2+\lambda}\right\}\mathbf U^\top \mathbf y.$$ The fitted values of the PCA regression (PCR) with $k$ components are given by $$\hat {\mathbf y}_\mathrm{PCR} = \mathbf X_\mathrm{PCA} \beta_\mathrm{PCR} = \mathbf U\: \mathrm{diag}\left\{1,\ldots, 1, 0, \ldots 0\right\}\mathbf U^\top \mathbf y,$$ where there are $k$ ones followed by zeroes.

From here we can see that:

  1. If $\lambda=0$ then $\hat {\mathbf y}_\mathrm{ridge} = \hat {\mathbf y}_\mathrm{OLS}$.

  2. If $\lambda>0$ then the larger the singular value $s_i$, the less it will be penalized in ridge regression. Small singular values ($s_i^2 \approx \lambda$ and smaller) are penalized the most.

  3. In contrast, in PCA regression, large singular values are kept intact, and the small ones (after certain number $k$) are completely removed. This would correspond to $\lambda=0$ for the first $k$ ones and $\lambda=\infty$ for the rest.

  4. This means that ridge regression can be seen as a "smooth version" of PCR.

    (This intuition is useful but does not always hold; e.g. if all $s_i$ are approximately equal, then ridge regression will only be able to penalize all principal components of $\mathbf X$ approximately equally and can strongly differ from PCR).

  5. Ridge regression tends to perform better in practice (e.g. to have higher cross-validated performance).

  6. Answering now your question specifically: if $\lambda \to 0$, then $\hat {\mathbf y}_\mathrm{ridge} \to \hat {\mathbf y}_\mathrm{OLS}$. I don't see how it can correspond to removing the smallest $s_i$. I think this is wrong.

One good reference is The Elements of Statistical Learning, Section 3.4.1 "Ridge regression".

See also this thread: Interpretation of ridge regularization in regression and in particular the answer by @BrianBorchers.

| cite | improve this answer | |
  • $\begingroup$ Would it ever make sense to soft-threshold the singular values, max( $s_i -$ thresh, 0 ) ? (Lasso regression soft-thresholds $\beta_{Least-squares}$, not the spectrum.) $\endgroup$ – denis Jun 13 '16 at 9:01
  • $\begingroup$ One correction to your otherwise great answer: the fitted values in the regression on the first $k$ PC's are actually $$ \mathbf{U} {\text{diag}}(1_1,1_2,...,1_k,0,...,0)\mathbf{U}^T\mathbf{y} $$ This is an exercise at the end of the chapter that you mention. $\endgroup$ – Matthias Schmidtblaicher Oct 21 '16 at 10:42
  • $\begingroup$ This is beautiful. $\endgroup$ – xxx222 Nov 17 '17 at 1:06

Elements of Statistical Learning has a great discussion on this connection.

The way I interpreted this connection and logic is as follows:

  • PCA is a Linear Combination of the Feature Variables, attempting to maximize the variance of the data explained by the new space.
  • Data that suffers from multicollinearity (or more predictors than rows of data) leads to a Covariance Matrix that does not have full Rank.
  • With this Covariance Matrix, we cannot invert to determine the Least Squares solution; this causes the numerical approximation of the Least Squares Coefficients to blow up to infinity.
  • Ridge Regression introduces the penalty Lambda on the Covariance Matrix to allow for matrix inversion and convergence of the LS Coefficients.

The PCA connection is that Ridge Regression is calculating the Linear Combinations of the Features to determine where the multicollinearity is occurring. The Linear Combinations of Features (Principle Component Analysis) with the smallest variance (and hence smaller singular values and smaller eigenvalues in PCA) are the ones penalized the hardest.

Think of it this way; for the Linear Combinations of Features with smallest variance, we have found the Features that are most alike, hence causing the multicollinearity. Since Ridge does not reduce the Feature set, whichever direction this Linear Combination is describing, the original Feature corresponding to that direction is penalized the most.

| cite | improve this answer | |

Consider the linear equation $$ \mathbf X \beta = \mathbf y\,, $$ and the SVD of $\mathbf X$, $$ \mathbf X = \mathbf U \,\mathbf S \,\mathbf V^T, $$ where $\mathbf S = \text{diag}(s_i)$ is the diagonal matrix of singular values.

Ordinary least squares determines the parameter vector $\beta$ as $$ \beta_{OLS} = \mathbf V \,\mathbf S^{-1} \,\mathbf U^T $$ However, this approach fails as soon there is one singular value which is zero (as then the inverse does not exists). Moreover, even if no $s_i$ is excatly zero, numerically small singular values can render the matrix ill-conditioned and lead to a solution which is highly susceptible to errors.

Ridge regression and PCA present two methods to avoid these problems. Ridge regression replaces $\mathbf S^{-1}$ in the above equation for $\beta$ by \begin{align} \mathbf S^{-1}_{\text{ridge}} &= \text{diag}\bigg(\frac{s_i}{s^2_i+\alpha}\bigg),\\ \beta_{\text{ridge}} &= \ \mathbf V \,\mathbf S_{\text{ridge}}^{-1} \,\mathbf U^T \end{align}

PCA replaces $\mathbf S^{-1}$ by \begin{align} \mathbf S^{-1}_{\text{PCA}} &= \text{diag}\bigg(\frac{1}{s_i} \, \theta(s_i-\gamma)\bigg)\,,\\ \beta_{\text{PCA}} &= \ \mathbf V \,\mathbf S_{\text{PCA}}^{-1} \,\mathbf U^T \end{align} wehre $\theta$ is the step function, and $\gamma$ is the threshold parameter.

Both methods thus weaken the impact of subspaces corresponding to small values. PCA does that in a hard way, while the ridge is a smoother approach.

More abstractly, feel free to come up with your own regularization scheme $$ \mathbf S^{-1}_{\text{myReg}} = \text{diag}\big(R(s_i)\big)\,, $$ where $R(x)$ is a function that should approach zero for $x\rightarrow 0$ and $R(x)\rightarrow x^{-1}$ for $x$ large. But remember, there's no free lunch.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.