24
$\begingroup$

I remember having read somewhere on the web a connection between ridge regression (with $\ell_2$ regularization) and PCA regression: while using $\ell_2$-regularized regression with hyperparameter $\lambda$, if $\lambda \to 0$, then the regression is equivalent to removing the PC variable with the smallest eigenvalue.

  • Why is this true?
  • Does this have anything to do with the optimization procedure? Naively, I would have expected it to be equivalent to OLS.
  • Does anybody have a reference for this?
$\endgroup$
7
  • 1
    $\begingroup$ Could you explain more explicitly how PCA and regression are connected in your statement? Regression distinguishes dependent from independent variables, whereas nothing of the sort occurs in PCA. So what variables are you applying PCA to? It cannot be just the independent variables, for that would have little to do with the regression. But if it's applied to all the variables, then the eigenvectors are linear combinations of them all. What could it possibly mean to remove any such component from the dataset, since it involves the dependent variable? $\endgroup$
    – whuber
    Commented Jan 6, 2014 at 16:29
  • 2
    $\begingroup$ The connection (as I understand), is that if you use a very very small regularization penalty, an L2-regularized regression would be removing the variable that has the smallest eigenvalue. Therefore, doing SVD on the design matrix, and removing the variable with the smallest eigenvalue is equivalent to a regression with a "soft" regularization penalty... This is the closest explanation I've found to this: sites.stat.psu.edu/~jiali/course/stat597e/notes2/lreg.pdf $\endgroup$
    – Jose G
    Commented Jan 6, 2014 at 16:39
  • 3
    $\begingroup$ Your reference appears to demonstrate the opposite of what you are saying in your comments: for small $\lambda$, there is very little change in the results. Nothing is removed at all. In fact, several slides seem aimed at pointing out the difference between $L^2$ penalized regression (in which estimates are shrunk towards $0$) and "PCA regression" (in which the smallest components are entirely removed--which can be a very bad thing in some circumstances). $\endgroup$
    – whuber
    Commented Jan 6, 2014 at 16:59
  • 2
    $\begingroup$ Mmm.. found another reference: statweb.stanford.edu/~owen/courses/305/Rudyregularization.pdf In the slide, "$y^{ridge}$ and principal components", it says that ridge regression projects y onto these components with large dj * sigh * $\endgroup$
    – Jose G
    Commented Jan 6, 2014 at 17:11
  • 4
    $\begingroup$ Did you notice that p. 14 of that latest reference explicitly answers your question? $\endgroup$
    – whuber
    Commented Jan 6, 2014 at 17:22

3 Answers 3

32
$\begingroup$

Let $\mathbf X$ be the centered $n \times p$ predictor matrix and consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a diagonal matrix with diagonal elements $s_i$.

The fitted values of ordinary least squares (OLS) regression are given by $$\hat {\mathbf y}_\mathrm{OLS} = \mathbf X \beta_\mathrm{OLS} = \mathbf X (\mathbf X^\top \mathbf X)^{-1} \mathbf X^\top \mathbf y = \mathbf U \mathbf U^\top \mathbf y.$$ The fitted values of the ridge regression are given by $$\hat {\mathbf y}_\mathrm{ridge} = \mathbf X \beta_\mathrm{ridge} = \mathbf X (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf y = \mathbf U\: \mathrm{diag}\left\{\frac{s_i^2}{s_i^2+\lambda}\right\}\mathbf U^\top \mathbf y.$$ The fitted values of the PCA regression (PCR) with $k$ components are given by $$\hat {\mathbf y}_\mathrm{PCR} = \mathbf X_\mathrm{PCA} \beta_\mathrm{PCR} = \mathbf U\: \mathrm{diag}\left\{1,\ldots, 1, 0, \ldots 0\right\}\mathbf U^\top \mathbf y,$$ where there are $k$ ones followed by zeroes.

From here we can see that:

  1. If $\lambda=0$ then $\hat {\mathbf y}_\mathrm{ridge} = \hat {\mathbf y}_\mathrm{OLS}$.

  2. If $\lambda>0$ then the larger the singular value $s_i$, the less it will be penalized in ridge regression. Small singular values ($s_i^2 \approx \lambda$ and smaller) are penalized the most.

  3. In contrast, in PCA regression, large singular values are kept intact, and the small ones (after certain number $k$) are completely removed. This would correspond to $\lambda=0$ for the first $k$ ones and $\lambda=\infty$ for the rest.

  4. This means that ridge regression can be seen as a "smooth version" of PCR.

    (This intuition is useful but does not always hold; e.g. if all $s_i$ are approximately equal, then ridge regression will only be able to penalize all principal components of $\mathbf X$ approximately equally and can strongly differ from PCR).

  5. Ridge regression tends to perform better in practice (e.g. to have higher cross-validated performance).

  6. Answering now your question specifically: if $\lambda \to 0$, then $\hat {\mathbf y}_\mathrm{ridge} \to \hat {\mathbf y}_\mathrm{OLS}$. I don't see how it can correspond to removing the smallest $s_i$. I think this is wrong.

One good reference is The Elements of Statistical Learning, Section 3.4.1 "Ridge regression".


See also this thread: Interpretation of ridge regularization in regression and in particular the answer by @BrianBorchers.

$\endgroup$
4
  • $\begingroup$ Would it ever make sense to soft-threshold the singular values, max( $s_i -$ thresh, 0 ) ? (Lasso regression soft-thresholds $\beta_{Least-squares}$, not the spectrum.) $\endgroup$
    – denis
    Commented Jun 13, 2016 at 9:01
  • $\begingroup$ One correction to your otherwise great answer: the fitted values in the regression on the first $k$ PC's are actually $$ \mathbf{U} {\text{diag}}(1_1,1_2,...,1_k,0,...,0)\mathbf{U}^T\mathbf{y} $$ This is an exercise at the end of the chapter that you mention. $\endgroup$ Commented Oct 21, 2016 at 10:42
  • $\begingroup$ This is beautiful. $\endgroup$
    – xxx222
    Commented Nov 17, 2017 at 1:06
  • $\begingroup$ I don't think using singular value as the criterion for component removal is a good idea in the context of regression. Instead, you should apply regression with all principal components first then remove the components that don't contribute much to the fitting. See also: Examples of PCA where PCs with low variance are "useful" $\endgroup$
    – syockit
    Commented Aug 25, 2023 at 11:34
6
$\begingroup$

Elements of Statistical Learning has a great discussion on this connection.

The way I interpreted this connection and logic is as follows:

  • PCA is a Linear Combination of the Feature Variables, attempting to maximize the variance of the data explained by the new space.
  • Data that suffers from multicollinearity (or more predictors than rows of data) leads to a Covariance Matrix that does not have full Rank.
  • With this Covariance Matrix, we cannot invert to determine the Least Squares solution; this causes the numerical approximation of the Least Squares Coefficients to blow up to infinity.
  • Ridge Regression introduces the penalty Lambda on the Covariance Matrix to allow for matrix inversion and convergence of the LS Coefficients.

The PCA connection is that Ridge Regression is calculating the Linear Combinations of the Features to determine where the multicollinearity is occurring. The Linear Combinations of Features (Principle Component Analysis) with the smallest variance (and hence smaller singular values and smaller eigenvalues in PCA) are the ones penalized the hardest.

Think of it this way; for the Linear Combinations of Features with smallest variance, we have found the Features that are most alike, hence causing the multicollinearity. Since Ridge does not reduce the Feature set, whichever direction this Linear Combination is describing, the original Feature corresponding to that direction is penalized the most.

$\endgroup$
4
$\begingroup$

Consider the linear equation $$ \mathbf X \beta = \mathbf y\,, $$ and the SVD of $\mathbf X$, $$ \mathbf X = \mathbf U \,\mathbf S \,\mathbf V^T, $$ where $\mathbf S = \text{diag}(s_i)$ is the diagonal matrix of singular values.

Ordinary least squares determines the parameter vector $\beta$ as $$ \beta_{OLS} = \mathbf V \,\mathbf S^{-1} \,\mathbf U^T \, \mathbf y $$ However, this approach fails as soon there is one singular value which is zero (as then the inverse does not exists). Moreover, even if no $s_i$ is excatly zero, numerically small singular values can render the matrix ill-conditioned and lead to a solution which is highly susceptible to errors.

Ridge regression and PCA present two methods to avoid these problems. Ridge regression replaces $\mathbf S^{-1}$ in the above equation for $\beta$ by \begin{align} \mathbf S^{-1}_{\text{ridge}} &= \text{diag}\bigg(\frac{s_i}{s^2_i+\alpha}\bigg),\\ \beta_{\text{ridge}} &= \ \mathbf V \,\mathbf S_{\text{ridge}}^{-1} \,\mathbf U^T \, \mathbf y \end{align}

PCA replaces $\mathbf S^{-1}$ by \begin{align} \mathbf S^{-1}_{\text{PCA}} &= \text{diag}\bigg(\frac{1}{s_i} \, \theta(s_i-\gamma)\bigg)\,,\\ \beta_{\text{PCA}} &= \ \mathbf V \,\mathbf S_{\text{PCA}}^{-1} \,\mathbf U^T \, \mathbf y \end{align} wehre $\theta$ is the step function, and $\gamma$ is the threshold parameter.

Both methods thus weaken the impact of subspaces corresponding to small singular values. PCA does that in a hard way, while the ridge is a smoother approach.

More abstractly, feel free to come up with your own regularization scheme $$ \mathbf S^{-1}_{\text{myReg}} = \text{diag}\big(R(s_i)\big)\,, $$ where $R(x)$ is a function that should approach zero for $x\rightarrow 0$ and $R(x)\rightarrow x^{-1}$ for $x$ large. But remember, there's no free lunch.

$\endgroup$
3
  • 1
    $\begingroup$ Where is $y$ in your estimators ? Isn't it $\beta_{ols} = \mathbf{VS^{-1}V}^\intercal y$? $\endgroup$
    – Cryckx
    Commented Jun 15, 2022 at 10:33
  • $\begingroup$ Also, could you explain how you found $\mathbf{S}^{-1}_{PCA}$? $\endgroup$
    – Cryckx
    Commented Jun 15, 2022 at 10:39
  • $\begingroup$ @Cryckx: sure, in all betas the right-hand side is missing. I'll update the question. The PCA stuff is rarely seen in this notation -- usually the recipe goes as "drop all entries of the pseudoinverse that correspond to singular values below a chosen threshold". The heaveside step function is just a mathematical notation for that. $\endgroup$
    – davidhigh
    Commented Jun 15, 2022 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.