239 votes
Accepted

How to reverse PCA and reconstruct original variables from several principal components?

PCA computes eigenvectors of the covariance matrix ("principal axes") and sorts them by their eigenvalues (amount of explained variance). The centered data can then be projected onto these principal ...
amoeba's user avatar
  • 105k
47 votes
Accepted

The proof of shrinking coefficients using ridge regression through "spectral decomposition"

The question appears to ask for a demonstration that Ridge Regression shrinks coefficient estimates towards zero, using a spectral decomposition. The spectral decomposition can be understood as an ...
whuber's user avatar
  • 323k
42 votes

Relationship between SVD and PCA. How to use SVD to perform PCA?

I wrote a Python & Numpy snippet that accompanies @amoeba's answer and I leave it here in case it is useful for someone. The comments are mostly taken from @amoeba's answer. ...
33 votes

Relationship between SVD and PCA. How to use SVD to perform PCA?

Let me start with PCA. Suppose that you have $n$ data points comprised of $d$ numbers (or dimensions) each. If you center this data (subtract the mean data point $\mu$ from each data vector $x_i$) you ...
Andre P's user avatar
  • 511
28 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

amoeba already gave a good answer in the comments, but if you want a formal argument, here it goes. The singular value decomposition of a matrix $A$ is $A=U\Sigma V^T$, where the columns of $V$ are ...
cangrejo's user avatar
  • 2,251
25 votes
Accepted

Difference between scikit-learn implementations of PCA and TruncatedSVD

PCA and TruncatedSVD scikit-learn implementations seem to be exactly the same algorithm. No: PCA is (truncated) SVD on centered data (by per-feature mean substraction). If the data is already ...
ogrisel's user avatar
  • 3,769
19 votes

What is the intuition behind SVD?

Let $A$ be a real $m \times n$ matrix. I'll assume that $m \geq n$ for simplicity. It's natural to ask in which direction $v$ does $A$ have the most impact (or the most explosiveness, or the most ...
littleO's user avatar
  • 590
17 votes

What are the pros and cons of applying pointwise mutual information on a word cooccurrence matrix before SVD?

according to Dan Jurafsky and James H. Martin book: "It turns out, however, that simple frequency isn’t the best measure of association between words. One problem is that raw frequency is very skewed ...
Maryam Hnr's user avatar
17 votes
Accepted

What does it mean when PCA does not produce a reduction in dimensionality?

Results suggest that your features are mutually orthogonal. Accounting for total variance means accounting for both variance and covariance. Orthogonality limits covariance. Standardization equates ...
Ed Rigdon's user avatar
  • 1,514
15 votes
Accepted

Why is non-negativity important for collaborative filtering/recommender systems?

I am not a specialist in recommender systems, but as far I understand, the premise of this question is wrong. Non-negativity is not that important for collaborative filtering. The Netflix prize was ...
amoeba's user avatar
  • 105k
15 votes
Accepted

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

Analysis of the Problem The SVD of a matrix is never unique. Let matrix $A$ have dimensions $n\times k$ and let its SVD be $$A = U D V^\prime$$ for an $n\times p$ matrix $U$ with orthonormal ...
whuber's user avatar
  • 323k
15 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

@amoeba had excellent answers to PCA questions, including this one on relation of SVD to PCA. Answering to your exact question I'll make three points: mathematically there is no difference whether ...
Aksakal's user avatar
  • 61.4k
14 votes
Accepted

Cholesky versus eigendecomposition for drawing samples from a multivariate normal distribution

The problem was studied by by Straka et.al for the Unscented Kalman Filter which draws (deterministic) samples from a multivariate Normal distribution as part of the algorithm. With some luck, the ...
Damien's user avatar
  • 695
13 votes

What is the intuition behind SVD?

Take an hour of your day and watch this lecture. This guy is super straight-forward; It's important not to skip any of it because it all comes together in the end. Even if it might seem a little slow ...
Tim Johnsen's user avatar
11 votes

Contrasting covariance calculation using R, Matlab, Pandas, NumPy cov, NumPy linalg.svd

Note that numpy.cov() considers its input data matrix to have observations in each column, and variables in each row, so to get ...
grand_chat's user avatar
  • 2,842
10 votes
Accepted

Why does the reconstruction error of truncated SVD equal the sum of squared singular values?

Let $$X = U\Sigma V^\prime$$ be the SVD of the $n\times r$ matrix $X$. Let $||\quad ||$ be any matrix norm that is left- and right-invariant under orthogonal transformations (reflections and ...
whuber's user avatar
  • 323k
10 votes
Accepted

Why do we need the regularization term for NMF but not for SVD?

NMF does not always include regularization -- for example, see the first two cost functions here. But, regularized NMF can be useful: If you're willing to add extra constraints beyond nonnegativity, ...
eric_kernfeld's user avatar
10 votes
Accepted

Explaining dimensionality reduction using SVD (without reference to PCA)

The SVD can be linked to dimensionality reduction from the standpoint of low rank matrix approximation. SVD for low rank matrix approximation Suppose we have a matrix $X$ and want to approximate it ...
user20160's user avatar
  • 32.5k
9 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

For Python users, I'd like to point out that for symmetric matrices (like the covariance matrix), it is better to use numpy.linalg.eigh function instead of a ...
Mosalx's user avatar
  • 91
9 votes
Accepted

Why is my LDA performance a non-monotonic function of the amount of training data?

You discovered an interesting phenomenon. LDA computations rely on inverting within-class scatter matrix $\mathbf S_W$. Usually LDA solution is presented as an eigenvalue decomposition of $\mathbf ...
amoeba's user avatar
  • 105k
9 votes
Accepted

Whitening/Decorrelation - why does it work?

Whitening is par for the course in computer vision applications, and may help a variety of machine learning algorithms converge to an optimal solution, beyond SVMs. (More on that towards the end of my ...
A. G.'s user avatar
  • 2,151
9 votes

How does eigenvalues measure variance along the principal components in PCA?

We start from data covariance matrix $$ S = \mathbb E(XX^{T})- \mathbb E(X) \mathbb E(X)^{T}$$ Say $\mu$ is a column vector of the same dimension of $X$ and $\mu^{T}\mu = 1$, then $$\mu^{T}S\mu=\mu^{...
meTchaikovsky's user avatar
9 votes

SVD : Why right singular matrix is written as transpose

It's written as a transpose for linear algebraic reasons. Consider the trivial rank-one case $A = uv^T$, where $u$ and $v$ are, say, unit vectors. This expression tells you that, as a linear ...
8 votes
Accepted

How to perform SVD to impute missing values, a concrete example

SVD is only defined for complete matrices. So if you stick to plain SVD you need to fill in these missing values before (SVD is not a imputing-algorithm per se). The errors you introduce will ...
sascha's user avatar
  • 344
8 votes
Accepted

Why do prcomp() and eigen(cov()) in R return different signs of PCA eigenvectors?

Looking at the code, stats:::prcomp.default uses singular value decomposition at its core, rather than eigendecomposition of the variance-covariance matrix. In ...
Ben Bolker's user avatar
  • 43.7k
8 votes
Accepted

Why is computing ridge regression with a Cholesky decomposition much quicker than using SVD?

First, we note that ridge regression can be transformed into an OLS problem via the data augmentation trick.. This adds $p$ rows to the design matrix $\mathbf{X}$, so the new matrix $\mathbf{X}_\...
olooney's user avatar
  • 3,210
8 votes

What does it mean when PCA does not produce a reduction in dimensionality?

PCA can be used to scale and rotate data, if we select all transformed features instead of a subset of them. My answer here gives an example of scaling and rotating the data, but without dimension ...
Haitao Du's user avatar
  • 36.9k
7 votes
Accepted

Confirming an understanding of SVD

I have tried to dispel some of the popular myths regarding the SVD. ``A common application of SVD is to make low-rank approximations to a matrix, $A$'' This is one of the applications of the SVD ...
Vini's user avatar
  • 266
7 votes

Do the principal components change if we apply PCA more than once (recursively) on data?

PCA at it's heart involves diagonalizing a matrix which means solving for the eigenvalues and eigenvectors of said matrix. The whole purpose of the calculation is to find a diagonal representation of ...
Greg Petersen's user avatar
7 votes
Accepted

Rotating a new matrix into the same basis as another using SVD

What you are doing is essentially PCA, even though you did not use this term. It is exactly PCA if your $X$ is centered; if not, then it is a sort of "uncentered PCA". In any case, to get from $A$ to ...
amoeba's user avatar
  • 105k

Only top scored, non community-wiki answers of a minimum length are eligible