62 votes
Accepted

How does centering make a difference in PCA (for SVD and eigen decomposition)?

As you remarked yourself and as explained by @ttnphns in the comments, computing covariance matrix implicitly performs centering: variance, by definition, is the average squared deviation from the ...
amoeba's user avatar
  • 104k
43 votes
Accepted

If I generate a random symmetric matrix, what's the chance it is positive definite?

If your matrices are drawn from standard-normal iid entries, the probability of being positive-definite is approximately $p_N\approx 3^{-N^2/4}$, so for example if $N=5$, the chance is 1/1000, and ...
Alex R.'s user avatar
  • 13.9k
31 votes
Accepted

What is principal subspace in probabilistic PCA?

This is an excellent question. Probabilistic PCA (PPCA) is the following latent variable model \begin{align} \mathbf z &\sim \mathcal N(\mathbf 0, \mathbf I) \\ \mathbf x &\sim \mathcal N(\...
amoeba's user avatar
  • 104k
28 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

amoeba already gave a good answer in the comments, but if you want a formal argument, here it goes. The singular value decomposition of a matrix $A$ is $A=U\Sigma V^T$, where the columns of $V$ are ...
cangrejo's user avatar
  • 2,251
23 votes
Accepted

Why eigenvectors reveal the groups in Spectral Clustering

This is a great, and a subtle question. Before we turn to your algorithm, let us first observe the similarity matrix $S$. It is symmetrical and, if your data form convex clusters (see below), and with ...
Igor F.'s user avatar
  • 8,837
15 votes
Accepted

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

Analysis of the Problem The SVD of a matrix is never unique. Let matrix $A$ have dimensions $n\times k$ and let its SVD be $$A = U D V^\prime$$ for an $n\times p$ matrix $U$ with orthonormal ...
whuber's user avatar
  • 321k
15 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

@amoeba had excellent answers to PCA questions, including this one on relation of SVD to PCA. Answering to your exact question I'll make three points: mathematically there is no difference whether ...
Aksakal's user avatar
  • 60.9k
14 votes
Accepted

Quadratic form and Chi-squared distribution

In general, the quadratic form is a weighted sum of $\chi_1^2$ It is not true in general that $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi^2_p$ for any symmetric positive-definite (...
Ben's user avatar
  • 123k
13 votes

Does the sign of eigenvectors matter?

No, there is no difference. Notice that if $v$ is an eigenvector to $A$ with eigenvalue $\lambda$ and $\alpha$ is a scalar, then $$ A \alpha v = \alpha A v = \lambda \alpha v $$ and thus $\alpha v$ ...
Chris Rackauckas's user avatar
12 votes
Accepted

Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

I will try to explain how orthogonality of $a_1$ and $a_2$ ensures that $y_1$ and $y_2$ be uncorrelated. We want $a_1$ to maximize $Var(y_1)=a_1^T \Sigma a_1$. This will not be achieved unless we ...
Bananin's user avatar
  • 718
10 votes
Accepted

What's the relationship between initial eigenvalues and sums of squared loadings in factor analysis?

The two citations do not generally contradict each other and both look to me correct. The only underwork is in ...
ttnphns's user avatar
  • 57.2k
10 votes

Making sense of principal component analysis, eigenvectors & eigenvalues

I think that everyone starts explaining PCA from the wrong end: from eigenvectors. My answer starts at the right place: coordinate system. Eigenvectors, and eigenproblem in general, are the ...
Aksakal's user avatar
  • 60.9k
10 votes
Accepted

In PCA, why do we assume that the covariance matrix is always diagonalizable?

Covariance matrix is a symmetric matrix, hence it is always diagonalizable. In fact, in the diagonalization, $C=PDP^{-1}$, we know that we can choose $P$ to be an orthogonal matrix. It belongs to a ...
Siong Thye Goh's user avatar
10 votes
Accepted

Get accurate eigenvectors, when eigenvalues are minuscule

The problem is due to "leakage" from the large eigenvectors. I will present a brief analysis and then offer a solution, with code, followed by some remarks about the nature and limitations ...
whuber's user avatar
  • 321k
10 votes

Eigenvalues/Eigenvectors of Correlation and Covariance matrices

Expanding on my comment: Since $P = \text{diag}(\Sigma)^{-1/2} \Sigma \text{diag}(\Sigma)^{-1/2}$, where $\text{diag}(\Sigma)$ is the diagonal matrix obtained by considering only the diagonal entries ...
mhdadk's user avatar
  • 4,745
9 votes

Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$?

Let's say we have a matrix $X=[x_1, x_2, \cdots, x_n]$ , where each $x_i$ is an obervation (sample) from $d$ dimension space, so $X$ is a $d$ by $n$ matrix, and $d > n$. If we first centered the ...
Here's user avatar
  • 91
9 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

For Python users, I'd like to point out that for symmetric matrices (like the covariance matrix), it is better to use numpy.linalg.eigh function instead of a ...
Mosalx's user avatar
  • 91
9 votes

How does eigenvalues measure variance along the principal components in PCA?

We start from data covariance matrix $$ S = \mathbb E(XX^{T})- \mathbb E(X) \mathbb E(X)^{T}$$ Say $\mu$ is a column vector of the same dimension of $X$ and $\mu^{T}\mu = 1$, then $$\mu^{T}S\mu=\mu^{...
meTchaikovsky's user avatar
9 votes
Accepted

Expectation of a function of the sample covariance matrix

This is a quite challenging and interesting problem as it calls for many classical results of multivariate Gaussian distribution and technical matrix operations. To begin with, note that since $\hat{\...
Zhanxiong's user avatar
  • 17.3k
8 votes
Accepted

Why do prcomp() and eigen(cov()) in R return different signs of PCA eigenvectors?

Looking at the code, stats:::prcomp.default uses singular value decomposition at its core, rather than eigendecomposition of the variance-covariance matrix. In ...
Ben Bolker's user avatar
8 votes

Why does the first eigenvector in PCA resemble the derivative of an underlying trend?

Let's ignore the mean-centering for a moment. One way to understand the data is to view each time series as being approximately a fixed multiple of an overall "trend," which itself is a time series $...
whuber's user avatar
  • 321k
8 votes
Accepted

Why is $\mathbf\Phi^{\top}\mathbf\Phi$ a positive definite matrix?

Note we have $$ v^\top \left(\Phi^\top\Phi\right) v = \left(v^\top \Phi^\top\right) \left(\Phi v\right) = \left(\Phi v \right)^\top \left(\Phi v\right) = \|\Phi v \|_2^2 \geq 0 $$ for all $\Phi \in \...
statmerkur's user avatar
  • 5,830
8 votes

Why is $\mathbf\Phi^{\top}\mathbf\Phi$ a positive definite matrix?

Observation $1.$ A symmetric matrix $\mathbf A^\mathsf T\mathbf A$ is positive semi-definite. Note that $$\mathbf v^\mathsf T\mathbf A^\mathsf T\mathbf A\mathbf v= (\mathbf A\mathbf v)^\mathsf T\cdot (...
User1865345's user avatar
  • 7,932
8 votes
Accepted

Eigenvalues/Eigenvectors of Correlation and Covariance matrices

If $\Sigma$ is diagonal (with arbitrary eigenvalues) then $P$ is just the unit matrix (all eigenvalues equal to one), so there cannot be any general relation between the eigenvalues of $\Sigma$ (alone)...
J. Delaney's user avatar
  • 5,380
7 votes
Accepted

Prove that $XX^\top$ and $X^\top X$ have the same eigenvalues

You will be surprised by how simple the problem is if you approach it from the right angle. Start from the definitions of eigenvalues and eigenvectors. It only takes a few manipulations afterwards to ...
JohnK's user avatar
  • 20.2k
7 votes

Do the principal components change if we apply PCA more than once (recursively) on data?

PCA at it's heart involves diagonalizing a matrix which means solving for the eigenvalues and eigenvectors of said matrix. The whole purpose of the calculation is to find a diagonal representation of ...
Greg Petersen's user avatar
7 votes

Orthogonality in PCA vectors

From a simple geometric point of view: If the second eigenvector was not orthogonal to the first, then either the first eigenvector would not account for as much variation as possible, or the second ...
Beyer's user avatar
  • 1,222
7 votes

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

As I outlined in a comment to @whuber's answer, this method to compute the SVD doesn't work for every matrix. The issue is not limited to signs. The problem is that there may be repeated eigenvalues, ...
Federico Poloni's user avatar
7 votes

Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

PCA works by computing the eigenvectors of the covariance matrix of the data. That is, those eigenvectors correspond to the choices of $a_{1:M}$ that maximize the equations and meet the constraints ...
Ruben van Bergen's user avatar
7 votes
Accepted

Lower bound on smallest eigenvalue of covariance matrices

For a symmetric matrix the one norm and the infinity norm coincide. So the condition on the norm $$\Vert \Sigma(\theta) - I_{p}\Vert_{1} = \Vert\Sigma(\theta) - I_{p}\Vert_{\infty} \leq a$$ implies ...
Luca Citi's user avatar
  • 1,314

Only top scored, non community-wiki answers of a minimum length are eligible