Skip to main content
43 votes
Accepted

If I generate a random symmetric matrix, what's the chance it is positive definite?

If your matrices are drawn from standard-normal iid entries, the probability of being positive-definite is approximately $p_N\approx 3^{-N^2/4}$, so for example if $N=5$, the chance is 1/1000, and ...
Alex R.'s user avatar
  • 14k
28 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

amoeba already gave a good answer in the comments, but if you want a formal argument, here it goes. The singular value decomposition of a matrix $A$ is $A=U\Sigma V^T$, where the columns of $V$ are ...
cangrejo's user avatar
  • 2,251
23 votes
Accepted

Why eigenvectors reveal the groups in Spectral Clustering

This is a great, and a subtle question. Before we turn to your algorithm, let us first observe the similarity matrix $S$. It is symmetrical and, if your data form convex clusters (see below), and with ...
Igor F.'s user avatar
  • 9,418
15 votes
Accepted

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

Analysis of the Problem The SVD of a matrix is never unique. Let matrix $A$ have dimensions $n\times k$ and let its SVD be $$A = U D V^\prime$$ for an $n\times p$ matrix $U$ with orthonormal ...
whuber's user avatar
  • 328k
15 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

@amoeba had excellent answers to PCA questions, including this one on relation of SVD to PCA. Answering to your exact question I'll make three points: mathematically there is no difference whether ...
Aksakal's user avatar
  • 61.8k
14 votes
Accepted

Quadratic form and Chi-squared distribution

In general, the quadratic form is a weighted sum of $\chi_1^2$ It is not true in general that $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi^2_p$ for any symmetric positive-definite (...
Ben's user avatar
  • 129k
12 votes
Accepted

Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

I will try to explain how orthogonality of $a_1$ and $a_2$ ensures that $y_1$ and $y_2$ be uncorrelated. We want $a_1$ to maximize $Var(y_1)=a_1^T \Sigma a_1$. This will not be achieved unless we ...
Bananin's user avatar
  • 728
10 votes

Making sense of principal component analysis, eigenvectors & eigenvalues

I think that everyone starts explaining PCA from the wrong end: from eigenvectors. My answer starts at the right place: coordinate system. Eigenvectors, and eigenproblem in general, are the ...
Aksakal's user avatar
  • 61.8k
10 votes
Accepted

In PCA, why do we assume that the covariance matrix is always diagonalizable?

Covariance matrix is a symmetric matrix, hence it is always diagonalizable. In fact, in the diagonalization, $C=PDP^{-1}$, we know that we can choose $P$ to be an orthogonal matrix. It belongs to a ...
Siong Thye Goh's user avatar
10 votes
Accepted

Get accurate eigenvectors, when eigenvalues are minuscule

The problem is due to "leakage" from the large eigenvectors. I will present a brief analysis and then offer a solution, with code, followed by some remarks about the nature and limitations ...
whuber's user avatar
  • 328k
10 votes

Eigenvalues/Eigenvectors of Correlation and Covariance matrices

Expanding on my comment: Since $P = \text{diag}(\Sigma)^{-1/2} \Sigma \text{diag}(\Sigma)^{-1/2}$, where $\text{diag}(\Sigma)$ is the diagonal matrix obtained by considering only the diagonal entries ...
mhdadk's user avatar
  • 5,110
9 votes

Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$?

Let's say we have a matrix $X=[x_1, x_2, \cdots, x_n]$ , where each $x_i$ is an obervation (sample) from $d$ dimension space, so $X$ is a $d$ by $n$ matrix, and $d > n$. If we first centered the ...
Here's user avatar
  • 91
9 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

For Python users, I'd like to point out that for symmetric matrices (like the covariance matrix), it is better to use numpy.linalg.eigh function instead of a ...
Mosalx's user avatar
  • 91
9 votes

How does eigenvalues measure variance along the principal components in PCA?

We start from data covariance matrix $$ S = \mathbb E(XX^{T})- \mathbb E(X) \mathbb E(X)^{T}$$ Say $\mu$ is a column vector of the same dimension of $X$ and $\mu^{T}\mu = 1$, then $$\mu^{T}S\mu=\mu^{...
meTchaikovsky's user avatar
9 votes
Accepted

Expectation of a function of the sample covariance matrix

This is a quite challenging and interesting problem as it calls for many classical results of multivariate Gaussian distribution and technical matrix operations. To begin with, note that since $\hat{\...
Zhanxiong's user avatar
  • 20.7k
8 votes
Accepted

Why do prcomp() and eigen(cov()) in R return different signs of PCA eigenvectors?

Looking at the code, stats:::prcomp.default uses singular value decomposition at its core, rather than eigendecomposition of the variance-covariance matrix. In ...
Ben Bolker's user avatar
  • 44.5k
8 votes

Why does the first eigenvector in PCA resemble the derivative of an underlying trend?

Let's ignore the mean-centering for a moment. One way to understand the data is to view each time series as being approximately a fixed multiple of an overall "trend," which itself is a time series $...
whuber's user avatar
  • 328k
8 votes
Accepted

Why is $\mathbf\Phi^{\top}\mathbf\Phi$ a positive definite matrix?

Note we have $$ v^\top \left(\Phi^\top\Phi\right) v = \left(v^\top \Phi^\top\right) \left(\Phi v\right) = \left(\Phi v \right)^\top \left(\Phi v\right) = \|\Phi v \|_2^2 \geq 0 $$ for all $\Phi \in \...
statmerkur's user avatar
  • 6,325
8 votes

Why is $\mathbf\Phi^{\top}\mathbf\Phi$ a positive definite matrix?

Observation $1.$ A symmetric matrix $\mathbf A^\mathsf T\mathbf A$ is positive semi-definite. Note that $$\mathbf v^\mathsf T\mathbf A^\mathsf T\mathbf A\mathbf v= (\mathbf A\mathbf v)^\mathsf T\cdot (...
User1865345's user avatar
  • 9,427
8 votes
Accepted

Eigenvalues/Eigenvectors of Correlation and Covariance matrices

If $\Sigma$ is diagonal (with arbitrary eigenvalues) then $P$ is just the unit matrix (all eigenvalues equal to one), so there cannot be any general relation between the eigenvalues of $\Sigma$ (alone)...
J. Delaney's user avatar
  • 5,400
8 votes

Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)

The eigenvalues would really only reveal how many classes (single classifier) or how many classifiers are correlated with one another (multiple classifiers). But if you look at the quasi-diagonalized ...
wjktrs's user avatar
  • 860
7 votes

Orthogonality in PCA vectors

From a simple geometric point of view: If the second eigenvector was not orthogonal to the first, then either the first eigenvector would not account for as much variation as possible, or the second ...
Beyer's user avatar
  • 1,232
7 votes

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

As I outlined in a comment to @whuber's answer, this method to compute the SVD doesn't work for every matrix. The issue is not limited to signs. The problem is that there may be repeated eigenvalues, ...
Federico Poloni's user avatar
7 votes

Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

PCA works by computing the eigenvectors of the covariance matrix of the data. That is, those eigenvectors correspond to the choices of $a_{1:M}$ that maximize the equations and meet the constraints ...
Ruben van Bergen's user avatar
7 votes
Accepted

Lower bound on smallest eigenvalue of covariance matrices

For a symmetric matrix the one norm and the infinity norm coincide. So the condition on the norm $$\Vert \Sigma(\theta) - I_{p}\Vert_{1} = \Vert\Sigma(\theta) - I_{p}\Vert_{\infty} \leq a$$ implies ...
Luca Citi's user avatar
  • 1,366
7 votes
Accepted

If the eigenvalues of a covariance matrix have very low variance, what does it mean?

Each eigenvalue indicates the variance of the data along the direction of the corresponding eigenvector. If the data are jointly Gaussian, then the covariance matrix completely determines the shape ...
user20160's user avatar
  • 32.9k
7 votes

Adding a small constant to the diagonals of a matrix to stabilize

This is in addition to OmG's good answer (+1). Numerically, the size of the eigenvalues that is not directly a problem but rather the condition number of the matrix (i.e. the ratio between the ...
usεr11852's user avatar
  • 44.9k
7 votes
Accepted

Eigenvalues in Ridge regression

If $\lambda$ is a nonzero eigenvalue of an invertible matrix $A$, then $\lambda^{-1}$ is an eigenvalue of $A^{-1}$. To show this, note that if $Av=\lambda v$ then $v = A^{-1}(\lambda v) = \lambda A^{-...
angryavian's user avatar
  • 2,328
6 votes

Covariance matrix decomposition and coregionalization

A covariance matrix has ${n \choose 2} + n = \frac{n(n+1)}{2}$ free elements. The constraints for the spectral decomposition are: The eigenvalues are positive The eigenvectors are orthogonal The ...
Taylor's user avatar
  • 21.2k
6 votes
Accepted

How is PCA applied to new data?

I will answer each question: $A$ is indeed the covariance matrix (so $X^TX$ assuming $X$ is standardized) The output of PCA is 3 things: the vector of column means $\mu$ of $X$, the vector of column ...
Felipe Gerard's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible