# Tag Info

Accepted

### If I generate a random symmetric matrix, what's the chance it is positive definite?

If your matrices are drawn from standard-normal iid entries, the probability of being positive-definite is approximately $p_N\approx 3^{-N^2/4}$, so for example if $N=5$, the chance is 1/1000, and ...
• 14k

### Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

amoeba already gave a good answer in the comments, but if you want a formal argument, here it goes. The singular value decomposition of a matrix $A$ is $A=U\Sigma V^T$, where the columns of $V$ are ...
• 2,251
Accepted

### Why eigenvectors reveal the groups in Spectral Clustering

This is a great, and a subtle question. Before we turn to your algorithm, let us first observe the similarity matrix $S$. It is symmetrical and, if your data form convex clusters (see below), and with ...
• 9,418
Accepted

### Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

Analysis of the Problem The SVD of a matrix is never unique. Let matrix $A$ have dimensions $n\times k$ and let its SVD be $$A = U D V^\prime$$ for an $n\times p$ matrix $U$ with orthonormal ...
• 328k

### Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

@amoeba had excellent answers to PCA questions, including this one on relation of SVD to PCA. Answering to your exact question I'll make three points: mathematically there is no difference whether ...
• 61.8k
Accepted

### Quadratic form and Chi-squared distribution

In general, the quadratic form is a weighted sum of $\chi_1^2$ It is not true in general that $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi^2_p$ for any symmetric positive-definite (...
• 129k
Accepted

### Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

I will try to explain how orthogonality of $a_1$ and $a_2$ ensures that $y_1$ and $y_2$ be uncorrelated. We want $a_1$ to maximize $Var(y_1)=a_1^T \Sigma a_1$. This will not be achieved unless we ...
• 728

### Making sense of principal component analysis, eigenvectors & eigenvalues

I think that everyone starts explaining PCA from the wrong end: from eigenvectors. My answer starts at the right place: coordinate system. Eigenvectors, and eigenproblem in general, are the ...
• 61.8k
Accepted

### In PCA, why do we assume that the covariance matrix is always diagonalizable?

Covariance matrix is a symmetric matrix, hence it is always diagonalizable. In fact, in the diagonalization, $C=PDP^{-1}$, we know that we can choose $P$ to be an orthogonal matrix. It belongs to a ...
• 6,871
Accepted

### Get accurate eigenvectors, when eigenvalues are minuscule

The problem is due to "leakage" from the large eigenvectors. I will present a brief analysis and then offer a solution, with code, followed by some remarks about the nature and limitations ...
• 328k

### Eigenvalues/Eigenvectors of Correlation and Covariance matrices

Expanding on my comment: Since $P = \text{diag}(\Sigma)^{-1/2} \Sigma \text{diag}(\Sigma)^{-1/2}$, where $\text{diag}(\Sigma)$ is the diagonal matrix obtained by considering only the diagonal entries ...
• 5,110

### Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$?

Let's say we have a matrix $X=[x_1, x_2, \cdots, x_n]$ , where each $x_i$ is an obervation (sample) from $d$ dimension space, so $X$ is a $d$ by $n$ matrix, and $d > n$. If we first centered the ...
• 91

### Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

For Python users, I'd like to point out that for symmetric matrices (like the covariance matrix), it is better to use numpy.linalg.eigh function instead of a ...
• 91

• 9,427
Accepted

### Eigenvalues/Eigenvectors of Correlation and Covariance matrices

If $\Sigma$ is diagonal (with arbitrary eigenvalues) then $P$ is just the unit matrix (all eigenvalues equal to one), so there cannot be any general relation between the eigenvalues of $\Sigma$ (alone)...
• 5,400

### Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)

The eigenvalues would really only reveal how many classes (single classifier) or how many classifiers are correlated with one another (multiple classifiers). But if you look at the quasi-diagonalized ...
• 860

### Orthogonality in PCA vectors

From a simple geometric point of view: If the second eigenvector was not orthogonal to the first, then either the first eigenvector would not account for as much variation as possible, or the second ...
• 1,232

### Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

As I outlined in a comment to @whuber's answer, this method to compute the SVD doesn't work for every matrix. The issue is not limited to signs. The problem is that there may be repeated eigenvalues, ...

### Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

PCA works by computing the eigenvectors of the covariance matrix of the data. That is, those eigenvectors correspond to the choices of $a_{1:M}$ that maximize the equations and meet the constraints ...
• 7,742
Accepted

### Lower bound on smallest eigenvalue of covariance matrices

For a symmetric matrix the one norm and the infinity norm coincide. So the condition on the norm $$\Vert \Sigma(\theta) - I_{p}\Vert_{1} = \Vert\Sigma(\theta) - I_{p}\Vert_{\infty} \leq a$$ implies ...
• 1,366
Accepted

### If the eigenvalues of a covariance matrix have very low variance, what does it mean?

Each eigenvalue indicates the variance of the data along the direction of the corresponding eigenvector. If the data are jointly Gaussian, then the covariance matrix completely determines the shape ...
• 32.9k