Questions tagged [eigenvalues]

For questions involving calculation or interpretation of eigenvalues or eigenvectors.

Filter by
Sorted by
Tagged with
1178
votes
28answers
743k views

Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues. I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
29
votes
2answers
13k views

Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$?

In PCA, when the number of dimensions $d$ is greater than (or even equal to) the number of samples $N$, why is it that you will have at most $N-1$ non-zero eigenvectors? In other words, the rank of ...
46
votes
3answers
50k views

Why does correlation matrix need to be positive semi-definite and what does it mean to be or not to be positive semi-definite?

I have been researching the meaning of positive semi-definite property of correlation or covariance matrices. I am looking for any information on Definition of positive semi-definiteness; Its ...
14
votes
2answers
17k views

Why does PCA maximize total variance of the projection?

Christopher Bishop writes in his book Pattern Recognition and Machine Learning a proof, that each consecutive principal component maximizes the variance of the projection to one dimension, after the ...
4
votes
1answer
12k views

What's the relationship between initial eigenvalues and sums of squared loadings in factor analysis?

On the one hand I read in a comment here that: You can't speak of "eigenvalues" after rotation, even orthogonal rotation. Perhaps you mean sum of squared loadings for a principal component, ...
46
votes
3answers
49k views

How does centering make a difference in PCA (for SVD and eigen decomposition)?

What difference does centering (or de-meaning) your data make for PCA? I've heard that it makes the maths easier or that it prevents the first PC from being dominated by the variables' means, but I ...
6
votes
3answers
8k views

Is centering a valid solution for multicollinearity?

Let's assume that $y = a + a_1x_1 + a_2x_2 + a_3x_3 + e$ where $x_1$ and $x_2$ both are indexes both range from $0-10$ where $0$ is the minimum and $10$ is the maximum. I found by applying VIF, CI and ...
6
votes
1answer
3k views

PCA: Eigenvectors of opposite sign and not being able to compute eigenvectors with `solve` in R

I'm learning PCA in R language. I met two problems right now that I don't understand. I am performing a PCA analysis in R on a 318×17 dataset using some custom code. I take eigen function in R to ...
11
votes
2answers
5k views

Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (...
9
votes
2answers
777 views

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

I am trying to do SVD by hand: ...
6
votes
1answer
3k views

Eigenvalues of correlation matrices exhibit exponential decay

I have a data-set of $P$ samples of size $N$, and noticed that the eigenvalues of the correlation matrices $A^TA$, when presented in descending order, can in many cases be described as an exponential ...
7
votes
1answer
2k views

Do the principal components change if we apply PCA more than once (recursively) on data?

Consider a set $X=(X_1; \dots; X_n)$ of $n$ data points such that $X_i \in \mathbb{R}^d$ is a column vector. Let $Y = \text{pca_proj}(X)$ denote the projection of points in $X$ according to the PCA ...
5
votes
1answer
134 views

How to prove positive (semi-)definitness in matrix notation without numbers

I'd like to show that $$V[\hat\beta_{OLS}]-V[\hat\beta_{GLS}]=\sigma^2(X'X)^{-1}(X'\Omega X)(X'X)^{-1}-\sigma^2(X'\Omega X)^{-1}\geq 0$$ is positive (semi-) definite. $\Omega$ and $X$ are square-...
41
votes
7answers
21k views

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
14
votes
1answer
5k views

What is principal subspace in probabilistic PCA?

if $X$ is observed data matrix and $Y$ is latent variable then $$X=WY+\mu+\epsilon$$ Where $\mu$ is the mean of observed data, and $\epsilon$ is the Gaussian error/noise in data, and $W$ is called ...
12
votes
3answers
11k views

Is every correlation matrix positive definite?

I'm talking here about matrices of Pearson correlations. I've often heard it said that all correlation matrices must be positive semidefinite. My understanding is that positive definite matrices must ...
8
votes
1answer
5k views

What exactly is the procedure to compute principal components in kernel PCA?

In kernel PCA (principal component analysis) you first choose a desired kernel, use it to find your $K$ matrix, center the feature space via the $K$ matrix, find its eigenvalues and eigenvectors, then ...
7
votes
1answer
3k views

How is PCA applied to new data?

I understand the basic intuition behind PCA: reducing the dimensionality of data by finding the eigenvectors along which there is most variance in the data, and projecting the data along these ...
6
votes
4answers
5k views

Condition number of covariance matrix

I am interested in generating a covariance matrix of dimension say 100. I managed to get a correlation matrix with finite condition number. To construct a covariance matrix I need to have standard ...
1
vote
0answers
591 views

EFA - ML Eigenvalue estimates in SAS

I am conducting an EFA (exploratory factor analysis) in SAS using maximum likelihood estimation (ML). I am trying to understand the eigenvalues output and how they should be used in interpreting the ...
1
vote
1answer
146 views

Is it possible to have a basis for a covariance matrix such that the greatest variance is greater than the variance of the first eigenvector?

Suppose we have a covariance matrix $C$. Define the eigendecomposition, $C = Q^{-1} \Lambda Q$ and some other arbitrary basis $C = B^{-1} D B$. Define $V_\text{PCA} = \text{diag}(QCQ^{-1})$ and $V_B =...
13
votes
1answer
395 views

Explain how `eigen` helps inverting a matrix

My question relates to a computation technique exploited in geoR:::.negloglik.GRF or geoR:::solve.geoR. In a linear mixed model ...
12
votes
1answer
3k views

Why are eigen and svd decompositions of a covariance matrix based on sparse data yielding different results?

I am trying to decompose a covariance matrix based on a sparse / gappy data set. I'm noticing that the sum of lambda (explained variance), as calculated with svd, ...
11
votes
1answer
989 views

A paper mentions a “Monte Carlo simulation to determine the number of principal components”; how does it work?

I'm doing a Matlab analysis on MRI data where I have performed PCA on a matrix sized 10304x236 where 10304 is the number of voxels (think of them as pixels) and 236 is the number of timepoints. The ...
9
votes
1answer
6k views

Visualizing PCA in R: data points, eigenvectors, projections, confidence ellipse

I have a dataset of 17 people, ranking 77 statements. I want to extract principal components on a transposed correlation matrix of correlations between people (as variables) across statements (as ...
4
votes
1answer
13k views

Does the sign of eigenvectors matter? [duplicate]

I had an matrix ((2,1,1),(-11,4,5),(-1,1-0)) I got the eigen values to be -1,1,2 for the eigenvalue -1 I got an eigenvector (0,1,-1) on the answers it says the answer is (0,-1,1). Is there an actual ...
4
votes
1answer
1k views

When the sample covariance matrix becomes singular

Assume a data set $X$ which contains $k$ iid random vectors of size $p$. Denote by $S$ the sample covariance matrix. Really I have some questions and I need your very appreciated opinions: 1) Ledoit ...
4
votes
0answers
3k views

What is the correct way to calculate the explained variance of each EOF as calculated from a gappy data set?

I am trying to determine the correct amount of variance explained by each mode of an Empirical Orthogonal Function (EOF) analysis (similar to "PCA") as applied to a gappy data set. (i.e., containing ...
3
votes
1answer
5k views

Eigenvalue vs Variance

I recently read that eigenvalue indicates the variance for an attribute/dimension. But is there a relation/equation between eigenvalue and variance? Is is right to say eigenvalue is equal to variance (...
0
votes
1answer
105 views

Gradient descent derivation in Eigenspace [duplicate]

I am trying to decode article on https://distill.pub/2017/momentum/ I was able to follow everything until the part with a change of basis x$^k=Q^T(w^k−w^⋆)$ to eigenspace... I conceptually understand ...
6
votes
0answers
2k views

Appropriate negative eigenvalue correction for PCoA of genetic distances

I am trying to find the best way to represent genetic distances in a plane so that they may use them as response variables in canonical redundancy analysis (using ...
5
votes
2answers
4k views

Negative eigenvalues in principle component analysis in the presence of missing data

For the purpose of dimension reduction I have performed an eigen analysis (using Jacobi-iteration) on a correlation matrix R of 163 variables (based on 1500 cases). The scree plot is attached. The ...
3
votes
0answers
334 views

What does it mean to compute eigenvectors of a covariance matrix if the data were not centered first? [duplicate]

Say $\mathbf{X} \in \mathbb{R}^{n \times p}$ and $\boldsymbol{\Sigma} = \frac{1}{n}\mathbf{X}'\mathbf{X}$. The eigenvector decomposition of $\boldsymbol{\Sigma}$ gives $\boldsymbol{\Sigma} = \mathbf{P}...
3
votes
0answers
239 views

Does it make sense to add the principal components together to produce a single index? [duplicate]

I want to build an index from, say, 13 variables. I run a PCA for these 13 variables to produce 13 principal components, 5 of which have an Eigenvalue of more than 1. While some researchers use only ...
2
votes
1answer
650 views

Compute covariance matrix via rank-1 update to $M^\top M$

I have a large, sparse matrix $M\in\mathbb{R}^{n\times p}$. Centering $M$ to compute the covariance matrix $\Sigma$ would, in general, destroy the "zeros aren't stored" property of sparse matrices. ...
1
vote
1answer
6k views

Relation between best fit line and eigenvector of maximum eigen value of an estimated covariance matrix [closed]

(This question is from my pattern recognition course.) There is this exercise: Imagine we have $N$ samples with $n$ dimensions. First it asks to find a point $m$ where the summation of Euclidean ...
1
vote
1answer
1k views

Drawing 95% ellipse over scatter plot

The context is regression analysis using Eviews, but first I wanted to create a few scatter plots and overlay error ellipses on them. Eviews doesn't support that kind of graph ornamentation so I am ...
9
votes
1answer
539 views

Estimated distribution of eigenvalues for i.i.d. (uniform or normal) data

Assuming I have a data set with $d$ dimensions (e.g. $d=20$) so that each dimension is i.i.d. $X_i \sim U[0;1]$ (alternatively, each dimension $X_i \sim \mathcal N[0;1]$) and independent of each other....
8
votes
2answers
3k views

Quadratic form and Chi-squared distribution

It's about the demostration of the quadratic forms and chi-squared distribution. Let's split the problem: We have a $n$ vector with n standardized normal distribution called $z={[z_1,z_2...z_n]}$. ...
6
votes
1answer
1k views

Adversarial noise in PCA

PCA is known to be quite sensitive to outlier noise (and this is why several Robust PCA techniques exists.) However, I am looking for a concrete example of sensitivity of PCA to adversarial noise that ...
5
votes
1answer
4k views

Maximum number of principal components in PCA. Is sklearn wrong?

Recently I've been interested in applying PCA to a dataset I have and I wanted to develop a deep understanding of what I would actually be doing when I implement it. Today I encountered two ...
5
votes
1answer
337 views

Must a matrix of sample pairwise covariances be PSD?

Consider a random vector $\mathbf{X}=(X)_{i=1}^n$. Then the covariance matrix $$C=\mathbb{E}[(\mathbf{X}-\mu(\mathbf{X}))(\mathbf{X}-\mu(\mathbf{X}))^\top]$$ is by definition positive-semidefinite. (...
4
votes
0answers
6k views

What is the rationale behind the “eigenvalue > 1” criterion in factor analysis or PCA?

What is the meaning of "eigenvalue > 1" criterion? I understand what eigenvalues and eigenvectors are. This question is w.r.t. this link and this statement there: By default, VARCLUS stops ...
4
votes
0answers
340 views

How do random data eigenvalues change, as random variables are added?

I am using parallel analysis (Horn 1965) to determine how many principal components I can extract from my data. I can add more variables to my dataset, but I cannot add more cases (I know, that's ...
4
votes
1answer
162 views

Is a covariance matrix composed of matrixes derived from separate samples guaranteed to be positive definitive?

I have two samples that partially overlap on the variables they describe. The samples are taken from more or less the same population, and show similar values on the overlapping variables. Based on ...
3
votes
2answers
506 views

Compute the $k$ largest eigenvector in spectral clustering

In Spectral Clustering, we need to compute the top $k$ largest eigenvector of normalized $L$. $$L = D^{-\frac{1}{2}}SD^{-\frac{1}{2}}$$ In Andrew NG's paper, L is not positive definite (unless ...
3
votes
1answer
116 views

What is the duality relationship between eigensystems of X X' vs X' X?

I want to know the relationship between the eigensystems of two non-negative-definite (covariance-like) matrices. Both are derived from X which is a T-by-K real matrix (wlog say K > T). I avoid ...
3
votes
1answer
478 views

Givens rotation with Eigendecomposition

Setup Let $\Sigma_t$ be the time $t$ positive-definite covariance matrix of some $N$-dimensional random vector $X_t$. Denote $\Sigma_t = P_t \Lambda_t P_t$ be the spectral/eigen-decomposition of this ...
2
votes
0answers
274 views

Eigenvalue decomposition of a covariance matrix using a fast Cholesky decomposition

Let $\mathbf{C}$ be a $n \times n$ covariance matrix and assume that the LDL' Cholesky decomposition can be obtained efficiently. Can we take advantage of this to obtain a fast eigenvalue ...
2
votes
1answer
4k views

Analysis of compounds using PCA - selecting the right PCA “type” for the data…?

I'm completing scientific analysis of chemical compounds in consumer products. As a non-statistician, I would really appreciate any thoughts from the experts here. My data is non-normal so I've used ...