# Questions tagged [eigenvalues]

For questions involving calculation or interpretation of eigenvalues or eigenvectors.

55 questions
Filter by
Sorted by
Tagged with
28answers
743k views

### Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues. I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
2answers
13k views

### Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$?

In PCA, when the number of dimensions $d$ is greater than (or even equal to) the number of samples $N$, why is it that you will have at most $N-1$ non-zero eigenvectors? In other words, the rank of ...
3answers
50k views

### Why does correlation matrix need to be positive semi-definite and what does it mean to be or not to be positive semi-definite?

I have been researching the meaning of positive semi-definite property of correlation or covariance matrices. I am looking for any information on Definition of positive semi-definiteness; Its ...
2answers
17k views

### Why does PCA maximize total variance of the projection?

Christopher Bishop writes in his book Pattern Recognition and Machine Learning a proof, that each consecutive principal component maximizes the variance of the projection to one dimension, after the ...
1answer
12k views

### What's the relationship between initial eigenvalues and sums of squared loadings in factor analysis?

On the one hand I read in a comment here that: You can't speak of "eigenvalues" after rotation, even orthogonal rotation. Perhaps you mean sum of squared loadings for a principal component, ...
3answers
49k views

### How does centering make a difference in PCA (for SVD and eigen decomposition)?

What difference does centering (or de-meaning) your data make for PCA? I've heard that it makes the maths easier or that it prevents the first PC from being dominated by the variables' means, but I ...
3answers
8k views

### Is centering a valid solution for multicollinearity?

Let's assume that $y = a + a_1x_1 + a_2x_2 + a_3x_3 + e$ where $x_1$ and $x_2$ both are indexes both range from $0-10$ where $0$ is the minimum and $10$ is the maximum. I found by applying VIF, CI and ...
1answer
3k views

### PCA: Eigenvectors of opposite sign and not being able to compute eigenvectors with solve in R

I'm learning PCA in R language. I met two problems right now that I don't understand. I am performing a PCA analysis in R on a 318×17 dataset using some custom code. I take eigen function in R to ...
2answers
5k views

### Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (...
2answers
777 views

### Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

I am trying to do SVD by hand: ...
1answer
3k views

### Eigenvalues of correlation matrices exhibit exponential decay

I have a data-set of $P$ samples of size $N$, and noticed that the eigenvalues of the correlation matrices $A^TA$, when presented in descending order, can in many cases be described as an exponential ...
1answer
2k views

### Do the principal components change if we apply PCA more than once (recursively) on data?

Consider a set $X=(X_1; \dots; X_n)$ of $n$ data points such that $X_i \in \mathbb{R}^d$ is a column vector. Let $Y = \text{pca_proj}(X)$ denote the projection of points in $X$ according to the PCA ...
1answer
134 views

### How to prove positive (semi-)definitness in matrix notation without numbers

I'd like to show that $$V[\hat\beta_{OLS}]-V[\hat\beta_{GLS}]=\sigma^2(X'X)^{-1}(X'\Omega X)(X'X)^{-1}-\sigma^2(X'\Omega X)^{-1}\geq 0$$ is positive (semi-) definite. $\Omega$ and $X$ are square-...
7answers
21k views

### Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
1answer
5k views

### What is principal subspace in probabilistic PCA?

if $X$ is observed data matrix and $Y$ is latent variable then $$X=WY+\mu+\epsilon$$ Where $\mu$ is the mean of observed data, and $\epsilon$ is the Gaussian error/noise in data, and $W$ is called ...
3answers
11k views

### Is every correlation matrix positive definite?

I'm talking here about matrices of Pearson correlations. I've often heard it said that all correlation matrices must be positive semidefinite. My understanding is that positive definite matrices must ...
1answer
5k views

### What exactly is the procedure to compute principal components in kernel PCA?

In kernel PCA (principal component analysis) you first choose a desired kernel, use it to find your $K$ matrix, center the feature space via the $K$ matrix, find its eigenvalues and eigenvectors, then ...
1answer
3k views

### How is PCA applied to new data?

I understand the basic intuition behind PCA: reducing the dimensionality of data by finding the eigenvectors along which there is most variance in the data, and projecting the data along these ...
4answers
5k views

### Condition number of covariance matrix

I am interested in generating a covariance matrix of dimension say 100. I managed to get a correlation matrix with finite condition number. To construct a covariance matrix I need to have standard ...
0answers
591 views

### EFA - ML Eigenvalue estimates in SAS

I am conducting an EFA (exploratory factor analysis) in SAS using maximum likelihood estimation (ML). I am trying to understand the eigenvalues output and how they should be used in interpreting the ...
1answer
146 views

0answers
239 views

### Does it make sense to add the principal components together to produce a single index? [duplicate]

I want to build an index from, say, 13 variables. I run a PCA for these 13 variables to produce 13 principal components, 5 of which have an Eigenvalue of more than 1. While some researchers use only ...
1answer
650 views

### Compute covariance matrix via rank-1 update to $M^\top M$

I have a large, sparse matrix $M\in\mathbb{R}^{n\times p}$. Centering $M$ to compute the covariance matrix $\Sigma$ would, in general, destroy the "zeros aren't stored" property of sparse matrices. ...
1answer
6k views

### Relation between best fit line and eigenvector of maximum eigen value of an estimated covariance matrix [closed]

(This question is from my pattern recognition course.) There is this exercise: Imagine we have $N$ samples with $n$ dimensions. First it asks to find a point $m$ where the summation of Euclidean ...
1answer
1k views

### Drawing 95% ellipse over scatter plot

The context is regression analysis using Eviews, but first I wanted to create a few scatter plots and overlay error ellipses on them. Eviews doesn't support that kind of graph ornamentation so I am ...
1answer
539 views

### Estimated distribution of eigenvalues for i.i.d. (uniform or normal) data

Assuming I have a data set with $d$ dimensions (e.g. $d=20$) so that each dimension is i.i.d. $X_i \sim U[0;1]$ (alternatively, each dimension $X_i \sim \mathcal N[0;1]$) and independent of each other....
2answers
3k views

### Quadratic form and Chi-squared distribution

It's about the demostration of the quadratic forms and chi-squared distribution. Let's split the problem: We have a $n$ vector with n standardized normal distribution called $z={[z_1,z_2...z_n]}$. ...
1answer
1k views

### Adversarial noise in PCA

PCA is known to be quite sensitive to outlier noise (and this is why several Robust PCA techniques exists.) However, I am looking for a concrete example of sensitivity of PCA to adversarial noise that ...
1answer
4k views

### Maximum number of principal components in PCA. Is sklearn wrong?

Recently I've been interested in applying PCA to a dataset I have and I wanted to develop a deep understanding of what I would actually be doing when I implement it. Today I encountered two ...
1answer
337 views

### Must a matrix of sample pairwise covariances be PSD?

Consider a random vector $\mathbf{X}=(X)_{i=1}^n$. Then the covariance matrix $$C=\mathbb{E}[(\mathbf{X}-\mu(\mathbf{X}))(\mathbf{X}-\mu(\mathbf{X}))^\top]$$ is by definition positive-semidefinite. (...
0answers
6k views

### What is the rationale behind the “eigenvalue > 1” criterion in factor analysis or PCA?

What is the meaning of "eigenvalue > 1" criterion? I understand what eigenvalues and eigenvectors are. This question is w.r.t. this link and this statement there: By default, VARCLUS stops ...
0answers
340 views

### How do random data eigenvalues change, as random variables are added?

I am using parallel analysis (Horn 1965) to determine how many principal components I can extract from my data. I can add more variables to my dataset, but I cannot add more cases (I know, that's ...
1answer
162 views

### Is a covariance matrix composed of matrixes derived from separate samples guaranteed to be positive definitive?

I have two samples that partially overlap on the variables they describe. The samples are taken from more or less the same population, and show similar values on the overlapping variables. Based on ...
2answers
506 views

### Compute the $k$ largest eigenvector in spectral clustering

In Spectral Clustering, we need to compute the top $k$ largest eigenvector of normalized $L$. $$L = D^{-\frac{1}{2}}SD^{-\frac{1}{2}}$$ In Andrew NG's paper, L is not positive definite (unless ...
1answer
116 views

### What is the duality relationship between eigensystems of X X' vs X' X?

I want to know the relationship between the eigensystems of two non-negative-definite (covariance-like) matrices. Both are derived from X which is a T-by-K real matrix (wlog say K > T). I avoid ...
1answer
478 views

### Givens rotation with Eigendecomposition

Setup Let $\Sigma_t$ be the time $t$ positive-definite covariance matrix of some $N$-dimensional random vector $X_t$. Denote $\Sigma_t = P_t \Lambda_t P_t$ be the spectral/eigen-decomposition of this ...
0answers
274 views

### Eigenvalue decomposition of a covariance matrix using a fast Cholesky decomposition

Let $\mathbf{C}$ be a $n \times n$ covariance matrix and assume that the LDL' Cholesky decomposition can be obtained efficiently. Can we take advantage of this to obtain a fast eigenvalue ...
1answer
4k views

### Analysis of compounds using PCA - selecting the right PCA “type” for the data…?

I'm completing scientific analysis of chemical compounds in consumer products. As a non-statistician, I would really appreciate any thoughts from the experts here. My data is non-normal so I've used ...