# Questions tagged [eigenvalues]

For questions involving calculation or interpretation of eigenvalues or eigenvectors.

55 questions
Filter by
Sorted by
Tagged with
743k views

### Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues. I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
13k views

### Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$?

In PCA, when the number of dimensions $d$ is greater than (or even equal to) the number of samples $N$, why is it that you will have at most $N-1$ non-zero eigenvectors? In other words, the rank of ...
50k views

### Why does correlation matrix need to be positive semi-definite and what does it mean to be or not to be positive semi-definite?

I have been researching the meaning of positive semi-definite property of correlation or covariance matrices. I am looking for any information on Definition of positive semi-definiteness; Its ...
17k views

### Why does PCA maximize total variance of the projection?

Christopher Bishop writes in his book Pattern Recognition and Machine Learning a proof, that each consecutive principal component maximizes the variance of the projection to one dimension, after the ...
12k views

### What's the relationship between initial eigenvalues and sums of squared loadings in factor analysis?

On the one hand I read in a comment here that: You can't speak of "eigenvalues" after rotation, even orthogonal rotation. Perhaps you mean sum of squared loadings for a principal component, ...
49k views

### How does centering make a difference in PCA (for SVD and eigen decomposition)?

What difference does centering (or de-meaning) your data make for PCA? I've heard that it makes the maths easier or that it prevents the first PC from being dominated by the variables' means, but I ...
8k views

### Is centering a valid solution for multicollinearity?

Let's assume that $y = a + a_1x_1 + a_2x_2 + a_3x_3 + e$ where $x_1$ and $x_2$ both are indexes both range from $0-10$ where $0$ is the minimum and $10$ is the maximum. I found by applying VIF, CI and ...
3k views

### PCA: Eigenvectors of opposite sign and not being able to compute eigenvectors with solve in R

I'm learning PCA in R language. I met two problems right now that I don't understand. I am performing a PCA analysis in R on a 318×17 dataset using some custom code. I take eigen function in R to ...
5k views

### Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (...
777 views

### Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

I am trying to do SVD by hand: ...
3k views

### Eigenvalues of correlation matrices exhibit exponential decay

I have a data-set of $P$ samples of size $N$, and noticed that the eigenvalues of the correlation matrices $A^TA$, when presented in descending order, can in many cases be described as an exponential ...
2k views

### Do the principal components change if we apply PCA more than once (recursively) on data?

Consider a set $X=(X_1; \dots; X_n)$ of $n$ data points such that $X_i \in \mathbb{R}^d$ is a column vector. Let $Y = \text{pca_proj}(X)$ denote the projection of points in $X$ according to the PCA ...
134 views

### How to prove positive (semi-)definitness in matrix notation without numbers

I'd like to show that $$V[\hat\beta_{OLS}]-V[\hat\beta_{GLS}]=\sigma^2(X'X)^{-1}(X'\Omega X)(X'X)^{-1}-\sigma^2(X'\Omega X)^{-1}\geq 0$$ is positive (semi-) definite. $\Omega$ and $X$ are square-...
21k views

### Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
5k views

### What is principal subspace in probabilistic PCA?

if $X$ is observed data matrix and $Y$ is latent variable then $$X=WY+\mu+\epsilon$$ Where $\mu$ is the mean of observed data, and $\epsilon$ is the Gaussian error/noise in data, and $W$ is called ...
11k views

### Is every correlation matrix positive definite?

I'm talking here about matrices of Pearson correlations. I've often heard it said that all correlation matrices must be positive semidefinite. My understanding is that positive definite matrices must ...
5k views

### What exactly is the procedure to compute principal components in kernel PCA?

In kernel PCA (principal component analysis) you first choose a desired kernel, use it to find your $K$ matrix, center the feature space via the $K$ matrix, find its eigenvalues and eigenvectors, then ...
3k views

### How is PCA applied to new data?

I understand the basic intuition behind PCA: reducing the dimensionality of data by finding the eigenvectors along which there is most variance in the data, and projecting the data along these ...
5k views

### Condition number of covariance matrix

I am interested in generating a covariance matrix of dimension say 100. I managed to get a correlation matrix with finite condition number. To construct a covariance matrix I need to have standard ...
591 views

### EFA - ML Eigenvalue estimates in SAS

I am conducting an EFA (exploratory factor analysis) in SAS using maximum likelihood estimation (ML). I am trying to understand the eigenvalues output and how they should be used in interpreting the ...
146 views

239 views

### Does it make sense to add the principal components together to produce a single index? [duplicate]

I want to build an index from, say, 13 variables. I run a PCA for these 13 variables to produce 13 principal components, 5 of which have an Eigenvalue of more than 1. While some researchers use only ...
650 views

### Compute covariance matrix via rank-1 update to $M^\top M$

I have a large, sparse matrix $M\in\mathbb{R}^{n\times p}$. Centering $M$ to compute the covariance matrix $\Sigma$ would, in general, destroy the "zeros aren't stored" property of sparse matrices. ...
6k views

### Relation between best fit line and eigenvector of maximum eigen value of an estimated covariance matrix [closed]

(This question is from my pattern recognition course.) There is this exercise: Imagine we have $N$ samples with $n$ dimensions. First it asks to find a point $m$ where the summation of Euclidean ...
1k views

### Drawing 95% ellipse over scatter plot

The context is regression analysis using Eviews, but first I wanted to create a few scatter plots and overlay error ellipses on them. Eviews doesn't support that kind of graph ornamentation so I am ...
539 views

### Estimated distribution of eigenvalues for i.i.d. (uniform or normal) data

Assuming I have a data set with $d$ dimensions (e.g. $d=20$) so that each dimension is i.i.d. $X_i \sim U[0;1]$ (alternatively, each dimension $X_i \sim \mathcal N[0;1]$) and independent of each other....
3k views

### Quadratic form and Chi-squared distribution

It's about the demostration of the quadratic forms and chi-squared distribution. Let's split the problem: We have a $n$ vector with n standardized normal distribution called $z={[z_1,z_2...z_n]}$. ...
1k views

PCA is known to be quite sensitive to outlier noise (and this is why several Robust PCA techniques exists.) However, I am looking for a concrete example of sensitivity of PCA to adversarial noise that ...
4k views

### Maximum number of principal components in PCA. Is sklearn wrong?

Recently I've been interested in applying PCA to a dataset I have and I wanted to develop a deep understanding of what I would actually be doing when I implement it. Today I encountered two ...
337 views

### Must a matrix of sample pairwise covariances be PSD?

Consider a random vector $\mathbf{X}=(X)_{i=1}^n$. Then the covariance matrix $$C=\mathbb{E}[(\mathbf{X}-\mu(\mathbf{X}))(\mathbf{X}-\mu(\mathbf{X}))^\top]$$ is by definition positive-semidefinite. (...
6k views

### What is the rationale behind the “eigenvalue > 1” criterion in factor analysis or PCA?

What is the meaning of "eigenvalue > 1" criterion? I understand what eigenvalues and eigenvectors are. This question is w.r.t. this link and this statement there: By default, VARCLUS stops ...
340 views

### How do random data eigenvalues change, as random variables are added?

I am using parallel analysis (Horn 1965) to determine how many principal components I can extract from my data. I can add more variables to my dataset, but I cannot add more cases (I know, that's ...
162 views

### Is a covariance matrix composed of matrixes derived from separate samples guaranteed to be positive definitive?

I have two samples that partially overlap on the variables they describe. The samples are taken from more or less the same population, and show similar values on the overlapping variables. Based on ...
506 views

### Compute the $k$ largest eigenvector in spectral clustering

In Spectral Clustering, we need to compute the top $k$ largest eigenvector of normalized $L$. $$L = D^{-\frac{1}{2}}SD^{-\frac{1}{2}}$$ In Andrew NG's paper, L is not positive definite (unless ...
116 views

### What is the duality relationship between eigensystems of X X' vs X' X?

I want to know the relationship between the eigensystems of two non-negative-definite (covariance-like) matrices. Both are derived from X which is a T-by-K real matrix (wlog say K > T). I avoid ...
478 views

### Givens rotation with Eigendecomposition

Setup Let $\Sigma_t$ be the time $t$ positive-definite covariance matrix of some $N$-dimensional random vector $X_t$. Denote $\Sigma_t = P_t \Lambda_t P_t$ be the spectral/eigen-decomposition of this ...
Let $\mathbf{C}$ be a $n \times n$ covariance matrix and assume that the LDL' Cholesky decomposition can be obtained efficiently. Can we take advantage of this to obtain a fast eigenvalue ...