Questions tagged [eigenvalues]

For questions involving calculation or interpretation of eigenvalues or eigenvectors.

Filter by
Sorted by
Tagged with
1114
votes
28answers
684k views

Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues. I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
43
votes
3answers
46k views

Why does correlation matrix need to be positive semi-definite and what does it mean to be or not to be positive semi-definite?

I have been researching the meaning of positive semi-definite property of correlation or covariance matrices. I am looking for any information on Definition of positive semi-definiteness; Its ...
41
votes
3answers
41k views

How does centering make a difference in PCA (for SVD and eigen decomposition)?

What difference does centering (or de-meaning) your data make for PCA? I've heard that it makes the maths easier or that it prevents the first PC from being dominated by the variables' means, but I ...
38
votes
7answers
18k views

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
33
votes
1answer
2k views

If I generate a random symmetric matrix, what's the chance it is positive definite?

I got a strange question when I was experimenting some convex optimizations. The question is: Suppose I randomly (say standard normal distribution) generate a $N \times N$ symmetric matrix, (for ...
26
votes
2answers
11k views

Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$?

In PCA, when the number of dimensions $d$ is greater than (or even equal to) the number of samples $N$, why is it that you will have at most $N-1$ non-zero eigenvectors? In other words, the rank of ...
14
votes
1answer
1k views

What is the meaning of the eigenvectors of a mutual information matrix?

When looking at the eigenvectors of the covariance matrix, we get the directions of maximum variance (the first eigenvector is the direction in which the data varies the most, etc.); this is called ...
13
votes
1answer
369 views

Explain how `eigen` helps inverting a matrix

My question relates to a computation technique exploited in geoR:::.negloglik.GRF or geoR:::solve.geoR. In a linear mixed model ...
12
votes
3answers
10k views

Is every correlation matrix positive definite?

I'm talking here about matrices of Pearson correlations. I've often heard it said that all correlation matrices must be positive semidefinite. My understanding is that positive definite matrices must ...
12
votes
2answers
15k views

Why does PCA maximize total variance of the projection?

Christopher Bishop writes in his book Pattern Recognition and Machine Learning a proof, that each consecutive principal component maximizes the variance of the projection to one dimension, after the ...
12
votes
1answer
3k views

Why are eigen and svd decompositions of a covariance matrix based on sparse data yielding different results?

I am trying to decompose a covariance matrix based on a sparse / gappy data set. I'm noticing that the sum of lambda (explained variance), as calculated with svd, ...
10
votes
1answer
4k views

What is principal subspace in probabilistic PCA?

if $X$ is observed data matrix and $Y$ is latent variable then $$X=WY+\mu+\epsilon$$ Where $\mu$ is the mean of observed data, and $\epsilon$ is the Gaussian error/noise in data, and $W$ is called ...
10
votes
1answer
214 views

Why eigenvectors reveals the groups in Spectral Clustering

According to Handbook of Cluster Analysis Spectral Clustering is done with following algorithm: Input Similarity Matrix $S$, number of clusters $K$ Form the transition matrix $P$ with $P_{ij} = S_{...
10
votes
1answer
1k views

Confused about the visual explanation of eigenvectors: how can visually different datasets have the same eigenvectors?

A lot of statistics textbooks provide an intuitive illustration of what the eigenvectors of a covariance matrix are: The vectors u and z form the eigenvectors (well, eigenaxes). This makes sense. But ...
10
votes
1answer
899 views

A paper mentions a “Monte Carlo simulation to determine the number of principal components”; how does it work?

I'm doing a Matlab analysis on MRI data where I have performed PCA on a matrix sized 10304x236 where 10304 is the number of voxels (think of them as pixels) and 236 is the number of timepoints. The ...
9
votes
2answers
645 views

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

I am trying to do SVD by hand: ...
9
votes
2answers
532 views

Why is the amount of variance explained by my 1st PC so close to the average pairwise correlation?

What is the relationship between the first principal component(s) and the average correlation in the correlation matrix? For example, in an empirical application I observe that the average ...
9
votes
1answer
502 views

Estimated distribution of eigenvalues for i.i.d. (uniform or normal) data

Assuming I have a data set with $d$ dimensions (e.g. $d=20$) so that each dimension is i.i.d. $X_i \sim U[0;1]$ (alternatively, each dimension $X_i \sim \mathcal N[0;1]$) and independent of each other....
9
votes
1answer
5k views

Visualizing PCA in R: data points, eigenvectors, projections, confidence ellipse

I have a dataset of 17 people, ranking 77 statements. I want to extract principal components on a transposed correlation matrix of correlations between people (as variables) across statements (as ...
9
votes
1answer
388 views

Theoretical link between the graph diffusion/heat kernel and spectral clustering

The graph diffusion kernel of a Graph is the exponential of its Laplacian $\exp(-\beta L)$ (or a similar expression depending on how you define the kernel). If you have labels on some vertices, you ...
8
votes
2answers
3k views

Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (...
8
votes
1answer
4k views

What exactly is the procedure to compute principal components in kernel PCA?

In kernel PCA (principal component analysis) you first choose a desired kernel, use it to find your $K$ matrix, center the feature space via the $K$ matrix, find its eigenvalues and eigenvectors, then ...
8
votes
2answers
1k views

Why does the first eigenvector in PCA resemble the derivative of an underlying trend?

I am using PCA to analyze several spatially related time series, and it appears that the first eigenvector corresponds to the derivative of the mean trend of the series (example illustrated below). I ...
8
votes
1answer
230 views

Distribution of eigenvalues given one is known

I'm familiar with using insights from Random Matrix Theory to determine the number of principal components from the PCA of a covariance/correlation matrix to use to form factors. If the eigenvalue ...
7
votes
1answer
153 views

Multiplying vectors by the covariance matrix?

I thought I knew covariance but I'm starting to think that there's more to it. For example, what happens when you multiply observations by their corresponding covariance matrix? ...
7
votes
1answer
1k views

Do the principal components change if we apply PCA more than once (recursively) on data?

Consider a set $X=(X_1; \dots; X_n)$ of $n$ data points such that $X_i \in \mathbb{R}^d$ is a column vector. Let $Y = \text{pca_proj}(X)$ denote the projection of points in $X$ according to the PCA ...
7
votes
2answers
2k views

Quadratic form and Chi-squared distribution

It's about the demostration of the quadratic forms and chi-squared distribution. Let's split the problem: We have a $n$ vector with n standardized normal distribution called $z={[z_1,z_2...z_n]}$. ...
7
votes
1answer
2k views

How is PCA applied to new data?

I understand the basic intuition behind PCA: reducing the dimensionality of data by finding the eigenvectors along which there is most variance in the data, and projecting the data along these ...
7
votes
1answer
585 views

Shrinkage of the eigenvalues

Assume we have $n$ samples $X_1,..., X_n$ which are independent and identically distributed with mean = 0 and unknown non-singular covariance matrix $M$. Each sample $X_i$ is a vector of size $p\times ...
7
votes
1answer
2k views

Correlation between principal components

I have two matrices a, b of dimensions (100x500), (100x15000) and I am trying to find associations between sets of variables in both matrices. When I perform principal component analysis on matrix a,...
6
votes
3answers
7k views

Is centering a valid solution for multicollinearity?

Let's assume that $y = a + a_1x_1 + a_2x_2 + a_3x_3 + e$ where $x_1$ and $x_2$ both are indexes both range from $0-10$ where $0$ is the minimum and $10$ is the maximum. I found by applying VIF, CI and ...
6
votes
1answer
3k views

PCA: Eigenvectors of opposite sign and not being able to compute eigenvectors with `solve` in R

I'm learning PCA in R language. I met two problems right now that I don't understand. I am performing a PCA analysis in R on a 318×17 dataset using some custom code. I take eigen function in R to ...
6
votes
1answer
708 views

Interpretation of eigenvectors of Hessian inverse

I'm reading a paper in which they use the eigenvectors of the inverse Hessian of a continuous probability distribution to characterize dimensions along which the distribution is most and least ...
6
votes
1answer
645 views

In PCA, why do we assume that the covariance matrix is always diagonalizable?

In Principal Component Analysis, what is the justification for the assumption that the covariance matrix is always a diagonalizable matrix? What happens when the covariance matrix is not ...
6
votes
1answer
6k views

Confused about Cholesky and eigen decomposition

I'm looking to generate correlated random variables. I have a symmetric, positive definite matrix. So I know that you can use the Cholesky decomposition, however I keep being told that this only works ...
6
votes
1answer
1k views

Adversarial noise in PCA

PCA is known to be quite sensitive to outlier noise (and this is why several Robust PCA techniques exists.) However, I am looking for a concrete example of sensitivity of PCA to adversarial noise that ...
6
votes
1answer
890 views

Why does the direction with highest eigenvalue have the largest semi-axis?

So, in PCA, we decompose the covariance matrix by its eigenvalues and eigenvectors. I understand that an ellipsoid is fully characterized by the eigenvalues and eigenvectors of a positive definite ...
6
votes
1answer
8k views

How to interpret variation explained by principal coordinates?

I have recently seen a couple of Principal Coordinates Analysis (PCoA) projection plots which show "percentage variation explained" by the respective principal coordinates. Given that the analysis is ...
6
votes
1answer
560 views

Why do we use the eigenvectors of the Laplacian and not the Affinity matrix in spectral clustering?

In spectral clustering, it's standard practice to solve the eigenvector problem $$L v = \lambda v$$ where $L$ is the graph Laplacian, $v$ is the eigenvector related to eigenvalue $\lambda$. My ...
6
votes
0answers
2k views

Appropriate negative eigenvalue correction for PCoA of genetic distances

I am trying to find the best way to represent genetic distances in a plane so that they may use them as response variables in canonical redundancy analysis (using ...
5
votes
2answers
13k views

How does eigenvalues measure variance along the principal components in PCA? [duplicate]

I understand that eigenvalues measure variance along the principal components. Questions How are eigenvalues and variance same for PCA? What is the intuition behind this being the same? What is ...
5
votes
3answers
2k views

Perform PCA. Extract PCs. Can one then tell what the most important _original_ features were, from the PCs? [duplicate]

Suppose that you have 1000 features, and a data set made up of say, 50,000 points. Suppose then that we perform PCA, and we extract the top 5 PCs, since they explain 99.99 percent of the variance, and ...
5
votes
1answer
3k views

Eigenvalues of correlation matrices exhibit exponential decay

I have a data-set of $P$ samples of size $N$, and noticed that the eigenvalues of the correlation matrices $A^TA$, when presented in descending order, can in many cases be described as an exponential ...
5
votes
4answers
4k views

Condition number of covariance matrix

I am interested in generating a covariance matrix of dimension say 100. I managed to get a correlation matrix with finite condition number. To construct a covariance matrix I need to have standard ...
5
votes
1answer
384 views

Interpretation of the cluster criterion $\operatorname{tr}(S_W^{-1}S_B)$

There is a cluster criterion defined as: $$\mathcal{C} = \operatorname{tr}(S_W^{-1}S_B) = \sum_{i=1}^d \lambda_i,$$ where $\operatorname{tr}$ is the trace, $S_W$ is the pooled within-group scatter ...
5
votes
1answer
310 views

Must a matrix of sample pairwise covariances be PSD?

Consider a random vector $\mathbf{X}=(X)_{i=1}^n$. Then the covariance matrix $$C=\mathbb{E}[(\mathbf{X}-\mu(\mathbf{X}))(\mathbf{X}-\mu(\mathbf{X}))^\top]$$ is by definition positive-semidefinite. (...
5
votes
2answers
734 views

Why do the leading eigenvectors of $A$ maximize $\text{Tr}(D^TAD)$?

Given a matrix $X\in\mathbb{R}^{m\times n}$, I am trying to maximize $\text{Tr}(D^TX^TXD)$ over $D\in\mathbb{R}^{n\times l}$ ($n<l$) subject to $D^TD=I_l$, where $\text{Tr}$ denotes the trace, and $...
5
votes
2answers
3k views

Negative eigenvalues in principle component analysis in the presence of missing data

For the purpose of dimension reduction I have performed an eigen analysis (using Jacobi-iteration) on a correlation matrix R of 163 variables (based on 1500 cases). The scree plot is attached. The ...
5
votes
1answer
165 views

What happens if I do principal components of the principal components?

Just a hypothetical question: What happens if I do a second principal component analysis over the principal components derived from the first principal component analysis? What will be the ...
5
votes
1answer
731 views

How to best define a “contrast” in a Principal Component Analysis (PCA)?

I have been studying how to interpret principal components. I recently came across an example of a particular eigenvector: $$e_j^T = \left[ \frac{\sqrt{2} }{2}, \frac{-\sqrt{2} }{2}, 0, \dots,0 \...

1
2 3 4 5
7