Questions tagged [eigenvalues]
For questions involving calculation or interpretation of eigenvalues or eigenvectors.
416
questions
1314
votes
27
answers
895k
views
Making sense of principal component analysis, eigenvectors & eigenvalues
In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues.
I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
- 13.4k
56
votes
3
answers
73k
views
How does centering make a difference in PCA (for SVD and eigen decomposition)?
What difference does centering (or de-meaning) your data make for PCA? I've heard that it makes the maths easier or that it prevents the first PC from being dominated by the variables' means, but I ...
- 1,786
54
votes
3
answers
64k
views
Why does correlation matrix need to be positive semi-definite and what does it mean to be or not to be positive semi-definite?
I have been researching the meaning of positive semi-definite property of correlation or covariance matrices.
I am looking for any information on
Definition of positive semi-definiteness;
Its ...
- 549
47
votes
7
answers
30k
views
Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?
I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
- 613
33
votes
1
answer
3k
views
If I generate a random symmetric matrix, what's the chance it is positive definite?
I got a strange question when I was experimenting some convex optimizations. The question is:
Suppose I randomly (say standard normal distribution) generate a $N \times N$ symmetric matrix, (for ...
- 35k
31
votes
2
answers
19k
views
Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$?
In PCA, when the number of dimensions $d$ is greater than (or even equal to) the number of samples $N$, why is it that you will have at most $N-1$ non-zero eigenvectors? In other words, the rank of ...
- 411
17
votes
1
answer
2k
views
What is the meaning of the eigenvectors of a mutual information matrix?
When looking at the eigenvectors of the covariance matrix, we get the directions of maximum variance (the first eigenvector is the direction in which the data varies the most, etc.); this is called ...
- 777
16
votes
1
answer
6k
views
What is principal subspace in probabilistic PCA?
if $X$ is observed data matrix and $Y$ is latent variable then
$$X=WY+\mu+\epsilon$$
Where $\mu$ is the mean of observed data, and $\epsilon$ is the Gaussian error/noise in data, and $W$ is called ...
- 593
15
votes
2
answers
20k
views
Why does PCA maximize total variance of the projection?
Christopher Bishop writes in his book Pattern Recognition and Machine Learning a proof, that each consecutive principal component maximizes the variance of the projection to one dimension, after the ...
- 1,248
14
votes
1
answer
3k
views
Why eigenvectors reveal the groups in Spectral Clustering
According to Handbook of Cluster Analysis Spectral Clustering is done with following algorithm:
Input Similarity Matrix $S$, number of clusters $K$
Form the transition matrix $P$ with $P_{ij} = S_{...
- 195
14
votes
1
answer
500
views
Explain how `eigen` helps inverting a matrix
My question relates to a computation technique exploited in geoR:::.negloglik.GRF or geoR:::solve.geoR.
In a linear mixed model ...
- 1,432
13
votes
3
answers
12k
views
Is every correlation matrix positive definite?
I'm talking here about matrices of Pearson correlations.
I've often heard it said that all correlation matrices must be positive semidefinite. My understanding is that positive definite matrices must ...
13
votes
2
answers
9k
views
Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?
I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (...
- 551
12
votes
2
answers
6k
views
Quadratic form and Chi-squared distribution
It's about the demostration of the quadratic forms and chi-squared distribution.
Let's split the problem:
We have a $n$ vector with n standardized normal distribution called $z={[z_1,z_2...z_n]}$.
...
- 395
12
votes
1
answer
3k
views
Why are eigen and svd decompositions of a covariance matrix based on sparse data yielding different results?
I am trying to decompose a covariance matrix based on a sparse / gappy data set. I'm noticing that the sum of lambda (explained variance), as calculated with svd, ...
- 3,682
11
votes
1
answer
10k
views
What exactly is the procedure to compute principal components in kernel PCA?
In kernel PCA (principal component analysis) you first choose a desired kernel, use it to find your $K$ matrix, center the feature space via the $K$ matrix, find its eigenvalues and eigenvectors, then ...
- 283
11
votes
1
answer
1k
views
A paper mentions a "Monte Carlo simulation to determine the number of principal components"; how does it work?
I'm doing a Matlab analysis on MRI data where I have performed PCA on a matrix sized 10304x236 where 10304 is the number of voxels (think of them as pixels) and 236 is the number of timepoints. The ...
- 359
11
votes
1
answer
7k
views
Visualizing PCA in R: data points, eigenvectors, projections, confidence ellipse
I have a dataset of 17 people, ranking 77 statements.
I want to extract principal components on a transposed correlation matrix of correlations between people (as variables) across statements (as ...
- 732
10
votes
2
answers
1k
views
Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?
I am trying to do SVD by hand:
...
10
votes
1
answer
1k
views
Confused about the visual explanation of eigenvectors: how can visually different datasets have the same eigenvectors?
A lot of statistics textbooks provide an intuitive illustration of what the eigenvectors of a covariance matrix are:
The vectors u and z form the eigenvectors (well, eigenaxes). This makes sense. But ...
- 739
9
votes
3
answers
699
views
Eigenvalues/Eigenvectors of Correlation and Covariance matrices
Suppose $\Sigma$ is a covariance matrix $P$ is its corresponding correlation matrix. Let $\lambda_1, \dots, \lambda_p$ and $\tau_1, \dots, \tau_p$ denote the ordered eigenvalues of $\Sigma$ and $P$, ...
- 14.9k
9
votes
2
answers
704
views
Why is the amount of variance explained by my 1st PC so close to the average pairwise correlation?
What is the relationship between the first principal component(s) and the average correlation in the correlation matrix?
For example, in an empirical application I observe that the average ...
- 93
9
votes
1
answer
2k
views
In PCA, why do we assume that the covariance matrix is always diagonalizable?
In Principal Component Analysis, what is the justification for the assumption that the covariance matrix is always a diagonalizable matrix?
What happens when the covariance matrix is not ...
- 690
9
votes
1
answer
8k
views
Confused about Cholesky and eigen decomposition
I'm looking to generate correlated random variables. I have a symmetric, positive definite matrix. So I know that you can use the Cholesky decomposition, however I keep being told that this only works ...
- 684
9
votes
1
answer
753
views
Estimated distribution of eigenvalues for i.i.d. (uniform or normal) data
Assuming I have a data set with $d$ dimensions (e.g. $d=20$) so that each dimension is i.i.d. $X_i \sim U[0;1]$ (alternatively, each dimension $X_i \sim \mathcal N[0;1]$) and independent of each other....
- 41.1k
9
votes
1
answer
531
views
Theoretical link between the graph diffusion/heat kernel and spectral clustering
The graph diffusion kernel of a Graph is the exponential of its Laplacian $\exp(-\beta L)$ (or a similar expression depending on how you define the kernel). If you have labels on some vertices, you ...
- 2,362
9
votes
0
answers
677
views
Why are the discriminant axes in linear discriminant analysis (LDA) not orthogonal?
This may be a quite silly question and please correct me if I'm wrong.
The discriminants (discriminant axes) are essentially eigenvectors of $\mathrm{Cov}_\mathrm{within}^{-1} \mathrm{Cov}_\mathrm{...
- 485
8
votes
3
answers
1k
views
Get accurate eigenvectors, when eigenvalues are minuscule
I have a symmetric matrix A. I'm not able to compute all the eigenvectors accurately, and I believe it is due to the last few eigenvalues for ...
- 117
8
votes
2
answers
2k
views
Why does the first eigenvector in PCA resemble the derivative of an underlying trend?
I am using PCA to analyze several spatially related time series, and it appears that the first eigenvector corresponds to the derivative of the mean trend of the series (example illustrated below). I ...
- 81
8
votes
1
answer
421
views
Distribution of eigenvalues given one is known
I'm familiar with using insights from Random Matrix Theory to determine the number of principal components from the PCA of a covariance/correlation matrix to use to form factors.
If the eigenvalue ...
- 2,237
8
votes
1
answer
977
views
Why do we use the eigenvectors of the Laplacian and not the Affinity matrix in spectral clustering?
In spectral clustering, it's standard practice to solve the eigenvector problem
$$L v = \lambda v$$
where $L$ is the graph Laplacian, $v$ is the eigenvector related to eigenvalue $\lambda$.
My ...
- 951
8
votes
1
answer
3k
views
Pagerank vs Eigenvector centrality
What are the practical differences between PageRank and Eigenvector centralities? I don't mean the differents in how to compute the centralities, but the information they provide of a set of nodes in ...
8
votes
0
answers
3k
views
Appropriate negative eigenvalue correction for PCoA of genetic distances
I am trying to find the best way to represent genetic distances in a plane so that they may use them as response variables in canonical redundancy analysis (using ...
- 116
7
votes
3
answers
10k
views
Is centering a valid solution for multicollinearity?
Let's assume that $y = a + a_1x_1 + a_2x_2 + a_3x_3 + e$ where $x_1$ and $x_2$ both are indexes both range from $0-10$ where $0$ is the minimum and $10$ is the maximum. I found by applying VIF, CI and ...
- 71
7
votes
1
answer
3k
views
nearPD function in Matrix package
Does anyone know how the eigenvalues are adjusted to make a non-positive definite matrix into a positive definite matrix in Matrix package? I mean in nearPD function.
- 651
7
votes
1
answer
3k
views
Do the principal components change if we apply PCA more than once (recursively) on data?
Consider a set $X=(X_1; \dots; X_n)$ of $n$ data points such that $X_i \in \mathbb{R}^d$ is a column vector. Let $Y = \text{pca_proj}(X)$ denote the projection of points in $X$ according to the PCA ...
- 2,221
7
votes
1
answer
899
views
Multiplying vectors by the covariance matrix?
I thought I knew covariance but I'm starting to think that there's more to it. For example, what happens when you multiply observations by their corresponding covariance matrix? ...
- 2,185
7
votes
1
answer
5k
views
How is PCA applied to new data?
I understand the basic intuition behind PCA: reducing the dimensionality of data by finding the eigenvectors along which there is most variance in the data, and projecting the data along these ...
- 1,244
7
votes
1
answer
787
views
How to prove positive (semi-)definitness in matrix notation without numbers
I'd like to show that $$V[\hat\beta_{OLS}]-V[\hat\beta_{GLS}]=\sigma^2(X'X)^{-1}(X'\Omega X)(X'X)^{-1}-\sigma^2(X'\Omega X)^{-1}\geq 0$$ is positive (semi-) definite. $\Omega$ and $X$ are square-...
- 109
7
votes
1
answer
923
views
Shrinkage of the eigenvalues
Assume we have $n$ samples $X_1,..., X_n$ which are independent and identically distributed with mean = 0 and unknown non-singular covariance matrix $M$. Each sample $X_i$ is a vector of size $p\times ...
- 765
7
votes
1
answer
3k
views
Correlation between principal components
I have two matrices a, b of dimensions (100x500), (100x15000) and I am trying to find associations between sets of variables in both matrices.
When I perform principal component analysis on matrix a,...
- 159
7
votes
0
answers
327
views
What is the precise relation between the eigenvalues of a covariance *function* and the eigenvalues of a covariance *matrix*?
Assume we have a temporal Gaussian Process $\mathcal{GP}(t;\ m,k)$ (GP) with mean $m$ and covariance function (aka. kernel) $k$ on some compact time interval $[0,T]$. Then, the eigenvalues $\lambda$ ...
- 71
6
votes
2
answers
32k
views
How does eigenvalues measure variance along the principal components in PCA? [duplicate]
I understand that eigenvalues measure variance along the principal components.
Questions
How are eigenvalues and variance same for PCA?
What is the intuition behind this being the same?
What is ...
- 5,493
6
votes
1
answer
4k
views
PCA: Eigenvectors of opposite sign and not being able to compute eigenvectors with `solve` in R
I'm learning PCA in R language. I met two problems right now that I don't understand.
I am performing a PCA analysis in R on a 318×17 dataset using some custom code. I take eigen function in R to ...
- 73
6
votes
1
answer
18k
views
Does the sign of eigenvectors matter? [duplicate]
I had an matrix ((2,1,1),(-11,4,5),(-1,1-0)) I got the eigen values to be -1,1,2
for the eigenvalue -1 I got an eigenvector (0,1,-1) on the answers it says the answer is (0,-1,1). Is there an actual ...
- 61
6
votes
1
answer
878
views
Interpretation of eigenvectors of Hessian inverse
I'm reading a paper in which they use the eigenvectors of the inverse Hessian of a continuous probability distribution to characterize dimensions along which the distribution is most and least ...
- 3,013
6
votes
1
answer
4k
views
Eigenvalues of correlation matrices exhibit exponential decay
I have a data-set of $P$ samples of size $N$, and noticed that the eigenvalues of the correlation matrices $A^TA$, when presented in descending order, can in many cases be described as an exponential ...
- 427
6
votes
4
answers
7k
views
Condition number of covariance matrix
I am interested in generating a covariance matrix of dimension say 100. I managed to get a correlation matrix with finite condition number.
To construct a covariance matrix I need to have standard ...
- 651
6
votes
1
answer
714
views
Second directional derivate and Hessian matrix
I am reading the following from the book Deep Learning, and I have the following questions.
I don't quite understand second directional derivatives. The first directional derivative of a function $f:\...
- 1,189
6
votes
2
answers
2k
views
Why do the leading eigenvectors of $A$ maximize $\text{Tr}(D^TAD)$?
Given a matrix $X\in\mathbb{R}^{m\times n}$, I am trying to maximize $\text{Tr}(D^TX^TXD)$ over $D\in\mathbb{R}^{n\times l}$ ($n<l$) subject to $D^TD=I_l$, where $\text{Tr}$ denotes the trace, and $...
- 1,189