Skip to main content

Questions tagged [pca]

Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.

Filter by
Sorted by
Tagged with
1351 votes
27 answers

Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues. I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
claws's user avatar
  • 13.8k
619 votes
5 answers

Relationship between SVD and PCA. How to use SVD to perform PCA?

Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix ...
amoeba's user avatar
  • 106k
267 votes
15 answers

What are the differences between Factor Analysis and Principal Component Analysis?

It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use ...
Brandon Bertelsen's user avatar
221 votes
6 answers

Can principal component analysis be applied to datasets containing a mix of continuous and categorical variables?

I have a dataset that has both continuous and categorical data. I am analyzing by using PCA and am wondering if it is fine to include the categorical variables as a part of the analysis. My ...
Nikolina Icitovic's user avatar
213 votes
7 answers

PCA on correlation or covariance?

What are the main differences between performing principal component analysis (PCA) on the correlation matrix and on the covariance matrix? Do they give the same results?
Random's user avatar
  • 2,310
175 votes
1 answer

How to reverse PCA and reconstruct original variables from several principal components?

Principal component analysis (PCA) can be used for dimensionality reduction. After such dimensionality reduction is performed, how can one approximately reconstruct the original variables/features ...
amoeba's user avatar
  • 106k
168 votes
5 answers

What's the difference between principal component analysis and multidimensional scaling?

How are PCA and classical MDS different? How about MDS versus nonmetric MDS? Is there a time when you would prefer one over the other? How do the interpretations differ?
Stephen Turner's user avatar
146 votes
6 answers

Should one remove highly correlated variables before doing PCA?

I'm reading a paper where author discards several variables due to high correlation to other variables before doing PCA. The total number of variables is around 20. Does this give any benefits? It ...
type2's user avatar
  • 1,571
133 votes
4 answers

PCA and proportion of variance explained

In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a ...
user9097's user avatar
  • 3,293
118 votes
2 answers

Why do we need to normalize data before principal component analysis (PCA)? [duplicate]

I'm doing principal component analysis on my dataset and my professor told me that I should normalize the data before doing the analysis. Why? What would happen If I did PCA without normalization? ...
jjepsuomi's user avatar
  • 5,847
116 votes
6 answers

What is the relation between k-means clustering and PCA?

It is a common practice to apply PCA (principal component analysis) before a clustering algorithm (such as k-means). It is believed that it improves the clustering results in practice (noise reduction)...
mic's user avatar
  • 4,388
111 votes
5 answers

Loadings vs eigenvectors in PCA: when to use one or another?

In principal component analysis (PCA), we get eigenvectors (unit vectors) and eigenvalues. Now, let us define loadings as $$\text{Loadings} = \text{Eigenvectors} \cdot \sqrt{\text{Eigenvalues}}.$$ I ...
user2696565's user avatar
  • 1,419
94 votes
3 answers

What is the intuition behind SVD?

I have read about singular value decomposition (SVD). In almost all textbooks it is mentioned that it factorizes the matrix into three matrices with given specification. But what is the intuition ...
SHASHANK GUPTA's user avatar
94 votes
7 answers

What are principal component scores?

What are principal component scores (PC scores, PCA scores)?
vrish88's user avatar
  • 1,223
90 votes
4 answers

What're the differences between PCA and autoencoder?

Both PCA and autoencoder can do demension reduction, so what are the difference between them? In what situation I should use one over another?
RockTheStar's user avatar
  • 13.1k
88 votes
4 answers

How to visualize what canonical correlation analysis does (in comparison to what principal component analysis does)?

Canonical correlation analysis (CCA) is a technique related to principal component analysis (PCA). While it is easy to teach PCA or linear regression using a scatter plot (see a few thousand examples ...
figure's user avatar
  • 983
86 votes
7 answers

Using principal component analysis (PCA) for feature selection

I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out ...
Michael's user avatar
  • 2,471
85 votes
3 answers

What is the difference between ZCA whitening and PCA whitening?

I am confused about ZCA whitening and normal whitening (which is obtained by dividing principal components by the square roots of PCA eigenvalues). As far as I know, $$\mathbf x_\mathrm{ZCAwhite} = \...
RockTheStar's user avatar
  • 13.1k
84 votes
6 answers

Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis?

In some disciplines, PCA (principal component analysis) is systematically used without any justification, and PCA and EFA (exploratory factor analysis) are considered as synonyms. I therefore ...
Carine's user avatar
  • 849
84 votes
6 answers

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA ...
stackoverflowuser2010's user avatar
83 votes
4 answers

What is the difference between R functions prcomp and princomp?

I compared ?prcomp and ?princomp and found something about Q-mode and R-mode principal component analysis (PCA). But honestly – ...
hans0l0's user avatar
  • 2,145
76 votes
4 answers

What makes the Gaussian kernel so magical for PCA, and also in general?

I was reading about kernel PCA (1, 2, 3) with Gaussian and polynomial kernels. How does the Gaussian kernel separate seemingly any sort of nonlinear data exceptionally well? Please give an intuitive ...
Simon Kuang's user avatar
  • 2,121
72 votes
8 answers

Is PCA followed by a rotation (such as varimax) still PCA?

I have tried to reproduce some research (using PCA) from SPSS in R. In my experience, principal() function from package psych ...
Roman Luštrik's user avatar
70 votes
4 answers

Are there cases where PCA is more suitable than t-SNE?

I want to see how 7 measures of text correction behaviour (time spent correcting the text, number of keystrokes, etc.) relate to each other. The measures are correlated. I ran a PCA to see how the ...
user3744206's user avatar
69 votes
9 answers

Advanced statistics books recommendation

There are several threads on this site for book recommendations on introductory statistics and machine learning but I am looking for a text on advanced statistics including, in order of priority: ...
66 votes
3 answers

What is the objective function of PCA?

Principal component analysis can use matrix decomposition, but that is just a tool to get there. How would you find the principal components without the use of matrix algebra? What is the objective ...
Neil McGuigan's user avatar
62 votes
2 answers

How does centering the data get rid of the intercept in regression and PCA?

I keep reading about instances where we center the data (e.g., with regularization or PCA) in order to remove the intercept (as mentioned in this question). I know it's simple, but I'm having a hard ...
Alec's user avatar
  • 2,395
59 votes
3 answers

How does centering make a difference in PCA (for SVD and eigen decomposition)?

What difference does centering (or de-meaning) your data make for PCA? I've heard that it makes the maths easier or that it prevents the first PC from being dominated by the variables' means, but I ...
Zenit's user avatar
  • 1,846
57 votes
4 answers

Would PCA work for boolean (binary) data types?

I want to reduce the dimensionality of higher order systems and capture most of the covariance on a preferably 2 dimensional or 1 dimensional field. I understand this can be done via principal ...
Alvin Nunez's user avatar
57 votes
7 answers

Best PCA algorithm for huge number of features (>10K)?

I previously asked this on StackOverflow, but it seems like it might be more appropriate here, given that it didn't get any answers on SO. It's kind of at the intersection between statistics and ...
dsimcha's user avatar
  • 8,809
55 votes
4 answers

Does the sign of scores or of loadings in PCA or FA have a meaning? May I reverse the sign?

I performed principal component analysis (PCA) with R using two different functions (prcomp and princomp) and observed that the ...
user1320502's user avatar
  • 1,007
52 votes
7 answers

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
DongukJu's user avatar
  • 663
50 votes
3 answers

PCA and the train/test split

I have a dataset for which I have multiple sets of binary labels. For each set of labels, I train a classifier, evaluating it by cross-validation. I want to reduce dimensionality using principal ...
Bitwise's user avatar
  • 6,679
48 votes
1 answer

PCA objective function: what is the connection between maximizing variance and minimizing error?

The PCA algorithm can be formulated in terms of the correlation matrix (assume the data $X$ has already been normalized and we are only considering projection onto the first PC). The objective ...
Cam.Davidson.Pilon's user avatar
47 votes
1 answer

PCA and Correspondence analysis in their relation to Biplot

Biplot is often used to display results of principal component analysis (and of related techniques). It is a dual or overlay scatterplot showing component loadings and component scores simultaneously. ...
ttnphns's user avatar
  • 58.3k
45 votes
3 answers

Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?

In a recent assignment, we were told to use PCA on the MNIST digits to reduce the dimensions from 64 (8 x 8 images) to 2. We then had to cluster the digits using a Gaussian Mixture Model. PCA using ...
willk's user avatar
  • 643
45 votes
1 answer

What is the intuitive reason behind doing rotations in Factor Analysis/PCA & how to select appropriate rotation?

My Questions What is the intuitive reason behind doing rotations of factors in factor analysis (or components in PCA)? My understanding is, if variables are almost equally loaded in the top ...
GeorgeOfTheRF's user avatar
43 votes
1 answer

Doing principal component analysis or factor analysis on binary data

I have a dataset with a large number of Yes/No responses. Can I use principal components (PCA) or any other data reduction analyses (such as factor analysis) for this type of data? Please advise how I ...
Cathy's user avatar
  • 431
42 votes
1 answer

How to determine significant principal components using bootstrapping or Monte Carlo approach?

I am interested in determining the number of significant patterns coming out of a Principal Component Analysis (PCA) or Empirical Orthogonal Function (EOF) Analysis. I am particularly interested in ...
Marc in the box's user avatar
41 votes
2 answers

How does Factor Analysis explain the covariance while PCA explains the variance?

Here is a quote from Bishop's "Pattern Recognition and Machine Learning" book, section 12.2.4 "Factor analysis": According to the highlighted part, factor analysis captures the covariance between ...
avocado's user avatar
  • 3,633
41 votes
1 answer

Is there Factor analysis or PCA for ordinal or binary data?

I have completed the principal component analysis (PCA), exploratory factor analysis (EFA), and confirmatory factor analysis (CFA), treating data with likert scale (5-level responses: none, a little, ...
user116948's user avatar
40 votes
5 answers

Examples of PCA where PCs with low variance are "useful"

Normally in principal component analysis (PCA) the first few PCs are used and the low variance PCs are dropped, as they do not explain much of the variation in the data. However, are there examples ...
Michael's user avatar
  • 403
40 votes
1 answer

What are the advantages of kernel PCA over standard PCA?

I want to implement an algorithm in a paper which uses kernel SVD to decompose a data matrix. So I have been reading materials about kernel methods and kernel PCA etc. But it still is very obscure to ...
CyberPlayerOne's user avatar
39 votes
2 answers

How would PCA help with a k-means clustering analysis?

Background: I want to classify the residential areas of a city into groups based on their social-economic characteristics, including housing unit density, population density, green space area, housing ...
enaJ's user avatar
  • 595
38 votes
3 answers

Linearity of PCA

PCA is considered a linear procedure, however: $$\mathrm{PCA}(X)\neq \mathrm{PCA}(X_1)+\mathrm{PCA}(X_2)+\ldots+\mathrm{PCA}(X_n),$$ where $X=X_1+X_2+\ldots+X_n$. This is to say that the ...
AlphaOmega's user avatar
37 votes
2 answers

Interpretation of biplots in principal components analysis

I came across this nice tutorial: A Handbook of Statistical Analyses Using R. Chapter 13. Principal Component Analysis: The Olympic Heptathlon on how to do PCA in R language. I don't understand the ...
user862's user avatar
  • 2,759
37 votes
2 answers

Can PCA be applied for time series data?

I understand that Principal Component Analysis (PCA) can be applied basically for cross sectional data. Can PCA be used for time series data effectively by specifying year as time series variable and ...
Nisha Simon's user avatar
36 votes
3 answers

How to perform orthogonal regression (total least squares) via PCA?

I always use lm() in R to perform linear regression of $y$ on $x$. That function returns a coefficient $\beta$ such that $$y = \beta x.$$ Today I learned about ...
Dail's user avatar
  • 2,657
36 votes
3 answers

Building an autoencoder in Tensorflow to surpass PCA

Hinton and Salakhutdinov in Reducing the Dimensionality of Data with Neural Networks, Science 2006 proposed a non-linear PCA through the use of a deep autoencoder. I have tried to build and train a ...
Donbeo's user avatar
  • 3,179
36 votes
3 answers

PCA on correlation or covariance: does PCA on correlation ever make sense? [closed]

In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components (from their respective eigenvectors). These give different results (...
Lucozade's user avatar
  • 659

2 3 4 5