Skip to main content
239 votes
Accepted

How to reverse PCA and reconstruct original variables from several principal components?

PCA computes eigenvectors of the covariance matrix ("principal axes") and sorts them by their eigenvalues (amount of explained variance). The centered data can then be projected onto these principal ...
amoeba's user avatar
  • 105k
103 votes

Are there cases where PCA is more suitable than t-SNE?

$t$-SNE is a great piece of Machine Learning but one can find many reasons to use PCA instead of it. Of the top of my head, I will mention five. As most other computational methodologies in use, $t$-...
usεr11852's user avatar
  • 44.2k
80 votes
Accepted

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

Problem statement The geometric problem that PCA is trying to optimize is clear to me: PCA tries to find the first principal component by minimizing the reconstruction (projection) error, which ...
amoeba's user avatar
  • 105k
62 votes
Accepted

Is there Factor analysis or PCA for ordinal or binary data?

Traditional (linear) PCA and Factor analysis require scale-level (interval or ratio) data. Often likert-type rating data are assumed to be scale-level, because such data are easier to analyze. And the ...
ttnphns's user avatar
  • 57.6k
59 votes

What are the differences between Factor Analysis and Principal Component Analysis?

A basic, yet a kind of painstaking, explanation of PCA vs Factor analysis with the help of scatterplots, in logical steps. (I thank @amoeba who, in his comment to the question, has encouraged me to ...
ttnphns's user avatar
  • 57.6k
50 votes

Building an autoencoder in Tensorflow to surpass PCA

Here is the key figure from the 2006 Science paper by Hinton and Salakhutdinov: It shows dimensionality reduction of the MNIST dataset ($28\times 28$ black and white images of single digits) from the ...
amoeba's user avatar
  • 105k
49 votes
Accepted

Linearity of PCA

When we say that PCA is a linear method, we refer to the dimensionality reducing mapping $f:\mathbf x\mapsto \mathbf z$ from high-dimensional space $\mathbb R^p$ to a lower-dimensional space $\mathbb ...
amoeba's user avatar
  • 105k
48 votes
Accepted

Calculating PCA variance explained

Yes, that's correct. summary.prcomp brings that information as well: ...
Firebug's user avatar
  • 19.1k
43 votes
Accepted

Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?

The main reason that $t$-SNE is not used in classification models is that it does not learn a function from the original space to the new (lower) dimensional one. As such, when we would try to use our ...
usεr11852's user avatar
  • 44.2k
42 votes

Relationship between SVD and PCA. How to use SVD to perform PCA?

I wrote a Python & Numpy snippet that accompanies @amoeba's answer and I leave it here in case it is useful for someone. The comments are mostly taken from @amoeba's answer. ...
40 votes
Accepted

Why is PCA sensitive to outliers?

One of the reasons is that PCA can be thought as low-rank decomposition of the data that minimizes the sum of $L_2$ norms of the residuals of the decomposition. I.e. if $Y$ is your data ($m$ vectors ...
sega_sai's user avatar
  • 827
39 votes
Accepted

Is PCA always recommended?

Blindly using PCA is a recipe for disaster. (As an aside, automatically applying any method is not a good idea, because what works in one context is not guaranteed to work in another. We can formalize ...
Sycorax's user avatar
  • 91k
33 votes

Relationship between SVD and PCA. How to use SVD to perform PCA?

Let me start with PCA. Suppose that you have $n$ data points comprised of $d$ numbers (or dimensions) each. If you center this data (subtract the mean data point $\mu$ from each data vector $x_i$) you ...
Andre P's user avatar
  • 511
33 votes
Accepted

PCA in numpy and sklearn produces different results

The difference is because decomposition.PCA does not standardize your variables before doing PCA, whereas in your manual computation you call ...
amoeba's user avatar
  • 105k
31 votes
Accepted

What is principal subspace in probabilistic PCA?

This is an excellent question. Probabilistic PCA (PPCA) is the following latent variable model \begin{align} \mathbf z &\sim \mathcal N(\mathbf 0, \mathbf I) \\ \mathbf x &\sim \mathcal N(\...
amoeba's user avatar
  • 105k
28 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

amoeba already gave a good answer in the comments, but if you want a formal argument, here it goes. The singular value decomposition of a matrix $A$ is $A=U\Sigma V^T$, where the columns of $V$ are ...
cangrejo's user avatar
  • 2,251
27 votes

Why do we use PCA to speed up learning algorithms when we could just reduce the number of features?

Let's say you initially have $p$ features but this is too many so you want to actually fit your model on $d < p$ features. You could choose $d$ of your features and drop the rest. If $X$ is our ...
jld's user avatar
  • 20.2k
27 votes
Accepted

Is PCA optimization convex?

No, the usual formulations of PCA are not convex problems. But they can be transformed into a convex optimization problem. The insight and the fun of this is following and visualizing the sequence of ...
whuber's user avatar
  • 324k
25 votes
Accepted

Difference between scikit-learn implementations of PCA and TruncatedSVD

PCA and TruncatedSVD scikit-learn implementations seem to be exactly the same algorithm. No: PCA is (truncated) SVD on centered data (by per-feature mean substraction). If the data is already ...
ogrisel's user avatar
  • 3,769
24 votes
Accepted

Does PCA preserve linear separability for every linearly separable set?

No, it may be that the discriminative information is in the direction of a principal component that explains a relatively small amount of the total variance, and hence gets discarded. Consider a two-...
Dikran Marsupial's user avatar
23 votes
Accepted

What is the connection between partial least squares, reduced rank regression, and principal component regression?

These are three different methods, and none of them can be seen as a special case of another. Formally, if $\mathbf X$ and $\mathbf Y$ are centered predictor ($n \times p$) and response ($n\times q$) ...
amoeba's user avatar
  • 105k
23 votes
Accepted

How to decide between PCA and logistic regression?

The key difference between two approches PCA will NOT consider the response variable but only the variance of the independent variables. Logistic Regression will consider how each independent ...
Haitao Du's user avatar
  • 36.9k
23 votes

Arrows of underlying variables in PCA biplot in R

Consider upvoting @amoeba's and @ttnphns' post. Thank you both for your help and ideas. The following relies on the Iris dataset in R, and specifically the first three variables (columns): ...
Antoni Parellada's user avatar
21 votes

What're the differences between PCA and autoencoder?

The currently accepted answer by @bayerj states that the weights of a linear autoencoder span the same subspace as the principal components found by PCA, but they are not the same vectors. In ...
DeltaIV's user avatar
  • 18k
21 votes

Are there cases where PCA is more suitable than t-SNE?

https://stats.stackexchange.com/a/249520/7828 is an excellent general answer. I'd like to focus a bit more on your problem. You apparently want to see how your samples relate with respect to your 7 ...
Has QUIT--Anony-Mousse's user avatar
21 votes

Principal Component Analysis Eliminate Noise In The Data

Principal Component Analysis (PCA) is used to a) denoise and to b) reduce dimensionality. It does not eliminate noise, but it can reduce noise. Basically an orthogonal linear transformation is used ...
Nikolas Rieble's user avatar
21 votes

Practical usefulness of PCA

One important use of PCA is in analysis of electroencephalography (EEG) data. To measure an EEG, dozens of electrodes are attached to your scalp and measure electric currents in your brain, either at ...
20 votes

PCA in numpy and sklearn produces different results

Here is a nice implementation with discussion and explanation of PCA in python. This implementation leads to the same result as the scikit PCA. This is another indicator that your PCA is wrong. <...
Nikolas Rieble's user avatar
20 votes

Can PC1 explain more than 90% of variance?

Certainly that can happen. Below is a simulated example in R with just two original variables. As long as they are strongly enough correlated, they can very well be summarized by the projection to a ...
Stephan Kolassa's user avatar
19 votes

What is the intuition behind SVD?

Let $A$ be a real $m \times n$ matrix. I'll assume that $m \geq n$ for simplicity. It's natural to ask in which direction $v$ does $A$ have the most impact (or the most explosiveness, or the most ...
littleO's user avatar
  • 590

Only top scored, non community-wiki answers of a minimum length are eligible