241
votes
Accepted
How to reverse PCA and reconstruct original variables from several principal components?
PCA computes eigenvectors of the covariance matrix ("principal axes") and sorts them by their eigenvalues (amount of explained variance). The centered data can then be projected onto these principal ...
- 106k
105
votes
Are there cases where PCA is more suitable than t-SNE?
$t$-SNE is a great piece of Machine Learning but one can find many reasons to use PCA instead of it. Of the top of my head, I will mention five. As most other computational methodologies in use, $t$-...
- 45.2k
60
votes
What are the differences between Factor Analysis and Principal Component Analysis?
A basic, yet a kind of painstaking, explanation of PCA vs Factor analysis with the help of scatterplots, in logical steps. (I thank @amoeba who, in his comment to the question, has encouraged me to ...
- 58.4k
50
votes
Building an autoencoder in Tensorflow to surpass PCA
Here is the key figure from the 2006 Science paper by Hinton and Salakhutdinov:
It shows dimensionality reduction of the MNIST dataset ($28\times 28$ black and white images of single digits) from the ...
- 106k
49
votes
Accepted
Linearity of PCA
When we say that PCA is a linear method, we refer to the dimensionality reducing mapping $f:\mathbf x\mapsto \mathbf z$ from high-dimensional space $\mathbb R^p$ to a lower-dimensional space $\mathbb ...
- 106k
48
votes
Accepted
Calculating PCA variance explained
Yes, that's correct. summary.prcomp brings that information as well:
...
- 19.4k
43
votes
Relationship between SVD and PCA. How to use SVD to perform PCA?
I wrote a Python & Numpy snippet that accompanies @amoeba's answer and I leave it here in case it is useful for someone. The comments are mostly taken from @amoeba's answer.
...
43
votes
Accepted
Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?
The main reason that $t$-SNE is not used in classification models is that it does not learn a function from the original space to the new (lower) dimensional one. As such, when we would try to use our ...
- 45.2k
40
votes
Accepted
Why is PCA sensitive to outliers?
One of the reasons is that PCA can be thought as low-rank decomposition of the data that minimizes the sum of $L_2$ norms of the residuals of the decomposition. I.e. if $Y$ is your data ($m$ vectors ...
- 837
39
votes
Accepted
Is PCA always recommended?
Blindly using PCA is a recipe for disaster. (As an aside, automatically applying any method is not a good idea, because what works in one context is not guaranteed to work in another. We can formalize ...
- 92.6k
34
votes
Accepted
PCA in numpy and sklearn produces different results
The difference is because decomposition.PCA does not standardize your variables before doing PCA, whereas in your manual computation you call ...
- 106k
33
votes
Relationship between SVD and PCA. How to use SVD to perform PCA?
Let me start with PCA. Suppose that you have $n$ data points comprised of $d$ numbers (or dimensions) each. If you center this data (subtract the mean data point $\mu$ from each data vector $x_i$) you ...
- 511
28
votes
Accepted
Is PCA optimization convex?
No, the usual formulations of PCA are not convex problems. But they can be transformed into a convex optimization problem.
The insight and the fun of this is following and visualizing the sequence of ...
- 330k
27
votes
Why do we use PCA to speed up learning algorithms when we could just reduce the number of features?
Let's say you initially have $p$ features but this is too many so you want to actually fit your model on $d < p$ features. You could choose $d$ of your features and drop the rest. If $X$ is our ...
- 20.6k
27
votes
Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?
amoeba already gave a good answer in the comments, but if you want a formal argument, here it goes.
The singular value decomposition of a matrix $A$ is $A=U\Sigma V^T$, where the columns of $V$ are ...
- 2,249
25
votes
Accepted
Difference between scikit-learn implementations of PCA and TruncatedSVD
PCA and TruncatedSVD scikit-learn implementations seem to be exactly the same algorithm.
No: PCA is (truncated) SVD on centered data (by per-feature mean substraction). If the data is already ...
- 3,769
24
votes
Accepted
Does PCA preserve linear separability for every linearly separable set?
No, it may be that the discriminative information is in the direction of a principal component that explains a relatively small amount of the total variance, and hence gets discarded.
Consider a two-...
- 55.1k
23
votes
Accepted
How to decide between PCA and logistic regression?
The key difference between two approches
PCA will NOT consider the response variable but only the variance of the independent variables.
Logistic Regression will consider how each independent ...
- 37.2k
23
votes
Arrows of underlying variables in PCA biplot in R
Consider upvoting @amoeba's and @ttnphns' post. Thank you both for your help and ideas.
The following relies on the Iris dataset in R, and specifically the first three variables (columns): ...
- 26.7k
22
votes
What is the intuition behind SVD?
Let $A$ be a real $m \times n$ matrix. I'll assume that $m \geq n$ for simplicity. It's natural to ask in which direction $v$ does $A$ have the most impact (or the most explosiveness, or the most ...
- 620
22
votes
Are there cases where PCA is more suitable than t-SNE?
https://stats.stackexchange.com/a/249520/7828
is an excellent general answer.
I'd like to focus a bit more on your problem. You apparently want to see how your samples relate with respect to your 7 ...
- 42.8k
21
votes
What're the differences between PCA and autoencoder?
The currently accepted answer by @bayerj states that the weights of a linear autoencoder span the same subspace as the principal components found by PCA, but they are not the same vectors. In ...
- 18.3k
21
votes
Principal Component Analysis Eliminate Noise In The Data
Principal Component Analysis (PCA) is used to a) denoise and to b) reduce dimensionality.
It does not eliminate noise, but it can reduce noise.
Basically an orthogonal linear transformation is used ...
- 3,481
21
votes
Accepted
Normalizing vs Scaling before PCA
Scaling (what I would call centering and scaling) is very important for PCA because of the way that the principal components are calculated. PCA is solved via the Singular Value Decomposition, which ...
- 4,650
21
votes
Practical usefulness of PCA
One important use of PCA is in analysis of electroencephalography (EEG) data. To measure an EEG, dozens of electrodes are attached to your scalp and measure electric currents in your brain, either at ...
20
votes
PCA in numpy and sklearn produces different results
Here is a nice implementation with discussion and explanation of PCA in python. This implementation leads to the same result as the scikit PCA. This is another indicator that your PCA is wrong.
<...
- 3,481
20
votes
Accepted
The limit of "unit-variance" ridge regression estimator when $\lambda\to\infty$
#A geometrical interpretation
The estimator described in the question is the Lagrange multiplier equivalent of the following optimization problem:
$$\text{minimize $f(\beta)$ subject to $g(\beta) \leq ...
- 82.8k
20
votes
Can PC1 explain more than 90% of variance?
Certainly that can happen. Below is a simulated example in R with just two original variables. As long as they are strongly enough correlated, they can very well be summarized by the projection to a ...
- 129k
19
votes
Accepted
How does PCA behave when there is no correlation in the dataset?
If you have no observed correlation, then your covariance matrix is diagonal, and the PCA diagonalizes a matrix that is already diagonal (so it does nothing).
If you have no population correlation but ...
- 65.7k
18
votes
Accepted
Understanding this PCA plot of ice cream sales vs temperature
I know that PCA objective is to reduce dimensionality
This is often what people assume, but in fact PCA is just a representation of your data onto an orthogonal basis. This basis still has the same ...
- 4,787
Only top scored, non community-wiki answers of a minimum length are eligible