Questions tagged [pca]
Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.
535
questions
16
votes
1
answer
9k
views
How does "Fundamental Theorem of Factor Analysis" apply to PCA, or how are PCA loadings defined?
I'm currently going through a slide set I have for "factor analysis" (PCA as far as I can tell).
In it, the "fundamental theorem of factor analysis" is derived which claims that the correlation ...
- 583
15
votes
2
answers
21k
views
How to perform cross-validation for PCA to determine the number of principal components?
I'm trying to write my own function for principal component analysis, PCA (of course there's a lot already written but I'm just interested in implementing stuff by myself). The main problem I ...
- 632
42
votes
1
answer
9k
views
How to determine significant principal components using bootstrapping or Monte Carlo approach?
I am interested in determining the number of significant patterns coming out of a Principal Component Analysis (PCA) or Empirical Orthogonal Function (EOF) Analysis. I am particularly interested in ...
- 3,682
27
votes
2
answers
33k
views
Reversing PCA back to the original variables [duplicate]
I have a set of data that has $n$ samples described by $m$ variables. I do a PCA to reduce it to just 2 dimensions so I can make a nice 2D plot of the data. I understand that the $x,y$ coordinates (i....
- 373
26
votes
1
answer
4k
views
Properties of PCA for dependent observations
We usually use PCA as a dimensionality reduction technique for data where cases are assumed to be i.i.d.
Question: What are the typical nuances in applying PCA for dependent, non-i.i.d. data? What ...
- 62.1k
23
votes
1
answer
14k
views
What is the "horseshoe effect" and/or the "arch effect" in PCA / correspondence analysis?
There are many techniques in ecological statistics for exploratory data analysis of multidimensional data. These are called 'ordination' techniques. Many are the same or closely related to common ...
23
votes
6
answers
20k
views
PCA of non-Gaussian data
I have a couple of quick questions about PCA:
Does the PCA assume that the dataset is Gaussian?
What happens when I apply a PCA to inherently non-linear data?
Given a dataset, the process is to ...
- 1,180
21
votes
1
answer
25k
views
Is there any required amount of variance captured by PCA in order to do later analyses?
I have a dataset with 11 variables and PCA (orthogonal) was done to reduce the data. Deciding on the number of components to keep it was evident for me from my knowledge about the subject and the ...
- 1,085
20
votes
1
answer
5k
views
What is the connection between partial least squares, reduced rank regression, and principal component regression?
Are reduced rank regression and principal component regression just special cases of partial least squares?
This tutorial (Page 6, "Comparison of Objectives") states that when we do partial least ...
- 415
20
votes
2
answers
36k
views
Why are principal components in PCA (eigenvectors of the covariance matrix) mutually orthogonal? [duplicate]
Why are principal components in PCA mutually orthogonal? I know that PCA can be calculated by eig(cov(X)), where X is centered. ...
- 331
19
votes
2
answers
11k
views
Can I do a PCA on repeated measures for data reduction?
I have 3 trials each on 87 animals in each of 2 contexts (some missing data; no missing data = 64 animals). Within a context, I have many specific measures (time to enter, number of times returning ...
- 211
11
votes
4
answers
8k
views
Is it acceptable to reverse a sign of a principal component score? [duplicate]
I have two datasets from similar psycholinguistic experiments. In both of them, information about the participant's reading and spelling ability was collected, then converted into standardized scores ...
- 486
8
votes
1
answer
11k
views
How to project a new vector onto the PC space using kernel PCA?
Let $X_{N \times d}$ be the data matrix, where $N$ is the number of samples and $d$ the size of the features space.
Using kernel PCA (kPCA), one first computes a kernel matrix $K_{N \times N}$, and ...
- 545
7
votes
1
answer
4k
views
Volume of the 95% confidence ellipsoid
I'm dealing with 3D data that are the trajectory of a point over time. I would like to have an indication of how much it is "spread" in space and I thought about using the volume of the 95% confidence ...
- 301
7
votes
1
answer
2k
views
Can averaging all the variables be seen as a crude form of PCA?
This question just occurred to me out of the blue. PCA is a way to reduce dimensions. Another way that is often (perhaps too often) used is to take mean values of two or more variables. This is done a ...
- 1,117
4
votes
1
answer
5k
views
Categorical Principal Component Analysis - using Count, Continuous, Ordinal variables together
I have some variables and I want to reduce their number for further analysis. I initially thought of combining them using factor analysis. But since the variables are of all kinds (rating, count, ...
- 53
4
votes
1
answer
16k
views
What's the relationship between initial eigenvalues and sums of squared loadings in factor analysis?
On the one hand I read in a comment here that:
You can't speak of "eigenvalues" after rotation, even orthogonal
rotation. Perhaps you mean sum of squared loadings for a principal
component, ...
4
votes
1
answer
3k
views
How to compute PCA scores from eigendecomposition of the covariance matrix?
Given a data matrix $\mathbf X$ of $12 \times 7$ size with samples in rows and variables in columns, I have calculated centered data $\mathbf X_c$ by subtracting column means, and then computed ...
- 775
3
votes
1
answer
5k
views
Is it valid to perform PCA if Kaiser-Meyer-Olkin (KMO) index is very low?
I have a dataset that contains data from $307$ subjects and nine variables for each subject. I would like to run a PCA. My problem is that I get a Kaiser-Meyer-Olkin (KMO) value of $0.06$.
Can it be ...
- 33
32
votes
1
answer
9k
views
PCA, LDA, CCA, and PLS
How are PCA, LDA, CCA, and PLS related? They all seem "spectral" and linear algebraic and very well understood (say 50+ years of theory built around them). They are used for very different things (PCA ...
- 3,205
30
votes
4
answers
9k
views
Is PCA always recommended?
I was wondering if PCA can be always applied for dimensionality reduction before a classification or regression problem. My intuition tells me that the answer is no.
If we perform PCA then we ...
- 702
29
votes
3
answers
38k
views
PCA when the dimensionality is greater than the number of samples
I've come across a scenario where I have 10 signals/person for 10 people (so 100 samples) containing 14000 data points (dimensions) that I need to pass to a classifier. I would like to reduce the ...
- 345
28
votes
2
answers
17k
views
Is there any advantage of SVD over PCA?
I know how to calculate PCA and SVD mathematically, and I know that both can be applied to Linear Least Squares regression.
The main advantage of SVD mathematically seems to be that it can be applied ...
- 1,743
26
votes
1
answer
29k
views
Not normalizing data before PCA gives better explained variance ratio
I normalized my dataset then ran 3 component PCA to get small explained variance ratios ([0.50, 0.1, 0.05]).
When I didn't normalize but whitened my dataset then ran 3 component PCA, I got high ...
user46925
25
votes
2
answers
29k
views
Why log-transforming the data before performing principal component analysis?
Im following a tutorial here: http://www.r-bloggers.com/computing-and-visualizing-pca-in-r/ to gain a better understanding of PCA.
The tutorial uses the Iris dataset and applies a log transform prior ...
20
votes
1
answer
3k
views
Geometric understanding of PCA in the subject (dual) space
I am trying to get an intuitive understanding of how principal component analysis (PCA) works in subject (dual) space.
Consider 2D dataset with two variables, $x_1$ and $x_2$, and $n$ data points (...
- 100k
18
votes
1
answer
11k
views
Supervised dimensionality reduction
I have a data set consisting of 15K labeled samples (of 10 groups). I want to apply dimensionality reduction into 2 dimensions, that would take into consideration the knowledge of the labels.
When I ...
- 799
17
votes
2
answers
8k
views
When do we combine dimensionality reduction with clustering?
I am trying to perform document-level clustering. I constructed the term-document frequency matrix and I am trying to cluster these high dimensional vectors using k-means. Instead of directly ...
- 4,402
13
votes
3
answers
4k
views
The first principal component does not separate classes, but other PCs do; how is that possible?
I ran PCA on 17 quantitative variables in order to obtain a smaller set of variables, that is principal components, to be used in supervised machine learning for classifying instances into two classes....
- 155
13
votes
2
answers
19k
views
Arrows of underlying variables in PCA biplot in R
At the risk of making the question software-specific, and with the excuse of its ubiquity and idiosyncrasies, I want to ask about the function biplot() in R, and, ...
- 25.3k
13
votes
2
answers
7k
views
Which independent variables are most important in predicting the response variable? [duplicate]
I'm a biologist, and I have a large dataset that I'm trying to analyze. Here are the variables I'm working with:
levels of 211 different metabolites in 16 different blood samples (predictor variables)...
- 131
11
votes
3
answers
14k
views
On the use of oblique rotation after PCA
Several statistical packages, such as SAS, SPSS, and R, allow you to perform some kind of factor rotation following a PCA.
Why is a rotation necessary after a PCA?
Why would you apply an oblique ...
- 1,241
10
votes
2
answers
8k
views
Is it possible to use kernel PCA for feature selection?
Is it possible to use kernel principal component analysis (kPCA) for Latent Semantic Indexing (LSI) in the same way as PCA is used?
I perform LSI in R using the ...
- 101
9
votes
1
answer
10k
views
Should PCA be performed before I do classification?
I have got a problem about doing a classification. I have got around 50 datasets. Each of them has 15 features.
I am trying to use these features to classify the 50 datasets to either 'Good' or 'Bad'....
- 1,709
9
votes
1
answer
11k
views
Proportion of explained variance in PCA and LDA
I have some basic questions regarding PCA (principal component analysis) and LDA (linear discriminant analysis):
In PCA there is a way to calculate the proportion of variance explained. Is it also ...
- 101
8
votes
1
answer
4k
views
Does the first principal component differ from simply computing the mean of all variables?
I was just wondering if the first principal component, while I am trying to find it for a dataset of 18 variables, is different from simply adding all variables and finding the mean? I.e. to compute ...
- 107
6
votes
2
answers
2k
views
Is dimensionality reduction almost always useful for classification?
Is singular value decomposition almost always useful in practice for enhancing the predicative power of a trained classification model?
E.x. A dataset for classification has 20,000 features. Run SVD ...
- 838
6
votes
1
answer
4k
views
PCA: Eigenvectors of opposite sign and not being able to compute eigenvectors with `solve` in R
I'm learning PCA in R language. I met two problems right now that I don't understand.
I am performing a PCA analysis in R on a 318×17 dataset using some custom code. I take eigen function in R to ...
- 73
6
votes
2
answers
2k
views
Choosing number of PCA components when multiple samples for each data point are available
Update: I posted an answer below describing my current attempts to approach this problem.
I am facing a perennial problem of identifying significant principal components. This question has been ...
- 100k
5
votes
2
answers
650
views
Scree plot: $m$ vs $m-1$ components/factors
@ttnphns comments here that
there exist two expositions of the Cattell scree-plot rule: If the "elbow" is the m-th eigenvalue, (1) choose to extract m components; or (2) choose to extract m-...
5
votes
2
answers
3k
views
Strange results of varimax rotation of principal component analysis in Stata: rotated components are all zeros and ones
This is my initial output of Principal Component Analysis (PCA) using Stata and correlation matrix (because different scales and measurement units of inputs):
<...
- 3,911
4
votes
1
answer
11k
views
What are the units in this PCA biplot? [duplicate]
This is a plot of my data
These are the values:
...
- 151
4
votes
2
answers
39k
views
What is the difference between PCA and PAF method in factor analysis?
What is the difference between principal component analyses (PCA) and principal axis factoring (PAF)?
Also, I understand the difference between varimax and oblimin rotations, but is that the same as ...
- 49
2
votes
3
answers
17k
views
Use of PCA analysis to select variables for a regression analysis [duplicate]
I have too many environmental variables to use in a multiple regression analysis. If I use all the variables the models are just too complex. The use of the PCA axes in the regression analysis was ...
- 21
69
votes
4
answers
54k
views
Are there cases where PCA is more suitable than t-SNE?
I want to see how 7 measures of text correction behaviour (time spent correcting the text, number of keystrokes, etc.) relate to each other. The measures are correlated. I ran a PCA to see how the ...
- 887
43
votes
3
answers
27k
views
Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?
In a recent assignment, we were told to use PCA on the MNIST digits to reduce the dimensions from 64 (8 x 8 images) to 2. We then had to cluster the digits using a Gaussian Mixture Model. PCA using ...
- 603
36
votes
3
answers
16k
views
Building an autoencoder in Tensorflow to surpass PCA
Hinton and Salakhutdinov in Reducing the Dimensionality of Data with Neural Networks, Science 2006 proposed a non-linear PCA through the use of a deep autoencoder. I have tried to build and train a ...
- 3,129
27
votes
4
answers
24k
views
What's wrong with t-SNE vs PCA for dimensional reduction using R?
I have a matrix of 336x256 floating point numbers (336 bacterial genomes (columns) x 256 normalized tetranucleotide frequencies (rows), e.g. every column adds up to 1).
I get nice results when I run ...
- 271
23
votes
2
answers
2k
views
The limit of "unit-variance" ridge regression estimator when $\lambda\to\infty$
Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
- 100k
21
votes
2
answers
20k
views
How to interpret PCA on time-series data?
I am trying to understand the use of PCA in a recent journal article titled "Mapping brain activity at scale with cluster computing" Freeman et al., 2014 (free pdf available on the lab website). They ...
- 367