# Questions tagged [pca]

Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.

535 questions
Filter by
Sorted by
Tagged with
9k views

### How does "Fundamental Theorem of Factor Analysis" apply to PCA, or how are PCA loadings defined?

I'm currently going through a slide set I have for "factor analysis" (PCA as far as I can tell). In it, the "fundamental theorem of factor analysis" is derived which claims that the correlation ...
21k views

### How to perform cross-validation for PCA to determine the number of principal components?

I'm trying to write my own function for principal component analysis, PCA (of course there's a lot already written but I'm just interested in implementing stuff by myself). The main problem I ...
9k views

### How to determine significant principal components using bootstrapping or Monte Carlo approach?

I am interested in determining the number of significant patterns coming out of a Principal Component Analysis (PCA) or Empirical Orthogonal Function (EOF) Analysis. I am particularly interested in ...
33k views

### Reversing PCA back to the original variables [duplicate]

I have a set of data that has $n$ samples described by $m$ variables. I do a PCA to reduce it to just 2 dimensions so I can make a nice 2D plot of the data. I understand that the $x,y$ coordinates (i....
4k views

### Properties of PCA for dependent observations

We usually use PCA as a dimensionality reduction technique for data where cases are assumed to be i.i.d. Question: What are the typical nuances in applying PCA for dependent, non-i.i.d. data? What ...
14k views

### What is the "horseshoe effect" and/or the "arch effect" in PCA / correspondence analysis?

There are many techniques in ecological statistics for exploratory data analysis of multidimensional data. These are called 'ordination' techniques. Many are the same or closely related to common ...
20k views

### PCA of non-Gaussian data

I have a couple of quick questions about PCA: Does the PCA assume that the dataset is Gaussian? What happens when I apply a PCA to inherently non-linear data? Given a dataset, the process is to ...
25k views

### Is there any required amount of variance captured by PCA in order to do later analyses?

I have a dataset with 11 variables and PCA (orthogonal) was done to reduce the data. Deciding on the number of components to keep it was evident for me from my knowledge about the subject and the ...
5k views

### What is the connection between partial least squares, reduced rank regression, and principal component regression?

Are reduced rank regression and principal component regression just special cases of partial least squares? This tutorial (Page 6, "Comparison of Objectives") states that when we do partial least ...
36k views

### Why are principal components in PCA (eigenvectors of the covariance matrix) mutually orthogonal? [duplicate]

Why are principal components in PCA mutually orthogonal? I know that PCA can be calculated by eig(cov(X)), where X is centered. ...
11k views

### Can I do a PCA on repeated measures for data reduction?

I have 3 trials each on 87 animals in each of 2 contexts (some missing data; no missing data = 64 animals). Within a context, I have many specific measures (time to enter, number of times returning ...
8k views

### Is it acceptable to reverse a sign of a principal component score? [duplicate]

I have two datasets from similar psycholinguistic experiments. In both of them, information about the participant's reading and spelling ability was collected, then converted into standardized scores ...
11k views

### How to project a new vector onto the PC space using kernel PCA?

Let $X_{N \times d}$ be the data matrix, where $N$ is the number of samples and $d$ the size of the features space. Using kernel PCA (kPCA), one first computes a kernel matrix $K_{N \times N}$, and ...
4k views

### Volume of the 95% confidence ellipsoid

I'm dealing with 3D data that are the trajectory of a point over time. I would like to have an indication of how much it is "spread" in space and I thought about using the volume of the 95% confidence ...
2k views

### Can averaging all the variables be seen as a crude form of PCA?

This question just occurred to me out of the blue. PCA is a way to reduce dimensions. Another way that is often (perhaps too often) used is to take mean values of two or more variables. This is done a ...
5k views

### Categorical Principal Component Analysis - using Count, Continuous, Ordinal variables together

I have some variables and I want to reduce their number for further analysis. I initially thought of combining them using factor analysis. But since the variables are of all kinds (rating, count, ...
16k views

### What's the relationship between initial eigenvalues and sums of squared loadings in factor analysis?

On the one hand I read in a comment here that: You can't speak of "eigenvalues" after rotation, even orthogonal rotation. Perhaps you mean sum of squared loadings for a principal component, ...
3k views

### How to compute PCA scores from eigendecomposition of the covariance matrix?

Given a data matrix $\mathbf X$ of $12 \times 7$ size with samples in rows and variables in columns, I have calculated centered data $\mathbf X_c$ by subtracting column means, and then computed ...
5k views

### Is it valid to perform PCA if Kaiser-Meyer-Olkin (KMO) index is very low?

I have a dataset that contains data from $307$ subjects and nine variables for each subject. I would like to run a PCA. My problem is that I get a Kaiser-Meyer-Olkin (KMO) value of $0.06$. Can it be ...
9k views

### PCA, LDA, CCA, and PLS

How are PCA, LDA, CCA, and PLS related? They all seem "spectral" and linear algebraic and very well understood (say 50+ years of theory built around them). They are used for very different things (PCA ...
9k views

### Is PCA always recommended?

I was wondering if PCA can be always applied for dimensionality reduction before a classification or regression problem. My intuition tells me that the answer is no. If we perform PCA then we ...
38k views

### PCA when the dimensionality is greater than the number of samples

I've come across a scenario where I have 10 signals/person for 10 people (so 100 samples) containing 14000 data points (dimensions) that I need to pass to a classifier. I would like to reduce the ...
17k views

### Is there any advantage of SVD over PCA?

I know how to calculate PCA and SVD mathematically, and I know that both can be applied to Linear Least Squares regression. The main advantage of SVD mathematically seems to be that it can be applied ...
29k views

### Not normalizing data before PCA gives better explained variance ratio

I normalized my dataset then ran 3 component PCA to get small explained variance ratios ([0.50, 0.1, 0.05]). When I didn't normalize but whitened my dataset then ran 3 component PCA, I got high ... 29k views

### Why log-transforming the data before performing principal component analysis?

Im following a tutorial here: http://www.r-bloggers.com/computing-and-visualizing-pca-in-r/ to gain a better understanding of PCA. The tutorial uses the Iris dataset and applies a log transform prior ...
3k views

### Geometric understanding of PCA in the subject (dual) space

I am trying to get an intuitive understanding of how principal component analysis (PCA) works in subject (dual) space. Consider 2D dataset with two variables, $x_1$ and $x_2$, and $n$ data points (...
11k views

### Supervised dimensionality reduction

I have a data set consisting of 15K labeled samples (of 10 groups). I want to apply dimensionality reduction into 2 dimensions, that would take into consideration the knowledge of the labels. When I ...
8k views

### When do we combine dimensionality reduction with clustering?

I am trying to perform document-level clustering. I constructed the term-document frequency matrix and I am trying to cluster these high dimensional vectors using k-means. Instead of directly ...
4k views

### The first principal component does not separate classes, but other PCs do; how is that possible?

I ran PCA on 17 quantitative variables in order to obtain a smaller set of variables, that is principal components, to be used in supervised machine learning for classifying instances into two classes....
19k views

### Arrows of underlying variables in PCA biplot in R

At the risk of making the question software-specific, and with the excuse of its ubiquity and idiosyncrasies, I want to ask about the function biplot() in R, and, ...
7k views

### Which independent variables are most important in predicting the response variable? [duplicate]

I'm a biologist, and I have a large dataset that I'm trying to analyze. Here are the variables I'm working with: levels of 211 different metabolites in 16 different blood samples (predictor variables)...
14k views

### On the use of oblique rotation after PCA

Several statistical packages, such as SAS, SPSS, and R, allow you to perform some kind of factor rotation following a PCA. Why is a rotation necessary after a PCA? Why would you apply an oblique ...
8k views

### Is it possible to use kernel PCA for feature selection?

Is it possible to use kernel principal component analysis (kPCA) for Latent Semantic Indexing (LSI) in the same way as PCA is used? I perform LSI in R using the ...
10k views

### Should PCA be performed before I do classification?

I have got a problem about doing a classification. I have got around 50 datasets. Each of them has 15 features. I am trying to use these features to classify the 50 datasets to either 'Good' or 'Bad'....
11k views

### Proportion of explained variance in PCA and LDA

I have some basic questions regarding PCA (principal component analysis) and LDA (linear discriminant analysis): In PCA there is a way to calculate the proportion of variance explained. Is it also ...
4k views

### Does the first principal component differ from simply computing the mean of all variables?

I was just wondering if the first principal component, while I am trying to find it for a dataset of 18 variables, is different from simply adding all variables and finding the mean? I.e. to compute ...
2k views

### Is dimensionality reduction almost always useful for classification?

Is singular value decomposition almost always useful in practice for enhancing the predicative power of a trained classification model? E.x. A dataset for classification has 20,000 features. Run SVD ...
4k views

### PCA: Eigenvectors of opposite sign and not being able to compute eigenvectors with solve in R

I'm learning PCA in R language. I met two problems right now that I don't understand. I am performing a PCA analysis in R on a 318×17 dataset using some custom code. I take eigen function in R to ...
2k views

### Choosing number of PCA components when multiple samples for each data point are available

Update: I posted an answer below describing my current attempts to approach this problem. I am facing a perennial problem of identifying significant principal components. This question has been ...
650 views

### Scree plot: $m$ vs $m-1$ components/factors

@ttnphns comments here that there exist two expositions of the Cattell scree-plot rule: If the "elbow" is the m-th eigenvalue, (1) choose to extract m components; or (2) choose to extract m-...
3k views

### Strange results of varimax rotation of principal component analysis in Stata: rotated components are all zeros and ones

This is my initial output of Principal Component Analysis (PCA) using Stata and correlation matrix (because different scales and measurement units of inputs): <...
11k views

### What are the units in this PCA biplot? [duplicate]

This is a plot of my data These are the values: ...
39k views

### What is the difference between PCA and PAF method in factor analysis?

What is the difference between principal component analyses (PCA) and principal axis factoring (PAF)? Also, I understand the difference between varimax and oblimin rotations, but is that the same as ...
17k views

### Use of PCA analysis to select variables for a regression analysis [duplicate]

I have too many environmental variables to use in a multiple regression analysis. If I use all the variables the models are just too complex. The use of the PCA axes in the regression analysis was ...
54k views

### Are there cases where PCA is more suitable than t-SNE?

I want to see how 7 measures of text correction behaviour (time spent correcting the text, number of keystrokes, etc.) relate to each other. The measures are correlated. I ran a PCA to see how the ...
27k views

### Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?

In a recent assignment, we were told to use PCA on the MNIST digits to reduce the dimensions from 64 (8 x 8 images) to 2. We then had to cluster the digits using a Gaussian Mixture Model. PCA using ...
16k views

### Building an autoencoder in Tensorflow to surpass PCA

Hinton and Salakhutdinov in Reducing the Dimensionality of Data with Neural Networks, Science 2006 proposed a non-linear PCA through the use of a deep autoencoder. I have tried to build and train a ...
24k views

### What's wrong with t-SNE vs PCA for dimensional reduction using R?

I have a matrix of 336x256 floating point numbers (336 bacterial genomes (columns) x 256 normalized tetranucleotide frequencies (rows), e.g. every column adds up to 1). I get nice results when I run ...
### The limit of "unit-variance" ridge regression estimator when $\lambda\to\infty$
Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...