My coworker and I were discussing this. He insists that PCA only makes sense just when you know at least one variable is linear combination of the rest but I think it can be applied whenever there are variables that are dependent on the rest, not necessarily through a linear relationship.
$\begingroup$
$\endgroup$
7
-
3$\begingroup$ This needs more context to be answerable. Are you inquiring about interpreting PCA or else about employing some aspect of its results somehow in some model? Please elaborate. BTW, PCA is the key part of many--perhaps most--nonlinear dimensionality reduction techniques, because it provides a first order local approximation to a manifold. $\endgroup$– whuber ♦Aug 3 at 23:05
-
$\begingroup$ Certainly you can compute PCA on a numerical data matrix. But applying PCA will involve interpretation beyond just the math. What are you trying to learn from PCA? $\endgroup$– GalenAug 3 at 23:12
-
$\begingroup$ I try to do dimensionality reduction because multicolineality can be messy while applying multiple linear regression. $\endgroup$– ADayWithoutRainAug 4 at 1:35
-
1$\begingroup$ @ADayWithoutRain Multicollinearity can lead to larger standard errors of the linear coefficients (look up variance inflation), which isn't necessarily a problem depending on what you want to do with the linear regression. $\endgroup$– GalenAug 4 at 2:19
-
$\begingroup$ If you really must decorrelate the predictors in preparation for linear regression, I recommend redundancy analysis as a better alternative to PCA. Although I would still question why you're doing linear regression to understand whether multicollinearity is a problem. Sometimes it is a symptom of confounding variables, in which case you should look to other methods such as structural equation modelling. $\endgroup$– GalenAug 4 at 2:21
|
Show 2 more comments
3 Answers
$\begingroup$
$\endgroup$
6
I think neither of you is correct.
You can do a PCA on any set of data, even random noise. But, for random noise, it won't give you anything. But you talk about having it "make sense" which is sort of vague, but OK. Your friend says one variable is a linear combination of the rest. You say it need not be linear. But ... there need not even be a relationship with one variable vs. the rest, and it need not be a "linear combination". Any time some of the variables are correlated, PCA can give you sensible results. And some of the variables may not be related to the others at all.
-
1$\begingroup$ I think I may have not expressed it well. As far as I understand, you can't do any dimensionality reduction if all the variables are indepedent. However, if SOME of them are dependent then as far as I know you can using PCA, but my coworker says that depedence must be linear otherwise you can't do dimensionality reduction., $\endgroup$ Aug 4 at 1:31
-
1$\begingroup$ @ADayWithoutRain You can definitely perform dimensionality reduction even when all the variables are independent. I think that is what Peter was describing when he said you can do PCA on "random noise". $\endgroup$– GalenAug 4 at 1:38
-
1$\begingroup$ My issue is also with "dependence". This usually (n my experience and according to a couple sources) means that the dependence is exact. "Correlation" has no such implication. One benefit of PCA is finding components among variables where the correlation isn't obvious. If you do PCA on a set of variables that are not correlated at all, then it will run, but all the components will have roughly equal importance. $\endgroup$ Aug 4 at 10:39
-
$\begingroup$ @ADayWithoutRain You can perform PCA on non-linearly correlated data (or non-correlated data), but it will only "reduce" the linear part of the correlation. Try it yourself. Make data sets of e.g. x and x^2 (without noise) and PCA. You'll get relatively decent dimensionality reduction (as measured by eigenvalues/scree plot shape) if your data is exclusively in a region where the x vs x^2 plot is linear-like (e.g. x is +90 to +110), but much less of a reduction where it's significantly non-linear (e.g. x is -10 to +10). $\endgroup$– R.M.Aug 4 at 14:45
-
$\begingroup$ @PeterFlom Can you elaborate on what you mean by "the dependence is exact. "Correlation" has no such implication."? It wasn't clear to me what distinction you were making. $\endgroup$– GalenAug 4 at 15:24
$\begingroup$
$\endgroup$
I have a new case study on what is essentially nonlinear sparse principal components here. A redundancy analysis is also included, and missing data is handled by stacking multiple imputation-completed datasets.
This uses a 3 step approach. First do multiple imputation and stacking, then find optimal transformations using an unsupervised learning ACE algorithm, then use the transformed variables to do sparse principal components analysis. The latter essentially does variable clustering and PCA simultaneously to handle collinearities.
answered Aug 4 at 11:01
$\begingroup$
$\endgroup$
4
This is a response to this comment:
As far as I understand, you can't do any dimensionality reduction if all the variables are independent.
You can perform dimensionality reduction on independent variables.
Below is an example with a random data matrix $\mathbf{X}_{1000\times10}$ where we used $X_{i,j} \sim \mathcal{N}(0,1)$ as IID random variables. Then we reduce the data down to 2 dimensions using PCA.
import numpy as np
from sklearn.decomposition import PCA
# Make some fake data
np.random.seed(2018)
X = np.random.normal(0,1,size=1000*10).reshape(1000,10)
# PCA
pca = PCA(n_components=2)
Q = pca.fit_transform(X)
You'll find that PCA really has reduced those 10 variables down to 2 variables, falsifying the claim/intuition that this cannot be done.
-
2$\begingroup$ I mean, I know you can, you can also just take away 8 random features from the original dataset, but the idea of dimensionality reduction is being able to reduce dimensionality while preserving information. ¿What is the proportion of total explained variance with those components on that dataset? Surely less than 90% $\endgroup$ Aug 4 at 2:03
-
$\begingroup$ @ADayWithoutRain Retaining (useful) information is a goal we can have, but it is not a requirement. The explained variance in the above example of the first two components combined is about 22%. That's not 90%, but it isn't zero either. $\endgroup$– GalenAug 4 at 2:04
-
1$\begingroup$ Isn't that the proof that in fact the features are all indepedent? 2 from 10 features has around the 20% of the explained variance. $\endgroup$ Aug 4 at 2:11
-
1$\begingroup$ @ADayWithoutRain No, it isn't proof of independence. Of course we know they're independent because that is implemented in the simulation. But further assumptions would be required to take low explained variance in a PCA as evidence of statistical independence. $\endgroup$– GalenAug 4 at 2:15