13
$\begingroup$

I am a beginner in PCA, I am trying to apply it on a dataset I have. The features are different geometrical parameters with different units and variability, I do standardize the features matrix by subtracting the mean and dividing by the standard deviation.

For the PCA, I use the PCA() method in sklearn wich is based on the singular value decomposition (SVD). Once I fit the model I get the following results if I choose just 4 components:

Is this telling me something about my features? or there is something fundamentally wrong in my approach? Thank you!

$\endgroup$
  • 1
    $\begingroup$ I think the title of the question could be made more accurate/precise. $\endgroup$ – Richard Hardy Feb 27 at 13:31
  • $\begingroup$ The results I am obtaining show me that in order to get 99% variance I will need 12 PCs, therefore not reducing the dimensionality of the problem. what would you suggest? $\endgroup$ – cfd_aero Feb 27 at 13:35
  • $\begingroup$ well at least 90% percent to unsure that most information is not lost. $\endgroup$ – cfd_aero Feb 27 at 13:45
  • $\begingroup$ I added a new figure to explain what I want to achieve. $\endgroup$ – cfd_aero Feb 27 at 13:47
  • $\begingroup$ " do I standardize the features matrix by subtracting the mean and dividing by the standard deviation." Is that a question? $\endgroup$ – Acccumulation Feb 28 at 22:03
17
$\begingroup$

Results suggest that your features are mutually orthogonal. Accounting for total variance means accounting for both variance and covariance. Orthogonality limits covariance. Standardization equates variance across features. Put together, each feature makes a roughly equal contribution to total variance, as do your components.

$\endgroup$
  • 2
    $\begingroup$ Resolve this issue by computing a correlation matrix among the features. Are the correlations large or small, and are there any visually discernable patterns, or not? $\endgroup$ – Ed Rigdon Feb 27 at 20:12
  • 6
    $\begingroup$ @cfd_aero: It doesn't matter what effect the features have on the target. The definition of orthogonality doesn't involve a target at all. $\endgroup$ – user2357112 supports Monica Feb 28 at 2:38
  • 7
    $\begingroup$ Another important thing to note is that orthogonality does not imply independence. Some features could be redundant, but their relationship could be non-linear, and thus beyond the reach of PCA. $\endgroup$ – jpa Feb 28 at 5:59
  • 2
    $\begingroup$ @jpa that makes sense actually, are you aware of another method that could take account of possible non-linear relationships? $\endgroup$ – cfd_aero Feb 28 at 7:29
  • 3
    $\begingroup$ @cfd_aero There are many nonlinear dimensionality reduction techniques, tSNE/HSNE is a particularly popular one in the area I work in. It's mainly useful for visualizing highdimensional datasets in few dimensions. If you'd explain the goal behind reducing the number of dimensions, we could more easily indicate which technique would be appropriate. $\endgroup$ – Erik A Feb 28 at 13:09
8
$\begingroup$

PCA can be used to scale and rotate data, if we select all transformed features instead of a subset of them. My answer here gives an example of scaling and rotating the data, but without dimension reduction. How to decide between PCA and logistic regression?


Your plot shows that if we use all 12 features, the variance explained is 100%, i.e., without information loss. But if you select number of features smaller than 12, there will be information loss.

Note that in most cases, PCA can reduce dimension but at the cost of losing information. If you want to keep 99% variance, unless you have highly correlated (redundant features), PCA will not able to help.

In other words, your plot shows there are not too much redundancies in your data set.


Here are examples of both cases (with 5 features).

set.seed(0)
x1=matrix(rnorm(1000),ncol=5)

x2 = matrix(rnorm(600),ncol=3)
x2=cbind(x2,x2[,3]*runif(200)*0.01)
x2=cbind(x2,x2[,3]*runif(200)*0.01)

you may run PCA on x1 and x2 to see the difference on variance explained respect to number of features selected.

You would see for x2, 3 features will explain most of the variance, because other two features are highly correlated with the third column of x2.

$\endgroup$
  • $\begingroup$ I am not sure how your answer relates to the problem I have? could you please expand more? $\endgroup$ – cfd_aero Feb 27 at 13:36
  • $\begingroup$ thanks for the answer. yes, I a aware that the plot says that, however, what I would have expected is that the first 2 or 3 PC woud carry the majority of the variance something the PCA is useful for, but it is not happening in my case, even though I know that some of the features in the dataset have a big impact on the target and others don't. $\endgroup$ – cfd_aero Feb 27 at 13:44
  • 1
    $\begingroup$ @cfd_aero note PCA is unsupervised. Please read my linked post , "having impact on target" is not a good statement. $\endgroup$ – Haitao Du Feb 27 at 13:46
  • $\begingroup$ Thanks for the edit, it is helping me understand more the problem, but the fact is I know that some of the features are redundant, but if the PCA is unsupervised like you suggest in the post you linked, how is it possible that here I am finding out that there isn't redundancy in my data? $\endgroup$ – cfd_aero Feb 27 at 14:05
1
$\begingroup$

Well, clearly, if you were to perform PCA on a dataset, and then perform PCA on the result, you wouldn't get any benefit over just performing PCA once. So if your original dataset already had the properties of a PCA result (i.e. orthogonality), then applying PCA to it won't produce any further benefit. The more orthogonal a dataset is, the more it's already "PCA optimized", and the less further PCA helps. You should check the correlation matrix of the original dataset and see how much correlation there is between the variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.