1
$\begingroup$

I have a data matrix with N entries and n features.

I wanted to find which features are the most important in explaining the data.

So, I started with PCA, but PCA components are linear combination of the original features and a lot of features have non-zero weights.

So, I tried Sparse PCA and when I get the components: 

from sklearn.decomposition import SparsePCA

pca = SparsePCA(n_components=2)
x_reduced = pca.fit_transform(x)

print(pca.components_)

[[  0.00000000e+00  -5.51883148e-01   0.00000000e+00  -4.48853814e+03
    0.00000000e+00   0.00000000e+00   0.00000000e+00  -7.00452241e+05
   -1.32201606e+04  -9.07503173e+01  -4.13760727e+01  -7.59535971e+00
   -3.73699760e+02   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00  -2.57843033e+04  -6.00876332e+04  -2.83214559e+00
   -5.71307312e+00  -8.64860741e+03   0.00000000e+00  -1.10025048e+02
   -1.31741530e-01   0.00000000e+00  -3.35139193e+02  -3.19273051e+00
    0.00000000e+00  -1.55725943e+04   0.00000000e+00  -9.43423387e+01
    0.00000000e+00  -1.63657513e+02   0.00000000e+00  -3.02014818e+04
   -3.72207930e+02  -1.00095772e+04]
 [ -1.06618508e+01  -4.98409303e+00   0.00000000e+00   2.39169477e+01
   -2.69867091e+00   0.00000000e+00   0.00000000e+00   1.21823612e+04
   -2.46473015e+05  -1.32679881e+03  -7.32920896e+02  -1.43417641e+02
   -3.96088636e-02  -1.45871975e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
    0.00000000e+00  -1.25468810e+02  -6.56146675e+02  -7.98195673e+01
   -1.67858364e+01  -1.30710862e+05  -3.69910267e+01  -3.59490984e+02
   -6.82808350e+01   0.00000000e+00  -1.01658352e+03  -1.36297880e+02
   -4.01496541e-02  -2.58388818e+05  -1.82709863e+00  -2.60664817e+02
    0.00000000e+00  -6.50151510e+02   0.00000000e+00   1.53823591e+03
    8.10467754e+00  -8.98170228e+03]]

Now, my question is:

  1. Is the magnitude of a dimension's weight in the Principal component a measure of its importance for the overall data?

Further, I don't think the sign of the weight matters but any comments are welcome.

  1. Is there any way I can find how much an individual feature is responsible for explaining the variance? I guess this is just a corollary of 1. I can go through top components and sum their overall importance in terms of percentage.

  2. Any methods I can use to infer the importance of the dimensions in the feature space, knowing the feature's weights in the principal components' space?

Many Thanks for your help!!

Clarification: What do I mean by importance?

Like in x = a + b, we can say that a has a/(a+b) importance in the constituting the value of x.

Since PC is a linear combination of features, can we say that a feature has |Wi|/Sum(|Wi|) importance in constituting the value of PC (Wi = coefficients of the features in the PC) and hence, in explaining the amount of variance the PC does?

Improve this question
$\endgroup$
6
  • $\begingroup$ What's your goal? It seems you have an unusual application of PCA. Also, please, use the standard terms. It's not clear what you call a weight here. For instance, there's a score (of PC) and the coefficient (weight) of the input series in PC $\endgroup$
    – Aksakal
    Commented Aug 31, 2017 at 14:48
  • 1
    $\begingroup$ What exactly do you mean by importance? Also, did you try evaluating other hyperparameters alpha, ridge_alpha? You can control sparsity and reconstruction error using these parameters. $\endgroup$
    – Jakub Bartczuk
    Commented Aug 31, 2017 at 16:26
  • $\begingroup$ @Aksakal I guess score = variance explained by the PC and coefficient = weights of the original features in the PC (i.e. the linear combination). My goal is then to ascertain what is the relative importance of a particular feature in the first PC, in the second PC and so on... $\endgroup$
    – Rafael
    Commented Sep 1, 2017 at 4:37
  • 1
    $\begingroup$ @JakubBartczuk Thanks for the tip, I hadn't modified the hyperparams. By importance, I mean: Like in x = a + b, we can say that a has a/(a+b) importance in the constituting the value of x. Since PC is a linear combination of features, can we say that a feature has |Wi|/Sum(|Wi|) importance in constituting the value of PC and hence, in explaining the amount of variance the PC does. $\endgroup$
    – Rafael
    Commented Sep 1, 2017 at 4:42
  • $\begingroup$ If your refering to linear regression above, you can't really say that. You can use the word importance to mean that (by definition), but it doesn't make it a sensible concept. $\endgroup$
    – Matthew Drury
    Commented Sep 1, 2017 at 4:55

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.