1
$\begingroup$

While experimenting with Spark library MlLib, I questioned myself if I understood well the mechanism of PCA algorithm, because output of MlLib algorithm was not what I expected to get. so for given data of 20 features and 10 observations:

val rdd: RDD[Vector] = sc.range(0, 20).map { row =>
val values = 1.to(10).map { _ => rand.nextInt(2).toDouble }
Vectors.dense(values.toArray) }

I applied pca

val pca = new PCA(5).fit(rdd)

and then transform my initial data :

  val finalRdd = rdd.map { vector =>
    val newVector = pca.transform(vector)
    println("Before: " + vector)
    println("After : " + newVector)
    newVector }

I expected to have the rdd only with 5 features, but the values of those features should be the same, as in the initial rdd. As far as I understand pca, it keeps only the features/vectors that have the most important variance score. But instead of this I've got the rdd of 5 features, but values of those features were modified in the way I don't understand.

Before: [1.0,1.0,1.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0]

After : [1.14461218805706,0.8986586176492057,-0.580979228415323,0.09346270935831252,0.238925870721517]

Before: [1.0,1.0,0.0,1.0,0.0,0.0,0.0,1.0,1.0,0.0]

After : [1.1606629979632594,0.6110896934565042,-0.3717757311830274,-0.22954508416826663,0.18370214647002148]

Before: [0.0,1.0,0.0,1.0,1.0,1.0,0.0,1.0,1.0,0.0]

After : [1.6723486323814756,0.41082473274832476,-0.7526031076025419,-0.26291312583276166,0.11992366811728228]

Could you explain me why those features are modified? and when my understand of pca is wrong? Does the algorithm make something else with the data, and not only selection of features?

$\endgroup$
  • 1
    $\begingroup$ PCA itself is a (dimensionality reducing) feature construction, not feature selection, method. $\endgroup$ – ttnphns Jun 5 '16 at 20:39
3
$\begingroup$

PCA is not a feature selection method, but it can be used for that purpose.

PCA is simply a feature transformation, where it first chooses linear subspaces that are orthogonal to each other (eigenvectors), then projects your original data on these subspaces and returns you the projected vectors. That's why you see different values.

As for feature selection, what is usually done is to utilize the fact that the eigenvectors are ordered in decreasing variance, which is quantized by the eigenvalues. So when you select the first K (which is <N or <<N) components, you still hold the axes that explain a lot (90% or 99%) of the variance. Therefore, instead of your original N dimensions, you can continue your task with a smaller set of K dimensions (combinations of input features) without much information loss.

$\endgroup$
  • 1
    $\begingroup$ thank you very much for your explanation. One question persists, how in this case you will make the predictions if you don't know what features exactly you put in the entry of algorithm. just imagine that you have 20 variables into your linear regression and make regression on the training set with dimensionality reduction , but when I'd like to apply this model on new data, how do I know to what variables I should multiply model coefficients? $\endgroup$ – Scana Jun 5 '16 at 16:16
  • $\begingroup$ As for feature selection... I'm sorry. What you are describing in that paragraph is not feature selection. The latter implies selecting a subset of intact features, and not their linear combinations of input features. $\endgroup$ – ttnphns Jun 5 '16 at 20:43
  • $\begingroup$ That's correct. Well, it constructs new features and selects (or lets us select) among them. So not sure what to call it in this context :) $\endgroup$ – jeff Jun 5 '16 at 20:48
  • $\begingroup$ Usually term "feature selection" is reserved for old variables only, not some new, constructed variables. Also, the selection of few first strong components is not what PCA does. It is what we do having done PCA. $\endgroup$ – ttnphns Jun 5 '16 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.