While experimenting with Spark library MlLib, I questioned myself if I understood well the mechanism of PCA algorithm, because output of MlLib algorithm was not what I expected to get. so for given data of 20 features and 10 observations:
val rdd: RDD[Vector] = sc.range(0, 20).map { row =>
val values = 1.to(10).map { _ => rand.nextInt(2).toDouble }
Vectors.dense(values.toArray) }
I applied pca
val pca = new PCA(5).fit(rdd)
and then transform my initial data :
val finalRdd = rdd.map { vector =>
val newVector = pca.transform(vector)
println("Before: " + vector)
println("After : " + newVector)
newVector }
I expected to have the rdd only with 5 features, but the values of those features should be the same, as in the initial rdd. As far as I understand pca, it keeps only the features/vectors that have the most important variance score. But instead of this I've got the rdd of 5 features, but values of those features were modified in the way I don't understand.
Before: [1.0,1.0,1.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0]
After : [1.14461218805706,0.8986586176492057,-0.580979228415323,0.09346270935831252,0.238925870721517]
Before: [1.0,1.0,0.0,1.0,0.0,0.0,0.0,1.0,1.0,0.0]
After : [1.1606629979632594,0.6110896934565042,-0.3717757311830274,-0.22954508416826663,0.18370214647002148]
Before: [0.0,1.0,0.0,1.0,1.0,1.0,0.0,1.0,1.0,0.0]
After : [1.6723486323814756,0.41082473274832476,-0.7526031076025419,-0.26291312583276166,0.11992366811728228]
Could you explain me why those features are modified? and when my understand of pca is wrong? Does the algorithm make something else with the data, and not only selection of features?