3
$\begingroup$

I am doing a machine learning project where I need to do

  1. PCA
  2. then K-means clustering,
  3. then One class SVM

It seems all those procedure requires data scaling. Should I (A) scale my data just before PCA, or should I (B) scale my data every time I encounter a new procedure?

(B) will be like scaling the data before PCA, then scaling the data again before K-means,then scaling the data again before One class SVM. I am feeling (B) may lose the original information because I have been scaling it too many times.

$\endgroup$
2
  • $\begingroup$ What do you do with the cluster data in the SVM (so step 2 to step 3)? $\endgroup$ Dec 24 '15 at 16:15
  • 1
    $\begingroup$ for each cluster, I will perform the One Class SVM separately.Let's say there will be 2 clusters, then there will be 2 learning frontier separately. Then when do testing, the testing point will first be identified using the the first learning frontier, if it was not recognized as normal, then it will go to the second learning frontier to be tested again. $\endgroup$
    – Shuang Zhou
    Dec 28 '15 at 16:15
1
$\begingroup$

In general the answer would be yes - many methods require (or behave better) once data is scaled. This will lose some original meaning, but this is the whole point of normalization - you remove some relations from data to reduce bias coming from the representation. This is just an idea of "assume as low as you can" - this does not mean that lack of normalization is always worse, it just means that "statisticaly speaking" there is a greater chance of good results after normalization, nothing more.

$\endgroup$
2
  • $\begingroup$ Hi lejlot, thank you for your reply.So you are saying in my case, I should scale the data again every time I am going to perform a new procedure? $\endgroup$
    – Shuang Zhou
    Dec 23 '15 at 20:28
  • $\begingroup$ Exactly. Although after PCA your data should be already normalized, so you can omit this part. $\endgroup$
    – lejlot
    Dec 23 '15 at 20:31
0
$\begingroup$

From what I can tell, you are going to feed the output from PCA into k-means clustering and then feed the output of that into SVM. If that's correct, it seems to me you'll only need to scale the data once, when it goes into PCA. The output from PCA will then be scaled (since the outputs are the principal components of the scaled data) and the output of k-means will also have the same scaling (since it is finding group means of scaled data). You can verify that the PCA and k-means outputs are scaled by, let's say, running on a relatively small subset of the data.

Of course, it goes without saying that it helps to understand what each model is doing, instead of treating each one as an incomprehensible black box. Take a look at Brian Ripley's book, "Pattern Recognition and Neural Networks" and/or Hastie, Tibshirani, and Friedman, "Introduction to Statistical Learning."

$\endgroup$
2
  • $\begingroup$ Hi Robert, thank you for your reply. I am not sure what do you mean by the output from PCA will be scaled.Do you mean they will be scaled as Z-score?I checked the PCA outputs, it seems their variance are not 1. $\endgroup$
    – Shuang Zhou
    Dec 28 '15 at 16:17
  • $\begingroup$ If the PCA inputs are scaled, so will be the outputs. If the PCA inputs are not scaled, neither are the outputs. $\endgroup$ Dec 28 '15 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.