18
$\begingroup$

When performing the linear SVM classification, it is often helpful to normalize the training data, for example by subtracting the mean and dividing by the standard deviation, and afterwards scale the test data with the mean and standard deviation of training data. Why this process changes dramatically the classification performance?

$\endgroup$
4
  • 1
    $\begingroup$ This question has already been answered stackoverflow.com/questions/15436367/svm-scaling-input-values $\endgroup$ – jpmuc Jul 22 '13 at 14:49
  • $\begingroup$ Thank you,juampa! However, i am still not quite clear why the test set needs to be scaled with the mean and std of the training set instead of its own? In some case, the later seems perform euqlly well or even better when the two classes of samples are well balanced in the test set. $\endgroup$ – Qinghua Jul 23 '13 at 7:27
  • 1
    $\begingroup$ because then you are not being consistent. You are testing on different data. Imagine you draw the samples from a Gaussian N(mu,sigma). You trained with N(0,1) (after centering and scaling) but tested with N(mu,sigma) $\endgroup$ – jpmuc Jul 23 '13 at 8:15
  • $\begingroup$ Related: stats.stackexchange.com/questions/77876/… $\endgroup$ – Marc Claesen Mar 1 '14 at 0:00
14
$\begingroup$

I think it can be made more clear through an example. Let's say you have two input vectors: X1 and X2. and let's say X1 has range(0.1 to 0.8) and X2 has range(3000 to 50000). Now your SVM classifier will be a linear boundary lying in X1-X2 plane. My claim is that the slope of linear decision boundary should not depend on the range of X1 and X2, but instead upon the distribution of points.

Now let make a prediction on the point (0.1, 4000) and (0.8, 4000). There will be hardly any difference in the value of the function, thus making SVM less accurate since it will have less sensitivity to points in the X1 direction.

$\endgroup$
1
  • $\begingroup$ Calling X1 and X2 to be input vectors is not strictly wrong but so uncommon that it is pretty close to being wrong: X1 and X2 are called "variables" or "features". Usually they are real numbers in practice and as such they are strictly speaking also vectors in the mathematical sense (if they are considered to be the underlying set of a vectors space). But in the context of a supervised learning in Euclidean space this is not what people mean. Usually $(X1, X2)$ is called to be the "input vector", instead of X1 and X2. $\endgroup$ – Make42 Aug 17 '20 at 11:09
10
$\begingroup$

SVM tries to maximize the distance between the separating plane and the support vectors. If one feature (i.e. one dimension in this space) has very large values, it will dominate the other features when calculating the distance. If you rescale all features (e.g. to [0, 1]), they all have the same influence on the distance metric.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.