When performing the linear SVM classification, it is often helpful to normalize the training data, for example by subtracting the mean and dividing by the standard deviation, and afterwards scale the test data with the mean and standard deviation of training data. Why this process changes dramatically the classification performance?
$\begingroup$
$\endgroup$
5
-
1$\begingroup$ This question has already been answered stackoverflow.com/questions/15436367/svm-scaling-input-values $\endgroup$– jpmucCommented Jul 22, 2013 at 14:49
-
$\begingroup$ Thank you,juampa! However, i am still not quite clear why the test set needs to be scaled with the mean and std of the training set instead of its own? In some case, the later seems perform euqlly well or even better when the two classes of samples are well balanced in the test set. $\endgroup$– QinghuaCommented Jul 23, 2013 at 7:27
-
1$\begingroup$ because then you are not being consistent. You are testing on different data. Imagine you draw the samples from a Gaussian N(mu,sigma). You trained with N(0,1) (after centering and scaling) but tested with N(mu,sigma) $\endgroup$– jpmucCommented Jul 23, 2013 at 8:15
-
$\begingroup$ Related: stats.stackexchange.com/questions/77876/… $\endgroup$– Marc ClaesenCommented Mar 1, 2014 at 0:00
-
$\begingroup$ @Qinghua because that is cheating. In a real-life scenario you would not even have the test data when you train the model. The best you can do is scale with the mean, variance of training set. $\endgroup$– SoumyajitCommented Aug 26, 2021 at 10:18
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
SVM tries to maximize the distance between the separating plane and the support vectors. If one feature (i.e. one dimension in this space) has very large values, it will dominate the other features when calculating the distance. If you rescale all features (e.g. to [0, 1]), they all have the same influence on the distance metric.
$\begingroup$
$\endgroup$
1
I think it can be made more clear through an example. Let's say you have two input vectors: X1 and X2. and let's say X1 has range(0.1 to 0.8) and X2 has range(3000 to 50000). Now your SVM classifier will be a linear boundary lying in X1-X2 plane. My claim is that the slope of linear decision boundary should not depend on the range of X1 and X2, but instead upon the distribution of points.
Now let make a prediction on the point (0.1, 4000) and (0.8, 4000). There will be hardly any difference in the value of the function, thus making SVM less accurate since it will have less sensitivity to points in the X1 direction.
-
2$\begingroup$ Calling X1 and X2 to be input vectors is not strictly wrong but so uncommon that it is pretty close to being wrong: X1 and X2 are called "variables" or "features". Usually they are real numbers in practice and as such they are strictly speaking also vectors in the mathematical sense (if they are considered to be the underlying set of a vectors space). But in the context of a supervised learning in Euclidean space this is not what people mean. Usually $(X1, X2)$ is called to be the "input vector", instead of X1 and X2. $\endgroup$– Make42Commented Aug 17, 2020 at 11:09