Why does the scaling of feature vectors improve performance of SVM classifier?

Question

I've found that performing scaling in SVM problems really improves the performance of SVM ... But I don't understand why!

I have read this explanation: "The main advantage of scaling is to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges."

Unfortunately this didn't help me... Can somebody provide me a better explanation?

Thank you in advance!

Answer 1

To understand the phrase:

"The main advantage of scaling is to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges."

it helps to consider a simple example:

For simplicity, suppose that you are working in a 2-D space, so each observation has two attributes, say $x_1$ and $x_2$. Now let's also suppose that the first attribute $x_1$ takes values such as 1000, 1200, 1100, ... while the attribute $x_2$ takes values that are 100 times smaller, such as 10, 12, 11, ...

We can now see how (and why) the larger values dominate the usual Euclidean measurement of distance (square root of sum of squares), namely $||x|| := \sqrt{x_1^2 + x_2^2}$, as below:

Consider the length of the first point in the example: ||(1000,10)|| = 1,000.050. Notice that the second attribute contributes 0.05 to the length, or less than 0.01% of the total value.

In this example the first attribute dominates so heavily that even a four-fold change in the second attribute will barely register in the distance metric, e.g. ||(1000,40)|| = 1,000.80.

The main point? In order to have a classifier treat attributes as being of equal importance one must (in most cases) normalise the intrinsic difference in the scales with which attributes are measured.

This applies to all classifiers where a distance-type metric is being used, so $k$-means, SVM, and many others as well.

---

Fine Print: The main point is true in most (but not all) cases; this question is a perfect example of an exceptional case.

Answer 2

The statistical techniques based in distances work best with the scaled variables. The basic idea of the SVM is the margin, and this is related with the distance and the scale of the different variables