When using SVMs, why do I need to scale the features?

Question

According to the documentation of the StandardScaler object in scikit-learn:

For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger that others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.

I should scale my features before classification. Is there any easy way to show why I should do this? References to scientific articles would be even better. I already found one but there are probably many other.

Answer 1

All kernel methods are based on distance. The RBF kernel function is $\kappa(\mathbf{u},\mathbf{v}) = \exp(-\|\mathbf{u}-\mathbf{v}\|^2)$ (using $\gamma=1$ for simplicity).

Given 3 feature vectors: $$ \mathbf{x}_1 = [1000, 1, 2], \quad \mathbf{x}_2 = [900, 1, 2], \quad \mathbf{x}_3 = [1050, -10, 20]. $$

then $\kappa( \mathbf{x}_1, \mathbf{x}_2) = \exp(-10000) \ll \kappa(\mathbf{x}_1, \mathbf{x}_3) = \exp(-2905)$, that is $\mathbf{x}_1$ is supposedly more similar to $\mathbf{x}_3$ then to $\mathbf{x}_2$.

The relative differences between $\mathbf{x}_1$ and: $$ \mathbf{x}_2 \rightarrow [0.1, 0, 0],\quad \mathbf{x}_3 \rightarrow [0.05, -10, 10]. $$

So without scaling, we conclude that $\mathbf{x}_1$ is more similar to $\mathbf{x}_3$ than to $\mathbf{x}_2$, even though the relative differences per feature between $\mathbf{x}_1$ and $\mathbf{x}_3$ are much larger than those of $\mathbf{x}_1$ and $\mathbf{x}_2$.

In other words, if you do not scale all features to comparable ranges, the features with the largest range will completely dominate in the computation of the kernel matrix.

You can find simple examples to illustrate this in the following paper: A Practical Guide to Support Vector Classification (Section 2.2).

Answer 2

It depends on what kernel you are using. By far the most commonly used (apart from linear) is the gaussian kernel, which has the form

$$ f = exp \left ( \frac{- || x{_{1}} - x{_{2}} || ^2 }{2\sigma ^2} \right ) $$

An SVM takes this function and uses it to compare the similarity of a point ($x1$) to every other point in the training set by summing the differences as:

$$ (x{_{1}}-l{_{1}})^2+(x{_{2}}-l{_{2}})^2...+(x{_{n}}-l{_{n}})^2 $$

where $x$ is your example and the values of $l$ are the landmarks.

If the feature $x{_{1}}$ ranges from 0 - 50,000 while the feature $x{_{2}}$ ranges from 0 - 0.01, you can see that $x{_{1}}$ is going to dominate that sum while $x{_{2}}$ will have virtually no impact. For this reason it is necessary to scale the features before applying the kernal.

If you want to learn more I recommend module 12 (Support Vector Machines) from the Stanford online course in machine learning at Coursera (free and available any time): https://www.coursera.org/course/ml