7
$\begingroup$

It is often recommended to do feature scaling (e.g. by normalization) when using a Support Vector Machine. For example here:

When using SVMs, why do I need to scale the features?

or also on wikipedia:

Application. In stochastic gradient descent, feature scaling can sometimes improve the convergence speed of the algorithm. In support vector machines, it can reduce the time to find support vectors. Note that feature scaling changes the SVM result.

Does this depend on the kernel? Do you recommend feature scaling on some kernels and on others not? What about:

  • Linear Kernel
  • Polynomial Kernel
  • Gaussian Kernel
  • RBF Kernel
  • and other Kernels?
$\endgroup$

1 Answer 1

8
$\begingroup$

Yes feature scaling depends on the kernel and in general it's a good idea. The kernel is effectively a distance and if different features vary on different scales then this can matter. For the RBF kernel, for instance, we have $$ K(x, x') = \exp\left(-\gamma ||x-x'||^2\right) $$ so if one dimension takes much larger values than others then it will dominate the kernel values and you'll lose some signal in other dimensions. This applies to the linear kernel too.

But this doesn't apply to all kernels, since some have scaling built in. For example, you could do something like the ARD kernel or Mahalanobis kernel with $$ K(x, x') = \exp\left(-\gamma (x-x')^T\hat \Sigma^{-1}(x-x')\right) $$ where $\hat \Sigma$ is the sample covariance matrix or maybe just the diagonal matrix of individual variances. As a function of $x$ and $x'$ this is still PD so it's a valid kernel.

As a general strategy for deciding if this is an issue for any particular kernel, just do what they did in the linked question and try it with data like $x=(1000,1,2,3)$, $x'=(500, .5, 3, 2)$ and see if the first dimension necessarily dominates.

Another way to try to assess a given kernel is to try to see if it inherits scale issues from subfunctions. For example, consider the polynomial kernel $K_{poly}(x,x') = (a+cx^Tx')^d$. We can write this as a function of the linear kernel $x^Tx'$, which we already know to be sensitive to scale, and the map $z \mapsto (a+cz)^d$ won't undo scale issues, so we can see that the polynomial kernel inherits these issues. We can do a similar analysis by writing the RBF kernel as a function of the scale-sensitive $||x-x'||^2$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.