Testing an SVM performance - unnormalized data performing better than normalized

Question

I'm training a SVM on a dataset in OpenCV that contains 14 features and thousands of observations.

I understand that for optimal performance, it is recommended to perform principal component analysis on your features before training the SVM on the features.

So I ZMUV'ed my features, and then performed PCA and trained the SVM on that. Then I tested on the training dataset and I got a whopping 41% error rate.

I took out the PCA part, and I got the error rate down to 29%.

And then I took out ZMUV normalization altogether and I got down to just 21%.

Does anyone know why this is happening, or is my understanding of SVM/PCA incorrect? I admit I am new to this.

PS. When I took out ZMUV and just did PCA, my error rate is around 22%.

Answer 1

Feature normalization isn't guaranteed to improve performance, it just usually does. The same applies for PCA, though this is not that important for SVM.

When using kernel methods, you need to be aware that everything is based on how you define similarity. For example, if you use the linear kernel, the similarity between two instances $\mathbf{u}$ and $\mathbf{v}$ is expressed by: $$\kappa(\mathbf{u},\mathbf{v})=\mathbf{u}^T\mathbf{v}$$

So what's all this feature normalization about?

Feature normalization is usually a good thing, because you don't know which features are actually useful for your model in advance. If feature $a$ is on a different scale than feature $b$, it will dominate in the inner product (i.e., similarity will be more heavily influenced by feature $a$ than $b$). Since you don't know in advance which features are relevant, it's best to give them all the same weight.

Feature normalization degrades performance when it actually turns out that your most informative features were the ones that originally had larger scales than others.

See also:

• Are there any theoretically rigorous justification for why scaling or normalizing data should improve statistical performance?
• (I have answered another similar question with a full numeric example but I can't seem to find it right now)