Raw data outperforms Z-score transformed data in SVM classification

Question

I've been trying to perform a binary classification using an SVM classifier (scikit-learn's SVC with RBF kernel). I have a sample size of about 100, with about 70 features each. The features are of approximately the same order of magnitude in their raw form, and the values tend to be already distributed around the 0 (not always though). The distribution of two such features is shown in the histograms below.

I performed a Z-score transformation on all features, as I know this to be considered a good practice when working with multiple features in machine learning. The problem is that when I use the raw data, I always manage to get better accuracy than with the Z-scores (about 2-3%). Bear in mind that the parameters of the SVMs in each case are optimized using a grid-search, so I'm not using exactly identical classifiers.

Does this make sense, getting worse results with Z-scores? I would expect to get the same or better results. What could be the mathematical logic behind this?

Edit

To answer two frequent questions from comments and answers:

1. My classes are indeed distributed equally (exactly 50%/50%)
2. I also use measures other than accuracy (AUC, F1, etc.), but having worked on this project for some time, accuracy correlates well with what I need.

Answer 1

Keep in mind why people typically scale features prior to estimating an SVM. The notion is that the data are on different scales, and this happenstance of how things were measured might not be desirable -- for example, measuring some length quantity in meters versus kilometers. Obviously one will have a much larger range even though both represent the same physical quantity.

However, there's no reason that the new scaling must be better. While it's true that the rescaled features will all vary in comparable units, it's also possible that the original scaling happened to encode the data such that some important features had more prominence in the model.

Consider the example of two different versions of the Gauissian RBF kernel: $K_1(x,x^\prime)=\exp(-\gamma||x-x^\prime||^2_2).$ This is an isotropic kernel, meaning that the same scaling ($\gamma$) is applied in all directions. A more general kernel function might have the form $K_2(x,x^\prime)=\exp\big(-(x-x^\prime)\Gamma(x-x^\prime)\big);$ it is anisotropic as $\Gamma$ is a diagonal PSD matrix, with each element applying a different scaling to each direction. The advantage of this kernel function is that it will vary more strongly in some directions than others.

Coming back to your question, it's possible to imagine that your data have, for whatever reason, some features that are more important than others, and that this coincides with the scale on which they are measured. Placing them on the new scale where they all appear on similar scales and are all treated as equally important means that unimportant or noise features cloud the signal.

As an aside, don't use accuracy as a metric for comparing models:

• Why is accuracy not the best measure for assessing classification models?
• "The Case Against Accuracy Estimation for Comparing Induction Algorithms", Foster Provost, Tom Fawcett, Ron Kohavi

Answer 2

SVM is minimizing hinge loss with ridge regularization

$$ \min_\mathbf w \sum_i(1-y_i \mathbf w\cdot \mathbf x_i)_+ +\lambda ||\mathbf w||^2 $$

So, the scaling will make differences when we have the regularization term.

My hypothesis would be the original scale of your features impacts regularization on different features and make the performance better, but after the scaling, that disappears.

For example, you have 2 features, the first feature is in scale of $10,000$ and second feature is in scale of $0.1$.

• if you do not perform scaling, the SVM will regularize much more on 2nd feature, and almost have no effects for the weight of on 1st feature.
• if you do perform scaling, the SVM will regularize both features equally

You can validate my hypothesis to check the "feature importance" in your data. If you see, features in larger magnitude is much more important, at the same time you have "many useless features" in small scale. Then, my hypothesis might be right.

Answer 3

Two points:

• First: are your classes distributed equally (I think this is in General Abrial's link; but I haven't read it so unsure)? I.e. do you have 50% class A, and 50% class B? Or is it that you have one class to be more dominant? Accuracy is very sensitive to the class imbalance issue. E.g. if 90% of test cases are A, and 10% are class B, then a dumb classifier that always predicts 'A' will score 90% accuracy (clearly not a good classifier, yet gets 90%). Therefore, you should tell us the distribution of classes in order to allow us to conclude whether the 2% to 3% increase in accuracy is actually due to good generalization (as opposed to a dumb model that is taking trivial advantage of an imbalanced class distribution).

• Second: Once we are happy that the accuracy is not abused, and right before we try to explain/justify what might be causing the 2% to 3% increase in accuracy, it is very important to first answer this question: Is this difference significant to begin with? Or is it due to sheer dumb luck?

  We have two hypothesis:

  • Null: there is no systematic difference, and observed change is due to random chance.
  • Alt: there is a systematic difference.

  In my view, usually 100 samples is too little to reject the null hypothesis when the difference in accuracy is only 2% to 3%.

  I really think you need to show statistical significance tests. By experience, I feel that you will fail to reject the null hypothesis with $p \le 0.05$ given that your samples are only 100.

  In my view, it is quite possible that such 2% to 3% difference could be due to the randomness associated with deciding the train-test split, or the initial randomization of k-fold cross-validation.

In summary:

• Report the ratio of number of samples from class A to that of class B.
• Report the $p$ value of the observed difference in accuracy.

Depending on this, we could conclude crazy things such as maybe the more accurate classifier is actually inferior if it is harmfully blindly sensitive to the class of majority. But maybe we could also conclude what you were expecting. Regardless, in my view, the input is not adequate to know what is happening and we need the above points addressed first.