6
$\begingroup$

I've been trying to perform a binary classification using an SVM classifier (scikit-learn's SVC with RBF kernel). I have a sample size of about 100, with about 70 features each. The features are of approximately the same order of magnitude in their raw form, and the values tend to be already distributed around the 0 (not always though). The distribution of two such features is shown in the histograms below. feature distribution examples

I performed a Z-score transformation on all features, as I know this to be considered a good practice when working with multiple features in machine learning. The problem is that when I use the raw data, I always manage to get better accuracy than with the Z-scores (about 2-3%). Bear in mind that the parameters of the SVMs in each case are optimized using a grid-search, so I'm not using exactly identical classifiers.

Does this make sense, getting worse results with Z-scores? I would expect to get the same or better results. What could be the mathematical logic behind this?

Edit

To answer two frequent questions from comments and answers:

  1. My classes are indeed distributed equally (exactly 50%/50%)
  2. I also use measures other than accuracy (AUC, F1, etc.), but having worked on this project for some time, accuracy correlates well with what I need.
$\endgroup$
  • $\begingroup$ If you say that the parameters of the svm's are not identical, do you mean the in both cases you have different radius r for the rbf? And/or different C? Did you try with identical values? $\endgroup$ – user83346 May 31 '16 at 9:52
  • $\begingroup$ Yes, I mean different C and different gamma. It seems that because the orders of magnitude are different, I need to find the optimal parameters for each type of data (normalized and raw) or I may not get a separation at all. Perhaps I am wrong, but working with the same params definitely doesn't work. FYI - The default SVC params work ok with the Z-score data and not at all with the raw, which makes sense to me. $\endgroup$ – Shovalt May 31 '16 at 10:56
  • $\begingroup$ see my answer below, hope it helps $\endgroup$ – user83346 Jun 9 '16 at 13:12
  • $\begingroup$ This question is a duplicate of the question asked here: stats.stackexchange.com/questions/172795/… but we can't close it because the bounty is open. $\endgroup$ – Sycorax says Reinstate Monica Jun 10 '16 at 16:33
  • $\begingroup$ @GeneralAbrial, thanks for your answer. You are correct, the question you mentioned does seems like a possible duplicate. On the other hand, I could imagine intuitively why 0-1 scaling can destroy results (a single outlier can do that), while z-score scaling doesn't seem so trivial to me. That's why I didn't search for problems with scaling in general. $\endgroup$ – Shovalt Jun 16 '16 at 9:30
7
+50
$\begingroup$

Keep in mind why people typically scale features prior to estimating an SVM. The notion is that the data are on different scales, and this happenstance of how things were measured might not be desirable -- for example, measuring some length quantity in meters versus kilometers. Obviously one will have a much larger range even though both represent the same physical quantity.

However, there's no reason that the new scaling must be better. While it's true that the rescaled features will all vary in comparable units, it's also possible that the original scaling happened to encode the data such that some important features had more prominence in the model.

Consider the example of two different versions of the Gauissian RBF kernel: $K_1(x,x^\prime)=\exp(-\gamma||x-x^\prime||^2_2).$ This is an isotropic kernel, meaning that the same scaling ($\gamma$) is applied in all directions. A more general kernel function might have the form $K_2(x,x^\prime)=\exp\big(-(x-x^\prime)\Gamma(x-x^\prime)\big);$ it is anisotropic as $\Gamma$ is a diagonal PSD matrix, with each element applying a different scaling to each direction. The advantage of this kernel function is that it will vary more strongly in some directions than others.

Coming back to your question, it's possible to imagine that your data have, for whatever reason, some features that are more important than others, and that this coincides with the scale on which they are measured. Placing them on the new scale where they all appear on similar scales and are all treated as equally important means that unimportant or noise features cloud the signal.

As an aside, don't use accuracy as a metric for comparing models:

$\endgroup$
  • $\begingroup$ "it's possible to imagine that your data have, for whatever reason, some features that are more important than others, and that this coincides with the scale on which they are measured." Is this as same as my hypothesis? Thanks $\endgroup$ – Haitao Du Jun 10 '16 at 17:42
  • $\begingroup$ +1 in general but I would say probably one more sentence on "don't use accuracy as a metric for comparing models." as it stands it is a mildly aphoristic note of caution and not a piece of advice. (eg. Friedman's ranking, Cohen $k$, accuracies of two classification models using hold-out data, etc.) $\endgroup$ – usεr11852 says Reinstate Monic Jun 10 '16 at 19:38
  • $\begingroup$ @hxd1011 I think it coincides with your explanation in the sentence beginning "my hypothesis..." from your answer. I'm not sure how the discussion of the properties of soft-margin linear SVM helps with the circumstances of OP's question, which is centered on RBF kernel SVM, particularly because kernel SVM does not have separable feature ranking in the same way as a linear SVM. $\endgroup$ – Sycorax says Reinstate Monica Jun 10 '16 at 20:08
2
$\begingroup$

SVM is minimizing hinge loss with ridge regularization

$$ \min_\mathbf w \sum_i(1-y_i \mathbf w\cdot \mathbf x_i)_+ +\lambda ||\mathbf w||^2 $$

So, the scaling will make differences when we have the regularization term.

My hypothesis would be the original scale of your features impacts regularization on different features and make the performance better, but after the scaling, that disappears.

For example, you have 2 features, the first feature is in scale of $10,000$ and second feature is in scale of $0.1$.

  • if you do not perform scaling, the SVM will regularize much more on 2nd feature, and almost have no effects for the weight of on 1st feature.
  • if you do perform scaling, the SVM will regularize both features equally

You can validate my hypothesis to check the "feature importance" in your data. If you see, features in larger magnitude is much more important, at the same time you have "many useless features" in small scale. Then, my hypothesis might be right.

$\endgroup$
  • 1
    $\begingroup$ Though this would be a very relevant note for a linear-kernel SVM, the OP here is using an RBF kernel, in which case the interaction of feature scaling and regularization is far less direct. $\endgroup$ – Dougal Jun 15 '16 at 21:30
  • $\begingroup$ @hxd1011, thanks for the idea. I shall look for correlations between feature scale and feature importance. $\endgroup$ – Shovalt Jun 16 '16 at 9:33
  • $\begingroup$ @Shovalt, would it be easy for you to make a run of linear-svm to see if scaling matters ? Or even better, to point to similar data on the web ? Thanks $\endgroup$ – denis Jun 26 '16 at 14:29
0
$\begingroup$

Two points:

  • First: are your classes distributed equally (I think this is in General Abrial's link; but I haven't read it so unsure)? I.e. do you have 50% class A, and 50% class B? Or is it that you have one class to be more dominant? Accuracy is very sensitive to the class imbalance issue. E.g. if 90% of test cases are A, and 10% are class B, then a dumb classifier that always predicts 'A' will score 90% accuracy (clearly not a good classifier, yet gets 90%). Therefore, you should tell us the distribution of classes in order to allow us to conclude whether the 2% to 3% increase in accuracy is actually due to good generalization (as opposed to a dumb model that is taking trivial advantage of an imbalanced class distribution).
  • Second: Once we are happy that the accuracy is not abused, and right before we try to explain/justify what might be causing the 2% to 3% increase in accuracy, it is very important to first answer this question: Is this difference significant to begin with? Or is it due to sheer dumb luck?

    We have two hypothesis:

    • Null: there is no systematic difference, and observed change is due to random chance.
    • Alt: there is a systematic difference.

    In my view, usually 100 samples is too little to reject the null hypothesis when the difference in accuracy is only 2% to 3%.

    I really think you need to show statistical significance tests. By experience, I feel that you will fail to reject the null hypothesis with $p \le 0.05$ given that your samples are only 100.

    In my view, it is quite possible that such 2% to 3% difference could be due to the randomness associated with deciding the train-test split, or the initial randomization of k-fold cross-validation.

In summary:

  • Report the ratio of number of samples from class A to that of class B.
  • Report the $p$ value of the observed difference in accuracy.

Depending on this, we could conclude crazy things such as maybe the more accurate classifier is actually inferior if it is harmfully blindly sensitive to the class of majority. But maybe we could also conclude what you were expecting. Regardless, in my view, the input is not adequate to know what is happening and we need the above points addressed first.

$\endgroup$
  • $\begingroup$ I think there is a fair chance that this is what you'd call a non-systematic difference (i.e. by chance), but I did try this on several different samples, and different CVs, and the behavior seems consistent. In the long run I do plan on continuing to use both scaled and non-scaled data, until I a have more definitive results. $\endgroup$ – Shovalt Jun 16 '16 at 9:41
  • $\begingroup$ So I now feel that you are right about there being a systematic difference. But what about the ratio of the classes? Are they balanced? $\endgroup$ – caveman Jun 16 '16 at 10:42
  • $\begingroup$ Yes they are, I forgot to mention that I edited the question in response to that. $\endgroup$ – Shovalt Jun 16 '16 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.