9
$\begingroup$

I have used scaling on features of a model which contains 40 features (all columns are numbers) and a binary output variable.

This is the Kaggle contest here I've scaled the features assuming it would deliver better performance, but with a rbf kernel SVM, the accuracy with 10 fold CV fell from 0.92 to 0.87

Here is a box plot of features before and after scaling:

enter image description here enter image description here

What I would like to know is why scaling decreases classifier performance? I have not seen any discussions that point at this type of outcome.

Improve this question
$\endgroup$
2
  • $\begingroup$ Did you use the same parameters ($c$, $\gamma$) before and after scaling? If so, that would be why. $\endgroup$
    – Marc Claesen
    Commented Nov 27, 2013 at 18:28
  • $\begingroup$ I've used the defaults for both cases, but is not that the point? That is, scaling improving performance while everything else stays the same? What would be the right method of handling (c, γ) then? Pointers would be much appreciated $\endgroup$
    – mahonya
    Commented Nov 27, 2013 at 20:07

1 Answer 1

Reset to default
8
$\begingroup$

The problem is that you used the default parameter values in both cases. Apparently, the default values happened to be better for your data set before scaling (this is a coincidence).

When using SVM, the parameters $c$ and $\gamma$ play a crucial role and it is your task to find the best values. Your intuition is correct: the optimal performance is better when all features are scaled properly (or at least 99.99% of the time). Unfortunately, neither of your settings had optimal parameters which led to a result that seemed to reject your intuition.

Searching the optimal values for $c$ and $\gamma$ is typically done via a grid search (e.g. search a set of $<c,\gamma>$ combinations). You can estimate the performance of an SVM for a given set of parameters using cross-validation.

In pseudo-code, the general idea is this:

for c in {set of possible c values}
    for gamma in {set of possible gamma values}
        perform k-fold cross-validation to find accuracy
    end
end
train svm model on full training set with best c,gamma-pair

You can find a good beginner's tutorial here.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Brilliant. I have not used SVM before, and I was under the impression that the parameter optimisation was part of the algorithm or implementation. Better go do my reading properly. $\endgroup$
    – mahonya
    Commented Nov 28, 2013 at 9:01
  • 1
    $\begingroup$ @sarikan Training an SVM means finding the optimal solution to a specific question posed by the user. This question includes a definition of distance via kernel parameters, $\gamma$ for RBF, and a misclassification penalty via $c$. Those parameters define the training problem which is then solved. The parameters themselves must be set by the user before the training optimization problem can be solved. $\endgroup$
    – Marc Claesen
    Commented Nov 28, 2013 at 9:21
  • $\begingroup$ Thanks for this. It is these short definitions that are hard to find elsewhere. Real gems :) $\endgroup$
    – mahonya
    Commented Nov 28, 2013 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.