1
$\begingroup$

I have a dataset with 1175 examples and 21 features which are in the range of [-1, +1], and two class labels 1 and 0. As I read in the most of the resources, it is good to have data in the range of [-1, +1] or [0, 1]. So I thought I don't need any preprocessing. But when I run SVM and decision tree classifiers from scikit-learn. I got 100% accuracy using cross-validation with 10 folds.

Afterwards, I tried to multiply the data by 10, 100, 1000 etc. Every time I increase the number, I get less accuracy. For example I got 91% for multiplying with 100. On the other hand, the decision tree classifier stays the same for all multiplications.

$\endgroup$
  • $\begingroup$ You mean you get less accuracy with the svm, correct? $\endgroup$ – Matthew Drury May 8 '15 at 19:18
  • $\begingroup$ Yes with svm, accuracy of decision tree stay same always : 100%. $\endgroup$ – robowolf May 8 '15 at 19:19
1
$\begingroup$

From the sklearn docs:

Support Vector Machine algorithms are not scale invariant, so it is highly recommended to scale your data. For example, scale each attribute on the input vector X to [0,1] or [-1,+1], or standardize it to have mean 0 and variance 1. Note that the same scaling must be applied to the test vector to obtain meaningful results. See section Preprocessing data for more details on scaling and normalization.

Decision trees though, are scale invariant, and in fact, are invariant under any monotonic transformation of the features.

To see why, notice that the default kernel, the rbf kernel, contains a scale parameter $\gamma$:

$$ exp( -\lambda |y - x|^2) $$

The $\lambda$ parameter controls the width of the kernel, so the scale is built in right there. You can pass a $\lambda$ into the svm as a parameter to the fit function, and by choosing it appropriately you can probably recover your initial results with all the features re-scaled.

On the other hand, the decision tree is just looking for optimal splits between data points, and the concept of a split is not dependent on scale, only ordering.

$\endgroup$
  • $\begingroup$ So how does it help? My data is already in the range of [-1,+1]. And getting 100% accuracy which is impossible in my case since I have noises on my data. $\endgroup$ – robowolf May 8 '15 at 19:28
  • $\begingroup$ Following the docs, I believe you should keep your data on the $[-1, 1]$ scale. To diagnose why you are getting 100% accuracy, when you believe that to be impossible, would require you to supply more details about your data and how you are using the algorithms. Also, just because you have noise in your data does not make 100% classification accuracy impossible, that depends on the signal to noise ratio. $\endgroup$ – Matthew Drury May 8 '15 at 19:29
  • $\begingroup$ What kind of detail is needed? I am collecting these datas from a Kinect device which gives me angles and positions of the certain joints of the human body. I am using the algorithms as described in this link $\endgroup$ – robowolf May 8 '15 at 19:43
  • $\begingroup$ Since this question was about why the algorithms were behaving differently with respect to scaling, I would open up a new question explicitly asking "Why am I getting 100% performance." Describe in detail your experiment, especially what your predictors and response represent. If possible, supply a small sample of your data. $\endgroup$ – Matthew Drury May 8 '15 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.