How would this influence the accuracy of the SVM model?

Let's suppose that I have one variable which max value is 100 and minimum is 0.

Currently, I send it to SVM as a single continuous feature, for example, if the value is 8 I send it as 0.08.

I am wondering though, what would happen if I were to represent this variable as a set of 100 dummy variables? For example if the variable has a value of 56, I set the feature number 56 to 1 and the rest stays at 0.

Would this work worse than a one continuous feature? How about combining these two approaches? Would that improve the accuracy or lower it? Or maybe it would make no difference?

  • $\begingroup$ I use a linear kernel - if that matters. $\endgroup$ – user3010273 Jan 23 '14 at 20:13

Interesting question. In general feature representation and preprocessing is a topic not really touched in literature. Most ML papers/books start analysis after having established the representation of the problem and I haven't really seen cases where they revisit/question the very first step. Usually after some first studies there is some consensus on what works best.

In your example you basically do a quantisation of a continuous variable. I don't think that having so many levels will make any difference, but having something like values 1-20, 21-40,... 81,100 (5 levels) may help. Also in similar techniques you actually put 0 to all features except the one that corresponds to your value. The reason is that you want your features as much uncorrelated as possible. This approach of preprocessing may add some value because you may help your classifier to generalise. It's like saying, "hey it doesn't really care if that value is from 0 to 20 treat this range as one". On the other hand, depending on the problem, many levels (up to continuous) may actually be better for your performance.

My overall advice is to try both approaches, compare and keep what's best for your problem. You can actually make a plot of performance vs. quantisation levels and see if there is a trend.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.