15
$\begingroup$

So I've been playing around with SVMs and I wonder if this is a good thing to do:

I have a set of continuous features (0 to 1) and a set of categorical features that I converted to dummy variables. In this particular case, I encode the date of the measurement in a dummy variable:

There are 3 periods that I have data from and I reserved 3 feature numbers for them:

20: 21: 22:

So depending on which period the data comes from, different features will get 1 assigned; the others will get 0.

Will the SVM work properly with this or this is a bad thing to do?

I use SVMLight and a linear kernel.

$\endgroup$
  • $\begingroup$ This works fine. $\endgroup$ – Marc Claesen Jan 21 '14 at 16:49
  • $\begingroup$ What you have done is good enough. I have given a slightly detailed answer here - quora.com/Machine-Learning/… $\endgroup$ – TenaliRaman Jan 22 '14 at 15:18
  • $\begingroup$ @TenaliRaman please don't post links to sites that require one to login before being able to actually read anything. $\endgroup$ – Marc Claesen Jan 22 '14 at 16:41
  • $\begingroup$ @MarcClaesen I have replicated the answer below. $\endgroup$ – TenaliRaman Jan 23 '14 at 5:45
8
$\begingroup$

SVMs will handle both binary and continuous variables as long as you make some preprocessing: all features should be scaled or normalised. After that step, from the algorithms' perspective it doesn't matter if features are continuous or binary: for binaries, it sees samples that are either "far" away, or very similar; for continuous there are also the in between values. Kernel doesn't matter in respect to the type of variables.

$\endgroup$
  • 1
    $\begingroup$ and what would be the best normalization technique? $\endgroup$ – Shlomi Schwartz Sep 26 '18 at 9:10
23
$\begingroup$

Replicating my answer from http://www.quora.com/Machine-Learning/What-are-good-ways-to-handle-discrete-and-continuous-inputs-together/answer/Arun-Iyer-1

  • Rescale bounded continuous features: All continuous input that are bounded, rescale them to $[-1, 1]$ through $x = \frac{2x - \max - \min}{\max - \min}$.
  • Standardize all continuous features: All continuous input should be standardized and by this I mean, for every continuous feature, compute its mean ($\mu$) and standard deviation ($\sigma$) and do $x = \frac{x - \mu}{\sigma}$.
  • Binarize categorical/discrete features: For all categorical features, represent them as multiple boolean features. For example, instead of having one feature called marriage_status, have 3 boolean features - married_status_single, married_status_married, married_status_divorced and appropriately set these features to 1 or -1. As you can see, for every categorical feature, you are adding k binary feature where k is the number of values that the categorical feature takes.

Now, you can represent all the features in a single vector which we can assume to be embedded in $\mathbb{R}^n$ and start using off-the-shelf packages for classification/regression etc.

Addendum: If you use Kernel Based Methods, you can avoid this explicit embedding to $\mathbb{R}^n$ and focus on designing custom kernels for your feature vectors. You can even split your kernel into multiple kernels and use MKL models to learn weights over them. However, you may want to ensure positive semi-definiteness of your kernel so that the solver doesn't have any problems. However, if you are unsure of whether you can design custom kernels, you can just follow the earlier embedding approach.

$\endgroup$
  • $\begingroup$ Well, this article is quite interesting regarding categorical features. It's not said that one-hot encoding is the best choice for categorical features is what I'm getting out of it. $\endgroup$ – displayname Sep 18 '17 at 16:55
  • 1
    $\begingroup$ this is an excellent answer, I read the link in @displayname comment and it is a useful comparison. From the article, it appears that binary encoding is the best, (not the one hot described in this answer) and quite simple as well) From the link " Binary: first the categories are encoded as ordinal, then those integers are converted into binary code, then the digits from that binary string are split into separate columns. This encodes the data in fewer dimensions that one-hot, but with some distortion of the distances." $\endgroup$ – shelbypereira Dec 11 '17 at 14:33
  • $\begingroup$ The article given by @displayname is a good article, but should not be taken at face value. The first thing to remember is that almost all ML methods work with either similarity or distance measure. The choice of encoding method directly influences how the distances or similarities are measured between two points. A 1 hot encoding says that an object of one category is similar to only itself or equivalently, it puts all categories in equal distance to each other. However, there are cases where certain categories are closer than others. In which case, a different encoding can help. $\endgroup$ – TenaliRaman Dec 12 '17 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.