2
$\begingroup$

ML newbie here, currently looking at a binary classification problem. I have quite a good number of training data (easily over 50k) which consists of both numeric and categorical data. The categorical data consists of both ordinal and nominal types.

Here's the problem. I am unsure of what is the most proper way of encoding the categorical data, and what are the factors I should consider when deciding the encoding method. I have came across several encoding methods, which can be summarized in this article.

As additional information, I am thinking of using logistic regression and random forest as my first test classifiers. I have read that certain encoding methods are more suitable for certain types of classifiers. Hope to have more insight on that matter as well.

I hope that you guys/girls can lend me a helping hand. Thank you very much in advance.

EDIT

Due to P&C, I cannot share instances of the data, however these are examples of the categorical features, and the number of different data for each feature:

  • Country (nominal) (40)
  • Job Grade (ordinal) (8)
  • Year/Quarter Joined (can be ordinal or nominal) (15)
  • Department/Business Unit (nominal) (10)

Library used: scikit-learn

$\endgroup$
  • $\begingroup$ Most standard classification algorithms work well with many types of variables. As to how you should do to specifically for your data we cannot tell. You should show us the data if we are to give advice. $\endgroup$ – user2974951 Sep 13 '18 at 7:34
  • $\begingroup$ @user2974951 I have updated my question to the best of my abilities. Hope it helps. Thank you for the tips. $\endgroup$ – Jialer Chew Sep 13 '18 at 8:03
  • $\begingroup$ You should mention which software you are using, if it's R or Python then you don't need to bother with this since they can handle this types of data without issues, they handle it internally. $\endgroup$ – user2974951 Sep 13 '18 at 8:09
  • $\begingroup$ @user2974951 I am currently using python scikit-learn. I understand there are default ways to handle it, but it doesn't mean the default way would be the more "proper" way of handling it, example it wont be able to differentiate between ordinal and nominal data. Unless I'm missing something here, would appreciate if you could provide some details. Thank you. $\endgroup$ – Jialer Chew Sep 13 '18 at 8:17
  • $\begingroup$ I don't know about Python, but in R you have the functions factor() and ordinal() to convert variables, so there must be something similar in Python too. As to which variables is which is something that you have to figure out, which will be more useful. $\endgroup$ – user2974951 Sep 13 '18 at 8:21
1
$\begingroup$

If you have ordinal variables you should encode them by mapping each one to a number. The numbers should be selected in such a way to depict the order or hierarchy of the values in your variable.

For example, say you have a variable called ratings which assumes the values "bad", "good", "very good". This is clearly an ordinal variable as these values have a clear order ("bad" < "good" < "very good"). In this case you want to map these to three numbers that preserve that order (e.g. "1", "2", "3").

If you have nominal (or categorical) variables you should perform a one-hot encoding. With this scheme you create $M$ new variables, where $M$ is the number of unique values in your variable (e.g. above $M$ was $3$). Each of these variables corresponds to one of the values. To encode the data this way, for each sample, you look at the value of the nominal variable and place $1$ to the corresponding new variable you created; the other variables take the value of $0$.

For example, say you have a variable called color and it takes the values "yellow", "red" and "green". These values can't be ordered in any way, so we clearly have a nominal variable. Like we said we now create $3$ new variables, each one corresponding to a value of the nominal (i.e. each one corresponds to a color). If a sample has a color of "red" the new red variable becomes $1$ while the other two are set to $0$.

While this increases your problem's dimensionality (which usually isn't a good thing), it avoids leading your to making false assumptions regarding the order of the variable (e.g. "red" < "yellow").

$\endgroup$
  • 1
    $\begingroup$ Thank you for the answer. However given the number of different categorical data as stated in my updated question, would you still think that one-hot encoding would be a good choice? Or would you think/feel that an algorithm that is independent of the ordering of the variable (such as decision tree / random forest) would be a more viable solution? $\endgroup$ – Jialer Chew Sep 13 '18 at 8:10
  • $\begingroup$ Well tree based algorithms are strong estimators, but I feel that you shouldn't exclude the others. My suggestions would be: First of all, encode the ordinal variables normally. The departments which has 10 values can be one-hot encoded without much of an issue. The problem is with the countries, which have 40 values and unless you have a lot of samples, some of the countries may have very few entries. If that is the case, I'd suggest grouping the countries into regions (e.g. by continent, by language, by economic status) depending on your goal. $\endgroup$ – Djib2011 Sep 13 '18 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.