This might be a naive question, but I am wondering why we (or maybe it's just me) are converting categorical class labels to integers before we feed them to a classifier in a software package such as Python's scikit-learn ML library?

Let's take the simple Iris dataset, why do we convert the class labels from "Setosa", "Virginica", and "Versicolor" to e.g., 0, 1, and 2?

This question came up when I was collaboratively working on a project and one of my colleagues didn't use a label encoder to convert the class labels from strings to integer. It worked (she was using scikit-learn); I intuitively "corrected" it (inserted a label encoder) and she asked me why: Well, I really had no good answer to that other than "most machine learning algorithms work better this way" (this is something I read sometime ago somewhere).

Now that I thought about it: What is the rationale behind it? Since in typical classification tasks class labels are nominal, not ordinal variables, is it computational efficiency (storing and processing less "data")?

  • 1
    It's just a matter of being practical. For binary classification the simplest way is using booleans, for multiclass it's integers. Most back-end libraries (C/C++) are typed, and typically use the most basic type that gets the job done. – Marc Claesen Jan 22 '15 at 14:46
  • How else would you do it? – shadowtalker Jan 22 '15 at 23:33
  • Its because algorithm knows only numbers not strings.Its a mathematical problem,so we must feed only numbers not string – aneesh cool Jul 9 at 5:15
up vote 4 down vote accepted

Scikit learn only handles real numbers I believe. So you need to do something like one hot encoding where n numerical dimensions are used to represent membership in the categories. If you just pass in strings they'll get cast to floats in unpredictable ways.

There are mathematical reasons some methods (like svm) need floats. IE they are only defined in the space of real numbers. Representing 3 categories as values 1,2,3 in a single method might work but it may also yield suboptimal performance compared to one hot encoding since the split (1,3) vs (2) is difficult to pick up on unless the method can capture very non linear behavior like that.

Other methods like random forest can be made to work directly on categorical values. Ie during decision learnings you can propose potential splits as diffrent combinations of categories. For such methods it is often convenient to use ints to represent the categories because an array of ints is much nicer to work with then an array of strings on a computational level. You can also do things like generate all possible combinations of n categories by looking at the bit values of an n-bit integer you are incrementing which can be much faster and memory efficient then searching for splits over n-floats.

  • 1
    Thanks for taking the time to write this very thorough answer. In scikit-learn, I typically use the OneHotEncoder only for categorical (nominal) features (to avoid implying an order). I have to try it out if it makes a difference whether I use the label encoder or OneHotEncoder for class labels. However, in the scikit-doc it seems like the LabelEncoder is what is typically used for class labels:… – biostats101 Jan 22 '15 at 21:16
  • 2
    I just got that you are asking about class labels in the target variable in particular. For those I think scikit will do an internal conversion to numbers if you provide strings? One reason to use numbers internally is that it is much more efficient to, say, keep class counts in an array indexed via an int then in a hash indexed via a string. – Ryan Bressler Jan 22 '15 at 22:03

For binary classification you usually use 0/1 or -1/1. Due to symmetry it does not matter which label corresponds to which class. For multiclass classification e.g. for 3-class classification you cannot use 0, 1 and 2 because this way of labeling implies an order (I am not familiar with Iris dataset though) and cannot be used for categorical data. One way to encode categorical labels is to use (1 0 0), (0 1 0) and (0 0 1). You can think of these labels as vertices of a an equilateral triangle in 3-D. Therefore, no order is implied. However if you are using a binary classifier (such as SVM) instead of a truly multiclass classifier we cannot use this labeling. Instead multiple binary classifiers are trained and their results are somehow combined with each other. For example if you have N categories you can train ${N \choose 2}$ classifiers and for each pair you use labels 0/1 to indicate the two classes (out of N) you are training against each other. At the test time majority vote between all ${N \choose 2}$ classifiers can be used to make a prediction.

If you are using an interface, perhaps it converts your 0/1/2 labels before interacting with the classifier(s) depending on what that classifier is.

  • Thanks for the answer. Yes, the 0,1,2 implies an order although we do classification without an order -- this is why I basically started asking myself why we do this. – biostats101 Jan 22 '15 at 21:06

Some algorithms can handle only numerical inputs, this might be main reason although storage is other reason.

Of course some algorithms can do conversion implicitly.

It's just a matter of being practical. For binary classification the simplest way is using booleans, for multiclass it's integers. Most back-end libraries are written in statically typed languages (C/C++), and typically use the most basic type that gets the job done without losing information.

  • That's what I assumed, but since scikit-learn didn't have any problems with the "string" labels I was maybe getting to philosophical. Guess I should close this question ... – biostats101 Jan 22 '15 at 14:54

protected by kjetil b halvorsen Jul 9 at 5:40

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.