Categorial features in linear machine learning algorithms

Question

How can I deal with categorial features inside linear ML algorithms? The obvious solution is to represent each value as a binary feature. For example, if the categorical feature color has three possible values: red, blue and green, then we replace it with three binary features [color == red], [color == green], [color == blue]. Are there better solutions?

Specifically, I am using Vowpal Wabbit's logistic regression. As far as I understood from the tutorial VW can operate with categorial features, but I did not find out how does in work.

Answer 1

What you told about categorial features is called feature binarization. That is extremely easy and commonly used. And that is basically the only working algorithm if your features consist of one word.

What is used in Vowpal Wabbit package (as well as in most packages I know, like scipy, scikit) is a little bit more common thing and called Bag of Words.

Suppose your features are of two or more words: "red wood" or "blue iron". Then you can evaluate the frequency of all words, not phrases. And for each sample you will now have a frequency vector of the words in it. If you will normalize it with overall frequencies, you will get the Bag of Words representation.

A little bit more detailed description is given, for example, here.

Speaking about Vowpal Wabbit package, they implement it in the function tokenize in the file *parse_primitives.cc*.

Hope it will help.