# Effect of combining features on classification

I have 2 string features F1 and F2 based on which I am trying to perform classification. I have two choices, either to use the features individually, or to concatenate the two features to generate feature F3 and then perform classification based on F3 alone.

As an example, let us say that I am trying to classify a bigram. Should I divide the bigram into its constituent words and use them as features, or should I use the bigram itself.

Examples: w1w2, label
Model 1: Y1(w1,w2) = label
Model 2: Y2(w1+w2) = label


What are the pros and cons of each approach? Which is expected to work better?

• Why not include both? It's equivalent to a traditional interaction model, y ~ a + b + a*b Commented Nov 25, 2014 at 15:49
• @spin-glass If using both makes sense, that means that a+b and a*b are different. Please can you explain how they are different? My intuition is that they are exactly the same, except a+b will perform better because of bigram scarcity. Commented Nov 25, 2014 at 16:20

Suppose your two strings are first and last names. Your data looks like this:

label, firstname, lastname
1, 'Tom', 'Jones'
0, 'Bob', 'Dylan'
1, 'Bob', 'Jones'


Your bigram—or interaction term firstname * lastname—would be

label, ..., fullname
1, ..., 'Tom Jones'
0, ..., 'Bob Dylan'
1, ..., 'Bob Jones'


Now, when you run your classification algorithm, your data will need to take this form (usually the data format is automatically converted to this in your program, such as R):

label, Tom, Jones, Bob, Dylan, TomJones, BobDylan, BobJones
1, 1, 1, 0, 0, 1, 0, 0
0, 0, 0, 1, 1, 0, 1, 0
1, 0, 1, 1, 0, 0, 0, 1


Since we have more than one "Bob" (i.e. both "Bob Dylan" and "Bob Jones"), then the variable "Bob" is not perfectly correlated with "Bob Dylan". Therefore, the classification algorithm is able to use the variation of "Bob", and "Bob Dylan" to predict the label.

However, "Tom" is perfectly correlated with "Tom Jones", since there are other people who have the first name "Tom" in the data. Therefore, the classification algorithm cannot use both "Tom", and "Tom Jones". Instead, it can only use one or the other.

Since you'd like to use all the available information you have (i.e. firstname, lastname and fullname), but you don't know which firstnames/lastnames are perfectly correlated with fullname, then you should use a ridge regression because this type of regression can deal with perfectly correlated independent variables. Example code (in R) looks like this:

library(glmnet)