Effect of combining features on classification

Question

I have 2 string features F1 and F2 based on which I am trying to perform classification. I have two choices, either to use the features individually, or to concatenate the two features to generate feature F3 and then perform classification based on F3 alone.

As an example, let us say that I am trying to classify a bigram. Should I divide the bigram into its constituent words and use them as features, or should I use the bigram itself.

Examples: w1w2, label
Model 1: Y1(w1,w2) = label
Model 2: Y2(w1+w2) = label

What are the pros and cons of each approach? Which is expected to work better?

Answer 1

Suppose your two strings are first and last names. Your data looks like this:

label, firstname, lastname
1, 'Tom', 'Jones'
0, 'Bob', 'Dylan'
1, 'Bob', 'Jones'

Your bigram—or interaction term firstname * lastname—would be

label, ..., fullname
1, ..., 'Tom Jones'
0, ..., 'Bob Dylan'
1, ..., 'Bob Jones'

Now, when you run your classification algorithm, your data will need to take this form (usually the data format is automatically converted to this in your program, such as R):

label, Tom, Jones, Bob, Dylan, TomJones, BobDylan, BobJones
1, 1, 1, 0, 0, 1, 0, 0
0, 0, 0, 1, 1, 0, 1, 0
1, 0, 1, 1, 0, 0, 0, 1

Since we have more than one "Bob" (i.e. both "Bob Dylan" and "Bob Jones"), then the variable "Bob" is not perfectly correlated with "Bob Dylan". Therefore, the classification algorithm is able to use the variation of "Bob", and "Bob Dylan" to predict the label.

However, "Tom" is perfectly correlated with "Tom Jones", since there are other people who have the first name "Tom" in the data. Therefore, the classification algorithm cannot use both "Tom", and "Tom Jones". Instead, it can only use one or the other.

Since you'd like to use all the available information you have (i.e. firstname, lastname and fullname), but you don't know which firstnames/lastnames are perfectly correlated with fullname, then you should use a ridge regression because this type of regression can deal with perfectly correlated independent variables. Example code (in R) looks like this:

library(glmnet)
df <- read.csv(...)
X <- sparse.model.matrix(~firstname + lastname + firstname*lastname, df)
y <- df$label
fit <- cv.glmnet(X, y, family='binomial', alpha=0)
predictions <- predict(fit, X, type='response')