How to replace scalars with vectors in simple models, such as classification of sentences where 1-hot encoding is replaced with word vectors

Question

I have a problem, which seems simple enough, but I don't know how it is solved in the industry. This has to do with the machinery of feeding data to a model, rather than trying to figure out the best sentence classification model.

Say I have a bunch of sentences and I want to classify them:

...
question: Do you like green eggs and ham?
question: Would you like them in a house?
question: Would you like them with a mouse?
answer: I do not like green eggs and ham.
answer: I would not like them here or there.
answer: I would not like them anywhere.
...

The labels are binary, 'question' and 'answer'. I have to figure out how to train this model.

One way is to use one-hot encoding. Take the whole corpus, sort it, use its index to to mark 0 or 1 if that word appears in the training observation, 0 if it doesn't. So the input becomes a simple matrix of numbers:

X:

0, 0, 0, 1, 0, 0, 1, 1
0, 1, 0, 0, 0, 0, 1, 0
1, 0, 1, 0, 0, 0, 1, 0
0, 1, 0, 0, 0, 0, 0, 0
...

(note that each row above is one-hot-encoding, so each 'cell' represents the presence or absence of a word in a training sample)

Naturally, the corresponding label vector will have two values: 1,0,0,1,1,...

So far, things are pretty straight forward in this silly example.

What if I decided to use word vectors instead? Now a single number (1 or 0) can't represent the presence or absence of a cell in an observation!

As far as I know, sklearn expects a matrix of numbers. Each 'cell' in the example above is now represented by a vector, not a scalar! How do I feed this to my model?

Further, outside of NLP this also applies. For example, in a straight forward business style dataset, it is normal to convert categorical variables to one-hot-encoding. However, what if I decided to use entity embeddings, which are becoming popular. How do I stuff vectors representing scalar values here as well?

I hope my question is clear. Please let me know if it isn't.

Answer 1

Since you would end up with one embedding per word and you need to somehow transform them into a single vector that will be the input to the classifier.

The simplest and surprisingly good is just doing an average of the embeddings. You just look up the word embeddings in a look-up table and compute the average. Usually, removing stopwords helps a lot, however, in your particular task, function words are a strong indicator of a sentence being a question.

Deep learning knows better ways of combining embeddings into a single vector. The most straightforward way would be using either an RNN and a 1-D CNN with max-pooling.

Note that both embedding averaging and one-hot bag-of-words features that you are using do not consider word order which plays an important role here. Introducing simple categorical features like: "Does the sentence start with Wh?" or "Does it end with question mark?" would help a lot, probably more than word embeddings.