What is the correct way to apply word embeddings to new data?

Question

Consider the words "banana" and "split". Assume that a pre-trained word embedding (say, word2vec GoogleNews) has the vectors like so:

banana_vec = array([[1.56, -2.46, 6.13, ... , -2.81]])

split_vec = array([[3.56, 9.45, -2.43, ... , 5.32]])

Now consider a completely new data set consisting of documents (sentences). Assume that one of the documents is the sentence "banana split".

How does one represent that sentence with the pre-trained word embeddings?

Things I've considered include:

Sum word embedding elements for each word

So this would give something like

bananana split = array([[v1, v2]])

where v1 is the sum of elements in banana_vec, and v2 is the same for split_vec.

L2 norm of word embedding elements for each word

So this would be the same, but the L2 norm instead of the sum.

Sum word embedding elements across words

So this would be:

banana split = array([[v1, v2, v3, ... , v100]])

where v1 = 1.56 + 3.56

L2 norm of embedding elements across words

Same as above except v1 = sqrt(1.56^2 + 3.56^).

Or is it something completely different?

Thanks in advance.

Answer 1

In the pre-trained word embeddings that have been loaded, each word will have a vector of the same length, $n$.

doc_vectors = []
for doc in documents:
    doc = tokenize(doc)
    feature_vector = zeros_array(n)
    for word in document:
        vec = vector (embedding) for the word
        feature_vector += vec
    feature_vec = feature_vec / len(doc)
    doc_vectors.append(feature_vector)

So for the example in the question:

vec("banana") = array([[1.56, -2.46, 6.13, ... , -2.81]])

vec("split") = array([[3.56, 9.45, -2.43, ... , 5.32]])

1. feature_vec("banana split") = vec(banana) + (split) = array([[5.12, 6.99, ..., 2.51]])

2. feature_vec("banana split") = feature_vec("banana split") / length

And that is how feature vectors are achieved for documents embeddings, which was one of the suggestions in the question.