What does average of word2vec vector mean?

Question

The paper that I am reading says,

tweet is represented by the average of the word embedding vectors of the words that compose the tweet.

Does this mean each word in the tweet (sentence) has to be represented as the average of its word vector (still having the same length),

or does it mean the sentence itself has to be the average of all values computed above (average of word vectors of the word that the sentence contains)?

I am confused.

Answer 1

You can think of it in terms of physical analogy. You can take a flat surface, like a table, and arrange 30 balls on it. Then you can cut legs from the table and replace it with a single leg. In order to figure out where to put this leg you need to find center of mass of all 30 balls on the table. Assuming that each ball has the same size and weight than center of mass would be average position of all balls.

In the picture above, in the first example with 3 objects, you can see that center of mass much closer to the two objects that form small cluster. The same idea with word vectors. Each word is an object and sentence (or tweet) is just a set of these objects. If many vectors from the tweet close to each other in space than the overall average will be close to this cluster and would be a good representation of the tweet.

One remark is that taking average can be the same as just summing vectors, because in most cases you will use cosine similarity for finding close vectors. And with cosine similarity, dividing vector by $n$ is the same as multiplying it by $1/n$ which is a scalar and scale of the vector doesn't matter if you measure distance using angles.

Answer 2

This means that embedding of all words are averaged, and thus we get a 1D vector of features corresponding to each tweet. This data format is what typical machine learning models expect, so in a sense it is convenient.

However, this should be done very carefully because averaging does not take care of word order. For example:

our president is a good leader he will not fail

our president is not a good leader he will fail

Have the same words, and therefore will have same average word embedding, but the tweets have very different meaning.

Edit:

To address this issue, one might look into: Sentence-BERT

https://github.com/UKPLab/sentence-transformers and https://arxiv.org/abs/1908.10084

Here is quick illustration with the above example:

from sentence_transformers import SentenceTransformer
import numpy as np

def cosine_similarity(sentence_embeddings, ind_a, ind_b):
    s = sentence_embeddings
    return np.dot(s[ind_a], s[ind_b]) / (np.linalg.norm(s[ind_a]) * np.linalg.norm(s[ind_b]))

model = SentenceTransformer('bert-base-nli-mean-tokens')

s0 = "our president is a good leader he will not fail"
s1 = "our president is not a good leader he will fail"
s2 = "our president is a good leader"
s3 = "our president will succeed"

sentences = [s0, s1, s2, s3]

sentence_embeddings = model.encode(sentences)

s = sentence_embeddings

print(f"{s0} <--> {s1}: {cosine_similarity(sentence_embeddings, 0, 1)}")
print(f"{s0} <--> {s2}: {cosine_similarity(sentence_embeddings, 0, 2)}")
print(f"{s0} <--> {s3}: {cosine_similarity(sentence_embeddings, 0, 3)}")

Result:

our president is a good leader he will not fail <--> our president is not a good leader he will fail: 0.46340954303741455
our president is a good leader he will not fail <--> our president is a good leader: 0.8822922110557556
our president is a good leader he will not fail <--> our president will succeed: 0.7640182971954346

Answer 3

You have a tweet $T$, which is composed of words $w_1,w_2,\cdots,w_n$. Each word has a word2vec embedding $u_{w_1},u_{w_2},..,u_{w_n}$. So you define the tweet embedding as: $u_T:=\frac{1}{n}\sum_{i=1}^nu_{w_i}$.