What does average of word2vec vector mean?

The paper that I am reading says,

tweet is represented by the average of the word embedding vectors of the words that compose the tweet.

Does this mean each word in the tweet (sentence) has to be represented as the average of its word vector (still having the same length),

or does it mean the sentence itself has to be the average of all values computed above (average of word vectors of the word that the sentence contains)?

I am confused.

• It'll be hard to answer this without more context. Could you link to the paper and provide a complete citation (in case the link dies)? – Sycorax Dec 14 '17 at 18:39
• It simply means that the author needed a single vector to represent a tweet so that he/she can run a classifier (probably). In other words, averaging of vectors was defined downstream by a tool that accepted a single vector. – Vladislavs Dovgalecs Dec 14 '17 at 19:23
• Can you post name of the paper for context? – StatguyUser Feb 1 '18 at 4:59

You can think of it in terms of physical analogy. You can take a flat surface, like a table, and arrange 30 balls on it. Then you can cut legs from the table and replace it with a single leg. In order to figure out where to put this leg you need to find center of mass of all 30 balls on the table. Assuming that each ball has the same size and weight than center of mass would be average position of all balls.

In the picture above, in the first example with 3 objects, you can see that center of mass much closer to the two objects that form small cluster. The same idea with word vectors. Each word is an object and sentence (or tweet) is just a set of these objects. If many vectors from the tweet close to each other in space than the overall average will be close to this cluster and would be a good representation of the tweet.

One remark is that taking average can be the same as just summing vectors, because in most cases you will use cosine similarity for finding close vectors. And with cosine similarity, dividing vector by $n$ is the same as multiplying it by $1/n$ which is a scalar and scale of the vector doesn't matter if you measure distance using angles.

• Beautiful answer – Vikram Murthy Sep 7 '20 at 3:13

This means that embedding of all words are averaged, and thus we get a 1D vector of features corresponding to each tweet. This data format is what typical machine learning models expect, so in a sense it is convenient.

However, this should be done very carefully because averaging does not take care of word order. For example:

our president is a good leader he will not fail

our president is not a good leader he will fail


Have the same words, and therefore will have same average word embedding, but the tweets have very different meaning.

Edit:

To address this issue, one might look into: Sentence-BERT

Here is quick illustration with the above example:

from sentence_transformers import SentenceTransformer
import numpy as np

def cosine_similarity(sentence_embeddings, ind_a, ind_b):
s = sentence_embeddings
return np.dot(s[ind_a], s[ind_b]) / (np.linalg.norm(s[ind_a]) * np.linalg.norm(s[ind_b]))

model = SentenceTransformer('bert-base-nli-mean-tokens')

s0 = "our president is a good leader he will not fail"
s1 = "our president is not a good leader he will fail"
s2 = "our president is a good leader"
s3 = "our president will succeed"

sentences = [s0, s1, s2, s3]

sentence_embeddings = model.encode(sentences)

s = sentence_embeddings

print(f"{s0} <--> {s1}: {cosine_similarity(sentence_embeddings, 0, 1)}")
print(f"{s0} <--> {s2}: {cosine_similarity(sentence_embeddings, 0, 2)}")
print(f"{s0} <--> {s3}: {cosine_similarity(sentence_embeddings, 0, 3)}")


Result:

our president is a good leader he will not fail <--> our president is not a good leader he will fail: 0.46340954303741455
our president is a good leader he will not fail <--> our president is a good leader: 0.8822922110557556
our president is a good leader he will not fail <--> our president will succeed: 0.7640182971954346

• You made a good point there. Any suggestions on what approaches to take in order to avoid such a trap? – Krishnang K Dalal Apr 15 '20 at 18:07

You have a tweet $T$, which is composed of words $w_1,w_2,\cdots,w_n$. Each word has a word2vec embedding $u_{w_1},u_{w_2},..,u_{w_n}$. So you define the tweet embedding as: $u_T:=\frac{1}{n}\sum_{i=1}^nu_{w_i}$.