7
$\begingroup$

The paper that I am reading says,

tweet is represented by the average of the word embedding vectors of the words that compose the tweet.

Does this mean each word in the tweet (sentence) has to be represented as the average of its word vector (still having the same length),

or does it mean the sentence itself has to be the average of all values computed above (average of word vectors of the word that the sentence contains)?

I am confused.

$\endgroup$
  • 1
    $\begingroup$ It'll be hard to answer this without more context. Could you link to the paper and provide a complete citation (in case the link dies)? $\endgroup$ – Sycorax says Reinstate Monica Dec 14 '17 at 18:39
  • 2
    $\begingroup$ It simply means that the author needed a single vector to represent a tweet so that he/she can run a classifier (probably). In other words, averaging of vectors was defined downstream by a tool that accepted a single vector. $\endgroup$ – Vladislavs Dovgalecs Dec 14 '17 at 19:23
  • 1
    $\begingroup$ Can you post name of the paper for context? $\endgroup$ – Enthusiast Feb 1 '18 at 4:59
10
$\begingroup$

You can think of it in terms of physical analogy. You can take a flat surface, like a table, and arrange 30 balls on it. Then you can cut legs from the table and replace it with a single leg. In order to figure out where to put this leg you need to find center of mass of all 30 balls on the table. Assuming that each ball has the same size and weight than center of mass would be average position of all balls.

enter image description here

In the picture above, in the first example with 3 objects, you can see that center of mass much closer to the two objects that form small cluster. The same idea with word vectors. Each word is an object and sentence (or tweet) is just a set of these objects. If many vectors from the tweet close to each other in space than the overall average will be close to this cluster and would be a good representation of the tweet.

One remark is that taking average can be the same as just summing vectors, because in most cases you will use cosine similarity for finding close vectors. And with cosine similarity, dividing vector by $n$ is the same as multiplying it by $1/n$ which is a scalar and scale of the vector doesn't matter if you measure distance using angles.

| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

This means that embedding of all words are averaged, and thus we get a 1D vector of features corresponding to each tweet. This data format is what typical machine learning models expect, so in a sense it is convenient.

However, this should be done very carefully because averaging does not take care of word order. For example:

our president is a good leader he will not fail

our president is not a good leader he will fail

Have the same words, and therefore will have same average word embedding, but the tweets have very different meaning.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You made a good point there. Any suggestions on what approaches to take in order to avoid such a trap? $\endgroup$ – Krishnang K Dalal Apr 15 at 18:07
2
$\begingroup$

You have a tweet $T$, which is composed of words $w_1,w_2,\cdots,w_n$. Each word has a word2vec embedding $u_{w_1},u_{w_2},..,u_{w_n}$. So you define the tweet embedding as: $u_T:=\frac{1}{n}\sum_{i=1}^nu_{w_i}$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.