60
$\begingroup$

After training word vectors with word2vec, is it better to normalize them before using them for some downstream applications? I.e what are the pros/cons of normalizing them?

$\endgroup$
2

1 Answer 1

51
$\begingroup$

When the downstream applications only care about the direction of the word vectors (e.g. they only pay attention to the cosine similarity of two words), then normalize, and forget about length.

However, if the downstream applications are able to (or need to) consider more sensible aspects, such as word significance, or consistency in word usage (see below), then normalization might not be such a good idea.


From Levy et al., 2015 (and, actually, most of the literature on word embeddings):

Vectors are normalized to unit length before they are used for similarity calculation, making cosine similarity and dot-product equivalent.

Also from Wilson and Schakel, 2015:

Most applications of word embeddings explore not the word vectors themselves, but relations between them to solve, for example, similarity and word relation tasks. For these tasks, it was found that using normalised word vectors improves performance. Word vector length is therefore typically ignored.

Normalizing is equivalent to losing the notion of length. That is, once you normalize the word vectors, you forget the length (norm, module) they had right after the training phase.

However, sometimes it's worth to take into consideration the original length of the word vectors.

Schakel and Wilson, 2015 observed some interesting facts regarding the length of word vectors:

A word that is consistently used in a similar context will be represented by a longer vector than a word of the same frequency that is used in different contexts.

Not only the direction, but also the length of word vectors carries important information.

Word vector length furnishes, in combination with term frequency, a useful measure of word significance.

$\endgroup$
5
  • $\begingroup$ Can we elaborate "it was found that using normalised word vectors improves performance"? Isn't normalization involves additional computation? $\endgroup$
    – neurite
    Mar 9, 2017 at 20:39
  • 6
    $\begingroup$ @neurite, it that context, a better performance refers to a better score on the evaluation tasks. $\endgroup$ Mar 10, 2017 at 22:37
  • 1
    $\begingroup$ I was making an out-of-domain input text detection model by training a OneClassSVM on doc2vec features. Personally I found, scaling actually was a disaster. The model started to find each input to be in-domain even if they were not. But when I trained without scaling the vectors the model performance was fair. $\endgroup$
    – hafiz031
    Jun 22, 2021 at 9:26
  • $\begingroup$ I think once you normalize all vectors to 1, you basically reduce the effective embedding space by 1 dimension, i.e. S^N -> S^(N-1). $\endgroup$
    – Louis Yang
    Mar 2, 2022 at 23:26
  • $\begingroup$ By normalizing the vectors before performing similarity calculations, you make the calculations faster and more efficient. This speedup is what was probably meant by 'improved performance'. $\endgroup$ Oct 19, 2022 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.