After training word vectors with word2vec, is it better to normalize them before using them for some downstream applications? I.e what are the pros/cons of normalizing them?
$\begingroup$
$\endgroup$
2
asked Oct 20, 2015 at 23:56
-
$\begingroup$ in a similarity task, normalization improved my system performance a little bit. $\endgroup$– keramatMay 20, 2019 at 9:16
-
$\begingroup$ Related: stackoverflow.com/q/36034454/1709587 $\endgroup$– Mark AmeryMay 20, 2019 at 10:17
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
When the downstream applications only care about the direction of the word vectors (e.g. they only pay attention to the cosine similarity of two words), then normalize, and forget about length.
However, if the downstream applications are able to (or need to) consider more sensible aspects, such as word significance, or consistency in word usage (see below), then normalization might not be such a good idea.
From Levy et al., 2015 (and, actually, most of the literature on word embeddings):
Vectors are normalized to unit length before they are used for similarity calculation, making cosine similarity and dot-product equivalent.
Also from Wilson and Schakel, 2015:
Most applications of word embeddings explore not the word vectors themselves, but relations between them to solve, for example, similarity and word relation tasks. For these tasks, it was found that using normalised word vectors improves performance. Word vector length is therefore typically ignored.
Normalizing is equivalent to losing the notion of length. That is, once you normalize the word vectors, you forget the length (norm, module) they had right after the training phase.
However, sometimes it's worth to take into consideration the original length of the word vectors.
Schakel and Wilson, 2015 observed some interesting facts regarding the length of word vectors:
A word that is consistently used in a similar context will be represented by a longer vector than a word of the same frequency that is used in different contexts.
Not only the direction, but also the length of word vectors carries important information.
Word vector length furnishes, in combination with term frequency, a useful measure of word significance.
-
$\begingroup$ Can we elaborate "it was found that using normalised word vectors improves performance"? Isn't normalization involves additional computation? $\endgroup$– neuriteMar 9, 2017 at 20:39
-
6$\begingroup$ @neurite, it that context, a better performance refers to a better score on the evaluation tasks. $\endgroup$ Mar 10, 2017 at 22:37
-
1$\begingroup$ I was making an out-of-domain input text detection model by training a
OneClassSVMondoc2vecfeatures. Personally I found, scaling actually was a disaster. The model started to find each input to be in-domain even if they were not. But when I trained without scaling the vectors the model performance was fair. $\endgroup$– hafiz031Jun 22, 2021 at 9:26 -
$\begingroup$ I think once you normalize all vectors to 1, you basically reduce the effective embedding space by 1 dimension, i.e. S^N -> S^(N-1). $\endgroup$ Mar 2, 2022 at 23:26
-
$\begingroup$ By normalizing the vectors before performing similarity calculations, you make the calculations faster and more efficient. This speedup is what was probably meant by 'improved performance'. $\endgroup$ Oct 19, 2022 at 21:24
$\begingroup$
$\endgroup$
This should be a comment on @turdus-merula's answer, but I don't have enough reputation. In Section 6 of (Schnabel et al., 2015), the authors found that length-normalized word embeddings also encodes the word frequency information. This could be a counter argument on (Schakel and Wilson, 2015)'s:
Word vector length furnishes, in combination with term frequency, a useful measure of word significance.
