4
$\begingroup$

From this paper https://aclanthology.org/D19-1006.pdf "How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings"

When they say word vectors are anisotropic, do they mean all embeddings for words in the vocabulary are distributed in this cone shape in semantic space, and so there is a cosine similarity of 0.99 often between two different words from two different context? So a cosine similarity between the same word of two different contexts of 0.99 is not special?

If so, isn't it a problem that if there is a large semantic space, the vectors for the whole vocabulary are clustering to a region of the space?

$\endgroup$

2 Answers 2

6
$\begingroup$

It seems that the authors are writing about all words in the vocabulary. From the paper:

In all layers of all three models, the contextualized word representations of all words are not isotropic: they are not uniformly distributed with respect to direction. Instead, they are anisotropic, occupying a narrow cone in the vector space. The anisotropy in GPT-2’s last layer is so extreme that two random words will on average have almost perfect cosine similarity!

Anisotropy might be a problem; takings steps to improve isotropy has been found to improve models. From the paper:

Our findings offer some new directions for future work. For one, as noted earlier in the paper, Mu et al. (2018) found that making static embeddings more isotropic – by subtracting their mean from each embedding – leads to surprisingly large improvements in performance on downstream tasks. Given that isotropy has benefits for static embeddings, it may also have benefits for contextualized word representations, although the latter have already yielded significant improvements despite being highly anisotropic. Therefore, adding an anisotropy penalty to the language modelling objective – to encourage the contextualized representations to be more isotropic – may yield even better results.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks, this was not how I had been thinking of word vectors at all, so I had to do a sanity check. $\endgroup$ Commented Sep 21, 2022 at 15:38
4
$\begingroup$

Anisotropy is a problem if you treat the embeddings as vectors in Euclidian space. If your vocabulary ends up effectively in a tiny volume inside Euclidian space, then you shouldn't treat the vectors as Euclidian vectors. This is consequential for distance metrics used in your problem. This also means that perhaps the embedding could be compressed (shortened), that it has too many dimensions.

The authors claim that the entire vocabulary's embeddings' distribution is not isotropic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.