What exactly is meant by isotropic and anisotropic with word vectors From this paper https://aclanthology.org/D19-1006.pdf "How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings"
When they say word vectors are anisotropic, do they mean all embeddings for words in the vocabulary are distributed in this cone shape in semantic space, and so there is a cosine similarity of 0.99 often between two different words from two different context? So a cosine similarity between the same word of two different contexts of 0.99 is not special?
If so, isn't it a problem that if there is a large semantic space, the vectors for the whole vocabulary are clustering to a region of the space?
 A: It seems that the authors are writing about all words in the vocabulary. From the paper:

In all layers of all three models, the contextualized word representations of all words
are not isotropic: they are not uniformly distributed with respect to direction. Instead,
they are anisotropic, occupying a narrow
cone in the vector space. The anisotropy in
GPT-2’s last layer is so extreme that two random words will on average have almost perfect cosine similarity!

Anisotropy might be a problem; takings steps to improve isotropy has been found to improve models. From the paper:

Our findings offer some new directions for future
work. For one, as noted earlier in the paper, Mu
et al. (2018) found that making static embeddings
more isotropic – by subtracting their mean from
each embedding – leads to surprisingly large improvements in performance on downstream tasks.
Given that isotropy has benefits for static embeddings, it may also have benefits for contextualized word representations, although the latter have
already yielded significant improvements despite
being highly anisotropic. Therefore, adding an
anisotropy penalty to the language modelling objective – to encourage the contextualized representations to be more isotropic – may yield even better
results.

A: Anisotropy is a problem if you treat the embeddings as vectors in Euclidian space. If your vocabulary ends up effectively in a tiny volume inside Euclidian space, then you shouldn't treat the vectors as Euclidian vectors. This is consequential for distance metrics used in your problem. This also means that perhaps the embedding could be compressed (shortened), that it has too many dimensions.
The authors claim that the entire vocabulary's embeddings' distribution is not isotropic.
