Evaluating Word Embeddings: Expected Cosine Distance

Question

One way to evaluate the quality of word embeddings is with tuples $(a, b, c, d)$ of words of analog relations of $a$ to $b$ and $c$ to $d$, such as

a            b            c          d

Philadelphia Pennsylvania Louisville Kentucky
implement    implementing fly        flying
efficient    efficiently  happy      happily

You can then evaluate the embedding by computing the average cosine distance between $v = c + (b - a)$ and $d$.

$$\frac{1}{n}\sum_{i=1}^n 1 - \frac{v \cdot d}{||v||~||d||}$$

$d$ is basically the "target" vector of $v$. The closer the average distance to the target vector, the better your embedding. In theory.

However, this approach is only useful when comparing embeddings to each other, really. Sure, the closer the average distance is to 0, the better. But what is a bad average distance, what a fair one and what a good one? For that we need to bring in expectation, I assume. So I computed the expected cosine distance for a range of dimensions:

X, Y = [], []
for i in range(1, 1000):
    m = np.random.random((i,1000))
    X.append(i)
    Y.append(np.mean(cosine_distances(m)))

So the value tends towards $\frac{1}{4}$. That means for an embedding of a few 100 dimensions, the expected cosine distance is $\sim .25$.

Ok, so we can express the quality of the embedding as the average distance over the expected distance

$$\frac{4}{n}\sum_{i=1}^n 1 - \frac{v \cdot d}{||v||~||d||}$$

First of all, does that make sense so far?

Now, what actually puzzles me, is that the expected value is so low. For the embeddings I was evaluating, I was getting an average distance of $.68$ for $d$ and $v$. Those embeddings are not great, however I can get at least reasonable predictions from them, for example:

word        distance

accident    .000
crash       .408
collision   .501
mishap      .506
collide     .596
smash       .638
rollover    .641
explosion   .642
fatality    .643
collapse    .670

These are the most similar vectors for accident. They are obviously not random. But their average distance is far above the expected distance of $.25$. This does not really make sense to me. If the expected distance of any two vectors in a random embedding is $.25$, then the average distance between related vectors should be smaller than that, should it not?

I guess I am confused about something here or I made a mistake in my logic of measuring these values.

Can you maybe recognize what my error is here and help me out?

Answer 1

Because the cosine measure is not linear, an embedding can have an expected cosine distance greater than the expected distance in a set of uniformly distributed vectors. When measuring the expectation in a trained embedding, the expectation will be greater than the average distance of $v$ and $d$. The fact that the expectation of a uniform random set of vectors is smaller, has nothing to do with that. Everything is as it should be.