merging two word embedding models?

Question

I have a task for which I am using word embeddings but because limited data, I can't train. For one part of my task Word2Vec model of Google News is working okay while for the other one, Glove embeddings are working fine. I tried to merge these two by taking the vector of the word from both models and averaging it. However, that seemed to have changed the entire scenario and the results are pretty much random.

One explaination that comes to my mind is that latent features learnt by both models are pretty much different and hence, averaging these two is resulting in a random embedding of words.

Am I right? Is there any other way to merge embeddings from two sources?

Answer 1

Addition doesn't make much sense, since the vectors are actually from different spaces (I don't see a reason that they should even be of the same dimensionality).

Did you try concatenating them?

If yes, did you try to remove possible correlations by performing PCA?

Answer 2

Let's call word2vec vector model $W$ & glove $G$. Now, an embedding is just a vector and $W$ is a vector space. These two embeddings are in different vector spaces. You need to either

1. align the 2 vector spaces like in this paper by Mikolov. The idea is that even though vector spaces are almost isomorphic, they are mostly at an angle and you need to multiply one space, say $G$ with a rotation matrix to align it with $W$. You could do this by initializing a random matrix and performing gradient descent to minimize reconstruction error.

2. or combine them by doing dimensionality reduction like in this paper by Conceptnet by doing something like PCA as answered by @Jakub. From the paper

Our goal is to learn a projection from k dimensions to k ′ dimensions that removes the redundancy that comes from concatenating multiple sources of embeddings.

Suggestions

1. Since your primary problem is handling out-of-vocabulary(OOV) words, try using FastText pretrained embeddings for handling OOV which keeps embeddings for subwords (char n-grams) too; one can use those to build up a word. ($v_{dies} = v_{die} + v_{ies}$)
2. Just use Conceptnet Numberbatch embeddings which itself combine glove & word2vec like you are trying to do