Word embedding algorithms in terms of performance

I'm trying to embed roughly 60 million phrases into a vector space, then calculate the cosine similarity between them. I've been using sklearn's CountVectorizer with a custom built tokenizer function that produces unigrams and bigrams. Turns out that to get meaningful representations, I have to allow for a tremendous number of columns, linear in the number of rows. This leads to incredibly sparse matrices and is killing performance. It wouldn't be so bad if there were only around 10,000 columns, which I think is pretty reasonable for word embeddings.

I'm thinking of trying to use Google's word2vec because I'm pretty sure it produces much lower dimensional and more dense embeddings. But before that, are there any other embeddings that might warrant a look at first? The key requirement would be being able to scale around 60 million phrases (rows).

I'm pretty new to the field of word embeddings so any advice would help.

I should also add that I'm already using singular value decomposition to improve performance.

• You are using Spark? – eliasah Sep 22 '15 at 21:37
• That's one of the reasons I have suggested Spark at first. I'm sorry, I'm on my phone. I don't have access to any reference whatsoever concerning pre-embedding PCA techniques. – eliasah Sep 23 '15 at 0:09
• I'm not sure that it is an overkill with that amount of data. – eliasah Sep 23 '15 at 0:13
• Removing superfluous tokens shouldn't reduce the dimension by much since you are working texts. Considering a 150000 word dictionary, removing stop words per example would benefit you with a couple of dozen. That won't help. – eliasah Sep 23 '15 at 0:16
• Otherwise, you might want to consider topics modeling with Latent Dirichlet Allocation to reduce your text vector size per phrase. – eliasah Sep 23 '15 at 0:24