I'm trying to embed roughly 60 million phrases into a vector space, then calculate the cosine similarity between them. I've been using sklearn's CountVectorizer with a custom built tokenizer function that produces unigrams and bigrams. Turns out that to get meaningful representations, I have to allow for a tremendous number of columns, linear in the number of rows. This leads to incredibly sparse matrices and is killing performance. It wouldn't be so bad if there were only around 10,000 columns, which I think is pretty reasonable for word embeddings.

I'm thinking of trying to use Google's word2vec because I'm pretty sure it produces much lower dimensional and more dense embeddings. But before that, are there any other embeddings that might warrant a look at first? The key requirement would be being able to scale around 60 million phrases (rows).

I'm pretty new to the field of word embeddings so any advice would help.

I should also add that I'm already using singular value decomposition to improve performance.

  • $\begingroup$ You are using Spark? $\endgroup$ – eliasah Sep 22 '15 at 21:37
  • 1
    $\begingroup$ That's one of the reasons I have suggested Spark at first. I'm sorry, I'm on my phone. I don't have access to any reference whatsoever concerning pre-embedding PCA techniques. $\endgroup$ – eliasah Sep 23 '15 at 0:09
  • 1
    $\begingroup$ I'm not sure that it is an overkill with that amount of data. $\endgroup$ – eliasah Sep 23 '15 at 0:13
  • 1
    $\begingroup$ Removing superfluous tokens shouldn't reduce the dimension by much since you are working texts. Considering a 150000 word dictionary, removing stop words per example would benefit you with a couple of dozen. That won't help. $\endgroup$ – eliasah Sep 23 '15 at 0:16
  • 1
    $\begingroup$ Otherwise, you might want to consider topics modeling with Latent Dirichlet Allocation to reduce your text vector size per phrase. $\endgroup$ – eliasah Sep 23 '15 at 0:24

There's been some work recently on dynamically assigning word2vec (skip gram) dimension using Boltzmann machines. Check out this paper:

"Infinite dimensional word embeddings" -Nalsnick, Ravi

The basic idea is to let your training set dictate the dimensionality of your word2vec model, which is penalized by a regularization term that's related to the dimension size.

The above paper does this for words, and I'd be curious to see how well this performs with phrases.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.