0
$\begingroup$

First of all, let me briefly describe my task:
There are about 10^9 sets of words, each sets contains about 10^4 words, which indicates it will be very space consuming to store all those sets in disk. I want to know the number of common words between two sets. However, I don't care what those common words are. I even don't need to know the accurate number which means that an estimated value is enough to me. The key is to find an appropriate representation for these sets so that they could be stored.
A natural idea is to represent a set of words as a vector, and then train a regression NN model whose input are two vectors, which represent two sets, and output is a value which estimates the common words between two input sets. The problem is how to represent these sets as vectors. It's definitely impossible to directly apply the bag of words model, since the dictionary is too big to store. Maybe I can utilize some dimensionality reduction methods on bag of words first to achieve a smaller representation of vector. Or maybe I can apply word embedding to each word, and then sum them up to represent a set. Is there any advice? Or is there any relative work already?

$\endgroup$
0
$\begingroup$

Why don't you do bag of words then?

then take not of only those that are the most common say the 5000 most common words.

You could also apply minimum or maximum word frequency. Which means that if the frequency of the word is less than 10 then you should disregard it, If the frequency of the word is say equals to the number of set of words that you have (which mean s it's in all of the sets), disregard it.

If you want to do document embedding. you can try gensims doc2vec or word2vec. here and here

$\endgroup$
  • $\begingroup$ or you could also use TFIDF for the matter to look for the words that best represent your set $\endgroup$ – Vincent Pakson Oct 17 '17 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.