Word2Vec models for irrelevant word order

Question

I'm searching for a ready-to-use model, preferably in TensorFlow, that learns embeddings for words from a corpus, but without taking word order into account.

So far I have a

• vocabulary of 8822 words
• corpus of ~1.5M documents of variable length (mean: 11, median: 8)

The corpus reflects users and items that hold tags (from the vocabulary). My tasks is to learn embeddings of their textual descriptions. I want to build pairs, e.g. a user's tag list is [t1, t2, t3, t4, t5, t6]. But since ordering is considered irrelevant, (t1, t6) are treated equally to (t1, t2) and thus this document yields 6*5=30 positive samples (pairs).

1) How can I apply those in the context of word2vec model which require fixed window sizes?

2) After having learnt the word embeddings, can I just add them up for a given document to compare it with another, e.g. through cosine similarity?

Thank you!

Answer 1

1.

The simplest idea would be to copy the tags several times with different orderings, to make the data (sort of) invariant to permuting tags. In computer vision people do something similar to achieve translation invariance.

You could also use node2vec if you have a meaningful graph structure on your tags (for example a graph where the two tags are adjacent if at least N documents are tagged with both of them).

2.

As mentioned in comments, it makes more sense to average these word vectors. Also, maybe this article can help.

Answer 2

I would suggest you study Word Embeddings a bit. You need to understand the mathematical significance of going from a one-hot encoded space to a dense vector space. However to answer your questions (based on what I understood):

1) Assuming you are talking about English, use some pre-trained Word2Vec model and simply convert all the tags to some fixed vector length, say 10 or 50.

2) Not sure what adding vectors would yield. You might be better off just multiplying all pairs of Doc 1 tags with Doc 2 tags, and then adding their cosine similarities. Like if Doc 1 has 5 tags and Doc 2 has 6 tags, then 30 pairs similarities can be found and those can be used.