I know that LSTM (or also CNN) is good choice to input a word sequence such as $(x_1, x_2, ... x_T)$. But I could not find a good practice to input a set of vectors $\{x_i\}_{i=1,2,3,...,T}$, where the order of vector does not have sense.

Let's say, if you have a sentence like "The president gave up to build a wall.'' on document classification, you may feed the sentence to LSTM to know the category of the sentence such as "politics''.

On the other hand, if you have a set of words like {wall, president}, how do you feed these words into a model?

One naive way is calculating embeddings of those words and summing up the embedding vectors to get a feature to feed a model. But, I do not think this feature does not work well.


If you want to learn $f(X)$ where $X$ is a set, that is the order of elements in the set is inconsequential, then check out the work on Deep Sets [A].

People have handled such things even using LSTMs in the following way:

  1. Given a set X, say with some label y
  2. Randomly sample elements from X to create a sequence
  3. Generate multiple such sequences from the same set X
  4. Feed all these sequences to LSTM with the same label y

These methods have been used particularly in Graph Embedding methods like GraphSage [B].

By the way, average of word embedding as a set representation is a fairly competitive model, so don't dismiss it. In [C], Arora et. al show that a simple weighted average of word embeddings for representing sentences competes amazingly well against LSTMs etc. for many tasks.

[A] Paper: https://arxiv.org/abs/1703.06114
Code: https://github.com/manzilzaheer/DeepSets
[B] Paper: https://arxiv.org/abs/1706.02216
[C] Paper: https://openreview.net/forum?id=SyK00v5xx


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.