Feeding a set of words to neural network

Question

I know that LSTM (or also CNN) is good choice to input a word sequence such as $(x_1, x_2, ... x_T)$. But I could not find a good practice to input a set of vectors $\{x_i\}_{i=1,2,3,...,T}$, where the order of vector does not have sense.

Let's say, if you have a sentence like "The president gave up to build a wall.'' on document classification, you may feed the sentence to LSTM to know the category of the sentence such as "politics''.

On the other hand, if you have a set of words like {wall, president}, how do you feed these words into a model?

One naive way is calculating embeddings of those words and summing up the embedding vectors to get a feature to feed a model. But, I do not think this feature does not work well.

Answer 1

If you want to learn $f(X)$ where $X$ is a set, that is the order of elements in the set is inconsequential, then check out the work on Deep Sets [A].

People have handled such things even using LSTMs in the following way:

1. Given a set X, say with some label y
2. Randomly sample elements from X to create a sequence
3. Generate multiple such sequences from the same set X
4. Feed all these sequences to LSTM with the same label y

These methods have been used particularly in Graph Embedding methods like GraphSage [B].

By the way, average of word embedding as a set representation is a fairly competitive model, so don't dismiss it. In [C], Arora et. al show that a simple weighted average of word embeddings for representing sentences competes amazingly well against LSTMs etc. for many tasks.

[A] Paper: https://arxiv.org/abs/1703.06114
Code: https://github.com/manzilzaheer/DeepSets
[B] Paper: https://arxiv.org/abs/1706.02216
[C] Paper: https://openreview.net/forum?id=SyK00v5xx