While brainstorming on a new project, I stumbled upon a problem. I would like to create a neural network to encode a sequence of objects and get the gist of the sequence. However, the sequence is of variable-length and order-insensitive, that is, the order of the objects should not matter to the output.

The variable-length part can be solved using padding, but the order-insensitive property troubles me. I've looked at LSTM, GRU and Transformer networks but they are seq2seq encoders and are not order-insensitive.

My search on the topic led to nothing. Do you know of any solution for this use-case ?

Thank you

  • $\begingroup$ Wait... if it's order-insensitive, doesn't it mean it's pretty much just your bog-standard tensor? $\endgroup$ – jkm Dec 7 '19 at 16:17
  • $\begingroup$ transformer without position embeddings should be order inv. $\endgroup$ – shimao Dec 7 '19 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.