# How to model an "order-invariant" function by neural networks

I want to approximate a multi-variable function $f(x_1,x_2,x_3,x_4,x_5,y)$ from data by neural networks, and $f$ satisfies $f(x_1,\ldots,x_5,y)=f(x_{i_1},\ldots,x_{i_5},y)$, where $(i_1,\ldots,i_5)$ is an arbitrary permutation of $(1,\ldots,5)$.

How can I impose this constraint on networks? Thanks!

• You could try tying the weights. Or, you could randomly permute your existing data so that the symmetry naturally falls out of the data. Aug 27, 2017 at 15:05
• The input is a multiset so a variadic network might be the answer: link.springer.com/chapter/10.1007/978-3-540-74690-4_47 Sep 9, 2018 at 8:12
• Duplicated of ai.stackexchange.com/questions/4655/… ? Mar 10, 2021 at 16:38
• A Graph Neural Network would work Jul 24, 2021 at 18:12
• just wanted to mention set transformers, which I've found useful for permutation invariant/equivariant learning: arxiv.org/abs/1810.00825 Dec 22, 2022 at 5:36

## 3 Answers

Your question is on the doorstep of geometric deep learning.

A common approach to obtaining a symmetry in a neural network is to place an entire layer in the network which has that symmetry. There are many functions which are permutation invariant to choose from as candidate layers.

The easiest way would be to train a fully connected neural network on randomly ordered inputs, ideally on all permutations.

EDIT:

Alternatively, if you ask for a design of a network that is order invariant, you can do the following: You would need to do a 1D convolutional neural network, where the stride size is equal to the array length of each individual X. You can then stride over the network and assess each X. In the case you present, I assume each X is scalar, so the stride size would be 1.

However, the disadvantage of this approach is, that each Xn would only be considered by itself, independently of all other X. The convolutional layer will have filters that are equally applied to all X, but the different X are not conneted with each other.

Have a look at this example.

• This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review Jan 29, 2018 at 14:44
• True, but it provides a solution to the problem. Jan 29, 2018 at 14:46
• Updated the answer with another possible solution of a 1D conv net. Now you can upvode ;) Jan 29, 2018 at 19:14
• We only have a limited number of options in the review interface. I flagged your original answer because it was too short. Also, I am not the one who downvoted, sorry. Jan 29, 2018 at 19:48

This constraint has got to be the same as equal weights on all variables, or that in fact you're dealing with one variable. There's really only one variable in the model.

• Maybe by "one variable" you mean "one parameter"? That would be correct if $f$ were linear, but generally it's not correct: even when the $x_i$ are scalars, the space they determine still has $5$ dimensions, not $1.$
– whuber
Apr 24, 2023 at 21:40