8
$\begingroup$

I want to approximate a multi-variable function $f(x_1,x_2,x_3,x_4,x_5,y)$ from data by neural networks, and $f$ satisfies $f(x_1,\ldots,x_5,y)=f(x_{i_1},\ldots,x_{i_5},y)$, where $(i_1,\ldots,i_5)$ is an arbitrary permutation of $(1,\ldots,5)$.

How can I impose this constraint on networks? Thanks!

$\endgroup$
5

3 Answers 3

3
$\begingroup$

Your question is on the doorstep of geometric deep learning.

A common approach to obtaining a symmetry in a neural network is to place an entire layer in the network which has that symmetry. There are many functions which are permutation invariant to choose from as candidate layers.

$\endgroup$
0
$\begingroup$

The easiest way would be to train a fully connected neural network on randomly ordered inputs, ideally on all permutations.

EDIT:

Alternatively, if you ask for a design of a network that is order invariant, you can do the following: You would need to do a 1D convolutional neural network, where the stride size is equal to the array length of each individual X. You can then stride over the network and assess each X. In the case you present, I assume each X is scalar, so the stride size would be 1.

However, the disadvantage of this approach is, that each Xn would only be considered by itself, independently of all other X. The convolutional layer will have filters that are equally applied to all X, but the different X are not conneted with each other.

Have a look at this example.

$\endgroup$
4
  • $\begingroup$ This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review $\endgroup$
    – Antoine
    Jan 29, 2018 at 14:44
  • $\begingroup$ True, but it provides a solution to the problem. $\endgroup$
    – Nickpick
    Jan 29, 2018 at 14:46
  • $\begingroup$ Updated the answer with another possible solution of a 1D conv net. Now you can upvode ;) $\endgroup$
    – Nickpick
    Jan 29, 2018 at 19:14
  • $\begingroup$ We only have a limited number of options in the review interface. I flagged your original answer because it was too short. Also, I am not the one who downvoted, sorry. $\endgroup$
    – Antoine
    Jan 29, 2018 at 19:48
0
$\begingroup$

This constraint has got to be the same as equal weights on all variables, or that in fact you're dealing with one variable. There's really only one variable in the model.

$\endgroup$
1
  • 1
    $\begingroup$ Maybe by "one variable" you mean "one parameter"? That would be correct if $f$ were linear, but generally it's not correct: even when the $x_i$ are scalars, the space they determine still has $5$ dimensions, not $1.$ $\endgroup$
    – whuber
    Apr 24, 2023 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.