Is there a better neural network architecture for this problem? I've got an interesting multiple-response task that I believe a neural network could be useful for solving in an elegant way.
I've got 6 predictors and 3 responses. The 6 predictors can be broken up in to 3 different groups of $\alpha$, $\beta$ and $\gamma$ with 2 columns each. The data is generated such that
$$
f(x_{\alpha_1},x_{\alpha_2},x_{\beta_1},x_{\beta_2},x_{\gamma_1},x_{\gamma_2}) = (y_A, y_B, y_\Gamma)
$$
but there is a structure to the grouping of the columns such that if I were to rearrange the columns (but preserve the inner grouping order), I would obtain
$$
f(x_{\beta_1},x_{\beta_2},x_{\gamma_1},x_{\gamma_2},x_{\alpha_1},x_{\alpha_2}) = (y_B,y_\Gamma,y_A)
$$
I note that the situation is not as simple as just creating a model for $f(x_{\alpha_1},x_{\alpha_2}) = y_A$ and applying it to the other sets of predictors because $y_A$ also depends on $x_{\beta_1},x_{\beta_2},x_{\gamma_1},x_{\gamma_2}$.
In my situation, the relationship between the predictors and the responses $f$ is nonlinear, but very low noise, so I could naively plug all the data in to a big fully connected neural network and with some tuning, get a very good fit.
However, I imagine there is a very clever neural network architecture that could be used here that makes use of weight sharing or something similar because the order of the columns/inputs only matters to an extent. I imagine the benefits of an improved architecture here would allow it to learn from fewer samples and perhaps the constraint demonstrated in the equations above would make it more robust?
Is there some fancy architecture or trick that could or should be employed here?
EDIT: It also turns out that the sum of the responses is equal to 1, which might also be useful in designing a better architecture. $$\sum_{i \in \{A,B,\Gamma\}} y_i = 1$$
 A: Yes, your intuition is right, you definitely want to exploit known symmetries. You can try to do it with a clever model architecture, but one of the easiest ways to encode some symmetries in neural networks is to use them as data augmentations. I.e. if you have one record with one ordering of inputs, but a different ordering should give them same answers (just with re-ordered answers), then just mix this up for training. Either randomly, or feeding all the permutations you know the answer for into the network in each epoch. Does this lead to overfitting? No, as long as you regularize the NN appropriately (e.g. right amount of dropout and weight decay) with the extent chosen according to a proper cross-validation. "Proper" in this case means that you don't make the mistake of augmenting the data and then randomly splitting, but instead you split first and then augment (or in some other way make sure you do not accidentally put an original record and an augmented version in both the training and validation part of a CV-split).
Using the same weights an initial layer or so for each pair of xs certainly sounds like something to try, even if you do the data augmentation as suggested above.
Other ways to encode the symmetries via the model architecture is by using a transformer neural network. They are of course most famous for being used in language modeling where order matters, but people do in fact introduce positional encodings to ensure the model respects order, which is not what one would want here. However, they usually take categorical input, so this would not be a simple basic tweak in a standard situation. I guess you could also consider it a graph with 3 nodes that are all adjacent and use a graph neural network, but I don't have much intuition for those (so this is more speculative on my part).
Other thoughts on architecture: Featuring all of fully connected (linear), BatchNorm, and drop-out layers seems to be relatively standard for neural networks for tabular data (see e.g. what the defaults of the fastai library are). So, perhaps an architecture that combines three separate inputs going through a layer or two with identical weigths, followed by some more linear-BN-drop-out layers could be an obvious try. Another architecture that has got some attention (pun intended) on Kaggle is TabNet that sort of tries to do boosting (just with neural networks).
Regarding outputs summing to 1, if they additionally are all in 0 to 1, then applying a softmax to the outputs would seem like an obvious thing to do (you can also try not predicting one of them, but calculating the softmax with that one set to, say, 0). If they are anywhere in $(-\infty, \infty)$, then only predicting 5 of the 6 and setting the 6th to $1-\sum_{i=1}^5 \hat{y}_i$ would seem to achieve this constraint.
Another question is how to pre-process inputs for the network. I don't know much about what your inputs are, but standardizing them, or applying a rank-Gauss transformation has sometimes been useful for continuous data (while for categorical data, it's of course embeddings).
