I am generating 10000 matrices of size 12x11 whereby each row corresponds to some summary statistics. I have a multi-lable classification problem. When I train and test my convolution neural network in which the summary statistics are arranged in a certain order, I get 94% as test accuracy. But when I shuffle the rows i.e. even if I exchange just two rows, the overall accuracy of my neural network decreases a lot. Why is that? And how can one make network more robust to these perturbations? One way I thought was to make different images in which I take into account different orders of summary statistics. But that is too many permutations and takes an extremely long time to create these matrices. Can someone suggest how to make my network more robust such that it is still able to classify a matrix regardless of order of summary statistics.
Using a convolutional network is a bad idea, because
The inductive bias of CNNs is that local elements (read: pixels / table entries) are more relevant than distant elements, and that the data is spatially structured at multiple-scales -- from the pixel level up to globally coherent patterns and objects.
A table where the row order means nothing because you can permute them, and the column order also presumably means nothing because they are just arbitrarily ordered statistics satisfies none of the criteria which CNNs are designed for.