# Ideas for adversarial attack on input data for more robust neural network classifier

I am generating 10000 matrices of size 12x11 whereby each row corresponds to some summary statistics. I have a multi-lable classification problem. When I train and test my convolution neural network in which the summary statistics are arranged in a certain order, I get 94% as test accuracy. But when I shuffle the rows i.e. even if I exchange just two rows, the overall accuracy of my neural network decreases a lot. Why is that? And how can one make network more robust to these perturbations? One way I thought was to make different images in which I take into account different orders of summary statistics. But that is too many permutations and takes an extremely long time to create these matrices. Can someone suggest how to make my network more robust such that it is still able to classify a matrix regardless of order of summary statistics.