Why did my neural network performance change when I re-arranged the input variables?

I have a data-set of 4184x20 and 4184x1, 20 are input parameters while 1 is my target variable. When I first trained my network, it gave a specific value of m.s.e, but a strange thing happened. When I re-arranged the variables or simply swapped the position of some variables, my NN performance has changed. Why is that so? Can someone guide me, I am new to this method. Thank you

• Did you retrain it after the swapping? – bayerj Dec 22 '16 at 15:31
• Basically I made the initial weight of the neural network to be constant i.e. '1'. So every time I train my NN, it gives same mse. However, mse changed by reordering the variables. Also giving the same result whenever retrained since weight was constant – Syed Subhan Ahsen Dec 22 '16 at 15:45
• Reordering the input variables and getting wildly different results definitely should not happen, since MLP are considered "permutation invariant wrt the inputs". I suggest you put more detail in your question: what exactly are you doing, what software and what kind of MLP are you using? – bayerj Dec 22 '16 at 15:58
• I have a set of 20 inputs and applied variable reduction techniques on them. Created 2 reduced models say 'a' and 'b' and want to test their accuracy against the model which had all inputs present i.e. 20,for this I used matlab gui, I added this line of code to make initial conditions same every time I train, otherwise I can't compare my models efficiently based on different initial conditions. 'RandStream.setDefaultStream(RandStream('mt19937ar','seed',1));' But when I reordered my inputs, I came up with the question posted. Even with the same initial conditions, getting different performance. – Syed Subhan Ahsen Dec 22 '16 at 16:28
• Please clarify whether you a) trained a set of network weights on the original data and then found test mse by feeding it re-ordered data or b) trained one set of weights with the original data, recorded its training/test mse and then trained another set of weights with the re-ordered data and then recorded its training/test mse. Please update your question so everyone is clear about what problem we are discussing. – highBandWidth Mar 10 '17 at 21:58

You'll see that the output when $(x,y)= (1,2)$ isn't equal to the the output when $(x,y)= (2,1)$.