3
$\begingroup$

I have a data-set of 4184x20 and 4184x1, 20 are input parameters while 1 is my target variable. When I first trained my network, it gave a specific value of m.s.e, but a strange thing happened. When I re-arranged the variables or simply swapped the position of some variables, my NN performance has changed. Why is that so? Can someone guide me, I am new to this method. Thank you

$\endgroup$
  • $\begingroup$ Did you retrain it after the swapping? $\endgroup$ – bayerj Dec 22 '16 at 15:31
  • $\begingroup$ Basically I made the initial weight of the neural network to be constant i.e. '1'. So every time I train my NN, it gives same mse. However, mse changed by reordering the variables. Also giving the same result whenever retrained since weight was constant $\endgroup$ – Syed Subhan Ahsen Dec 22 '16 at 15:45
  • $\begingroup$ Reordering the input variables and getting wildly different results definitely should not happen, since MLP are considered "permutation invariant wrt the inputs". I suggest you put more detail in your question: what exactly are you doing, what software and what kind of MLP are you using? $\endgroup$ – bayerj Dec 22 '16 at 15:58
  • $\begingroup$ I have a set of 20 inputs and applied variable reduction techniques on them. Created 2 reduced models say 'a' and 'b' and want to test their accuracy against the model which had all inputs present i.e. 20,for this I used matlab gui, I added this line of code to make initial conditions same every time I train, otherwise I can't compare my models efficiently based on different initial conditions. 'RandStream.setDefaultStream(RandStream('mt19937ar','seed',1));' But when I reordered my inputs, I came up with the question posted. Even with the same initial conditions, getting different performance. $\endgroup$ – Syed Subhan Ahsen Dec 22 '16 at 16:28
  • $\begingroup$ Please clarify whether you a) trained a set of network weights on the original data and then found test mse by feeding it re-ordered data or b) trained one set of weights with the original data, recorded its training/test mse and then trained another set of weights with the re-ordered data and then recorded its training/test mse. Please update your question so everyone is clear about what problem we are discussing. $\endgroup$ – highBandWidth Mar 10 '17 at 21:58
2
$\begingroup$

When I re-arranged the variables or simply swapped the position of some variables, my NN performance has changed. Why is that so?

Most neural networks are sensitive to the ordering of the input variables. For example, if you consider this neural network:

enter image description here

You'll see that the output when $(x,y)= (1,2)$ isn't equal to the the output when $(x,y)= (2,1)$.

Note that some neural networks aren't sensitive to the ordering of the input variables, for example:

enter image description here


Related: Why is the cost function of a neural network non-convex?:

(written by Abhinav, user contributions licensed under cc by-sa 3.0) If you permute the neurons in the hidden layer and do the same permutation on the weights of the adjacent layers then the loss doesn't change. Hence if there is a non-zero global minima as a function of weights, then it can't be unique since the permutation of weights gives another minima. Hence the function is not convex.

$\endgroup$
  • $\begingroup$ Here's one possibility. Back propagation is gradient descent and will find a local minima of the mse function. The local minima one finds depends on the initial conditions. If one keeps all the data constant, one will always get the same local minima using gradient descent. Changing variable order will not effect gradient descent. However, if the guts of the algo is using stochastic gradient descent or some other shortcut, then the local minima is no longer permuation invariant and changing variable order could create different initial conditions. $\endgroup$ – meh Dec 22 '16 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.