I tried an artificial neural network (ANN) model. Using same data set, it gives a different answer every time I run it in MATLAB. Does anyone why this happens and can suggested the best way to analyze the result?


Different model runs producing different outcomes is a common outcome with ANNs. The response surface over which the minimization takes places is nonconvex and many (all?) ANN estimation methods are sensitive to the starting point of the optimizer, among other inputs.

People who are new to neural networks are often surprised by non-determinism, since it stands in contrast to strongly convex optimization problems like OLS or logistic regression, where global minima can reliably be found (so long a certain regularity conditions are met).

To fix in your mind how this works, consider some random initialization $x^{(0)}$. You supply a mini-batch to the neural network and compute the gradient and apply the update, so now you have $x^{(1)} = x^{(0)} - \eta \nabla f(x^{(0)})$ where $\eta$ is the step size.

Now you start over and make a new random initialization $\tilde{x}^{(0)} \neq x^{(0)}$. Now the gradient update is given by $\tilde{x}^{(1)} = \tilde{x}^{(0)} - \eta \nabla f(\tilde{x}^{(0)})$. In general, we will have $\tilde{x}^{(1)} \neq x^{(1)}$: we started at a different spot on the optimization surface each time, and also we used stochastic gradient descent to take a step in the direction of the expected gradient, where the expectation is estimated using a small sample of the training data. Each of these conditions implies that we are likely to end up at a different location: we started in 2 different places ($x^{(0)}$ or $\tilde{x}^{(0)}$), and we moved in 2 different directions (the value of $\nabla f$ is different).

You could fix the model to start optimization at a particular point, which would remove one source of randomness, but perhaps not all sources of randomness (such as SGD, or dropout, or the random sampling step in a variational auto-encoder). Fixing the random seed should make the results reproducible in all places that seed is used. Depending on the software, use of third party libraries or similar corner-cases could mean that you have to set the random seed separately for each software component.

  • $\begingroup$ IF doing this the ANN output result will be consistent? $\endgroup$
    – bbadyalina
    Jan 16 '17 at 23:26
  • $\begingroup$ @bbadyalina Not necessarily. To produce a consistent output, you would have to fix every random component of the model estimation, and it is not clear that the only random component of the model that you are estimating is initialization. The last paragraph provides an example of a second source of randomness: mini-batching. $\endgroup$
    – Sycorax
    Jan 17 '17 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.