1
$\begingroup$

I recently finished my implementation of a multilayer artificial neural network in Julia; I train it with SGD (no momentum, no decay, no regularization, just basic SGD), computing the gradient by means of backpropagation. The weights and biases are initialized by sampling a standard normal distribution.

I tested the implementation against the MNIST dataset, and it takes 5000 iterations for a network with one hidden layer to get an accuracy over 0.90 on the test set. I've seen Python+Numpy examples online that achieve the same accuracy in just a few iterations, using what seems to be the same algorithm with the same hyperparameters. In addition, the accuracy reported in these sites is over 0.95, while I can only achieve an accuracy of 0.93 tops; sometimes I may even get an accuracy around 0.50 or 0.60.

What puzzles me is that the network seems to work, even if convergence is slow. An accuracy between 0.90 and 0.93 seems like an acceptable result for a network of these characteristics; however, the Python implementation seems to converge in much less iterations, and the accuracy seems to be a bit better on average.

Questions: For a good implementation of a multilayer neural network, using a mini batch size of 20, and a learning rate of 3.0, how many iterations should be required to classify the MNIST test set with an accuracy over 0.90 for most initializations? What kind of bug could cause slow convergence but an acceptable accuracy? Could the delta between the maximum accuracy I get with my implementation and the accuracy reported with the Python implementation be due to the stochastic nature of the initialization procedure?

|cite|improve this question
$\endgroup$
  • $\begingroup$ 3.0 seems pretty high for a learning rate. Also, pure SGD is not expected to be that fast. You should consider implementing momentum and regularization. $\endgroup$ – rcpinto Mar 27 '16 at 21:20
  • $\begingroup$ Thanks. In fact, I did consider that, they are in my list of features to add. What puzzles me is the fact that some sites online do report that kind of convergence speed with pure SGD, and with a learning rate of 3.0 indeed. I just wanted to make sure that my results make sense and I don't have some kind of obscure bug hidden in the code. For example, before I was computing the derivative of the sigmoid as a quotient of exponentials, which was completely ruining convergence because of numerical issues. $\endgroup$ – dsans Mar 27 '16 at 21:48
  • 2
    $\begingroup$ @dsans could you provide a link to these other MNIST implementations for reference, specifically where they use the LR of 3? Also, is 5000 iterations really a lot? If you are referring to minibatches, 5000 minibatches at a batchsize of 20, for the MNIST is roughly 2 epochs (full passes through the training set). Perhaps the more concerning issue is the instability of your learning dynamics. Have you tried the same configuration in Python and do you see the same type of behavior? $\endgroup$ – Indie AI Mar 28 '16 at 13:40
  • 1
    $\begingroup$ Well, this is embarrasing; I had the concepts mixed up. I thought an epoch was just another name for an iteration of gradient descent (that is, a single update of the weights). They were doing 30 epochs, not 30 iterations. When I looked at the Python code I focused in the backpropagation algorithm and I didn't notice what they were really doing. This does clear things up, thank you very much. $\endgroup$ – dsans Mar 28 '16 at 14:00
  • $\begingroup$ @dsans ah i see, good to see that cleared up. we've all been there, no need to be embarrassed. $\endgroup$ – Indie AI Mar 28 '16 at 20:27
2
$\begingroup$

It was my mistake, I had the concepts mixed up: in the examples online they were doing 30 full passes through the training set, not 30 iterations. At 20 samples per iteration, an epoch is 3000 iterations. Then 30 epochs would be 90000 iterations, which is way more than the amount I was using. I just tried with 50000 iterations and I got an accuracy of 0.9467 in the first try.

For reference, one of the Python implementations I mentioned is part of this online book on neural networks: http://neuralnetworksanddeeplearning.com/chap1.html

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.