2
$\begingroup$

I'm a beginner in neural networks, and I'm trying to improve my fully connected neural network. I'm training on the MNIST handwritten digits set, so I have 784 input neurons (28x28 images), 10 output neurons (using one-hot encoding), and 2 hidden layers with around 30 units each.

The nonlinearity function I'm using in between the layers right now is the sigmoid function, and I'm using cross entropy loss function, and it is working. After training for a few minutes, I get about 95% accuracy on the test data.

However, I did some research, and it seems that sigmoid function is highly not recommended. By the look of things, I should use softmax activation with cross entropy loss for the output neurons, and Rectified Linear Units (ReLUs) for the hidden neurons. Is this actually the way to go though? If not, what other improvements should I make?

I will be implementing some L2 regularization later on, but for now, I just want to make sure that switching away from sigmoid will benefit me. I also know that Convolutional Neural Networks are better, but my goal right now is to optimize using a fully connected neural network.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

sigmoid is costly to compute compared to ReLU, and has higher chance of falling into vanishing gradient problem, when propagated over layers. ReLU has the disadvantage of dying neurons, so I think better to use leaky ReLU instead, together with Batch Normalization. And, if could apply dropout, it'd decrease your chances of overfitting. Output softmax is a good choice, and I think switching from sigmoid will definitely benefit because at least you'll have faster results and have larger set of options to try out in the mean time.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the help. I implemented leaky ReLU and softmax, which improved things. I also applied a factor of sqrt(2/n) to my initial weights after I did further research. I tried to read the batch normalization paper by Ioffe and Szegedy, but I couldn't quite understand how to implement it. Can you direct me to resources or explain it in simpler terms? Thanks. $\endgroup$
    – Haji Giray
    Commented Mar 3, 2019 at 9:10
  • $\begingroup$ There are various medium articles trying to explain the concept although I can't point the best (all seem to lack some aspect). Reading a couple of them helps. I think for implementation you can refer to mxnet's doc: gluon.mxnet.io/chapter04_convolutional-neural-networks/… $\endgroup$
    – gunes
    Commented Mar 3, 2019 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.