I'm a beginner in neural networks, and I'm trying to improve my fully connected neural network. I'm training on the MNIST handwritten digits set, so I have 784 input neurons (28x28 images), 10 output neurons (using one-hot encoding), and 2 hidden layers with around 30 units each.
The nonlinearity function I'm using in between the layers right now is the sigmoid function, and I'm using cross entropy loss function, and it is working. After training for a few minutes, I get about 95% accuracy on the test data.
However, I did some research, and it seems that sigmoid function is highly not recommended. By the look of things, I should use softmax activation with cross entropy loss for the output neurons, and Rectified Linear Units (ReLUs) for the hidden neurons. Is this actually the way to go though? If not, what other improvements should I make?
I will be implementing some L2 regularization later on, but for now, I just want to make sure that switching away from sigmoid will benefit me. I also know that Convolutional Neural Networks are better, but my goal right now is to optimize using a fully connected neural network.