I am making a Neural Network with data that has images as input and a 4 possible labels as output. I created a 3 Layer Network with 1 hidden layer. The output layer uses the Softmax Activation Function and the Hidden Layer was using Sigmoid. The network uses Adam Optimizer and Cross-entropy loss function.

This network with Sigmoid activation function in hidden layer was achieving approximately 50% accuracy on test data. But when I replaced it with ReLU Activation Function, the network was achieving 85% accuracy. Am I achieving accurate results? Or is it not right to use ReLU in this case. Or must I calculate the accuracy differently?? And what properties of ReLU can cause this boost in accuracy?

Here is my Tensorflow code for this:

Line 2 is Sigmoid, and Line 3 is ReLU (commented out)

hiddenOut = tf.add(tf.matmul(x, W1), b1) 
hiddenOut = tf.nn.sigmoid(hiddenOut) #SIGMOID --> gets ~ 50% accuracy
##hiddenOut = tf.nn.relu(hiddenOut) #RELU ---> gets ~ 85% accuracy

logits = tf.add(tf.matmul(hiddenOut, W2), b2);

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer(0.001).minimize(cross_entropy)

#define an accuracy asessment operation
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(logits, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
  • $\begingroup$ I read that but I still don't understand why/how I got a 35 percent increase by using ReLU $\endgroup$
    – Amaan
    Nov 6, 2017 at 2:38
  • $\begingroup$ It's hard to say without more information about the data and what's happening to the loss and gradients during training, but I think either of the two explanations posted here are plausible: stats.stackexchange.com/a/126362/17760 $\endgroup$ Nov 6, 2017 at 2:47

1 Answer 1


To echo the sentiment of the comments, it's hard to say definitely why ReLU provides a better model.

Given it's a 3-layer network it shouldn't really make a difference whether you use ReLU or a sigmoid. ReLU are helpful with the vanishing gradient problem which is relevant to deep networks.

My guess it might just be a lucky initialisation state. Do you see this accuracy consistently?

  • $\begingroup$ I think the vanishing gradient problem is the main case where to use ReLU, it was indeed historically used to solve deep layered networks backpropagation slow convergence (Lecun, Hinton, etc). $\endgroup$
    – gaborous
    Feb 10, 2018 at 22:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.