I am making a Neural Network with data that has images as input and a 4 possible labels as output. I created a 3 Layer Network with 1 hidden layer. The output layer uses the Softmax Activation Function and the Hidden Layer was using Sigmoid. The network uses Adam Optimizer and Cross-entropy loss function.
This network with Sigmoid activation function in hidden layer was achieving approximately 50% accuracy on test data. But when I replaced it with ReLU Activation Function, the network was achieving 85% accuracy. Am I achieving accurate results? Or is it not right to use ReLU in this case. Or must I calculate the accuracy differently?? And what properties of ReLU can cause this boost in accuracy?
Here is my Tensorflow code for this:
Line 2 is Sigmoid, and Line 3 is ReLU (commented out)
hiddenOut = tf.add(tf.matmul(x, W1), b1)
hiddenOut = tf.nn.sigmoid(hiddenOut) #SIGMOID --> gets ~ 50% accuracy
##hiddenOut = tf.nn.relu(hiddenOut) #RELU ---> gets ~ 85% accuracy
logits = tf.add(tf.matmul(hiddenOut, W2), b2);
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer(0.001).minimize(cross_entropy)
#define an accuracy asessment operation
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(logits, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
