# Can ReLU replace a Sigmoid Activation Function in Neural Network without needing to change other parameters/functions of Network? [duplicate]

I am making a Neural Network with data that has images as input and a 4 possible labels as output. I created a 3 Layer Network with 1 hidden layer. The output layer uses the Softmax Activation Function and the Hidden Layer was using Sigmoid. The network uses Adam Optimizer and Cross-entropy loss function.

This network with Sigmoid activation function in hidden layer was achieving approximately 50% accuracy on test data. But when I replaced it with ReLU Activation Function, the network was achieving 85% accuracy. Am I achieving accurate results? Or is it not right to use ReLU in this case. Or must I calculate the accuracy differently?? And what properties of ReLU can cause this boost in accuracy?

Here is my Tensorflow code for this:

Line 2 is Sigmoid, and Line 3 is ReLU (commented out)

hiddenOut = tf.add(tf.matmul(x, W1), b1)
hiddenOut = tf.nn.sigmoid(hiddenOut) #SIGMOID --> gets ~ 50% accuracy
##hiddenOut = tf.nn.relu(hiddenOut) #RELU ---> gets ~ 85% accuracy

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))

#define an accuracy asessment operation
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(logits, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))