I was playing with a simple Neural Network with only one hidden layer, by Tensorflow, and then I tried different activations for the hidden layer:

  • Relu
  • Sigmoid
  • Softmax (well, usually softmax is used in the last layer..)

Relu gives the best train accuracy & validation accuracy. I am not sure how to explain this.

We know that Relu has good qualities, such as sparsity, such as no-gradient-vanishing, etc, but

Q: is Relu neuron in general better than sigmoid/softmax neurons ? Should we almost always use Relu neurons in NN (or even CNN) ? I thought a more complex neuron would introduce better result, at least train accuracy if we worry about overfitting.

Thanks PS: The code basically is from "Udacity-Machine learning -assignment2", which is recognition of notMNIST using a simple 1-hidden-layer-NN.

batch_size = 128
graph = tf.Graph()
with graph.as_default():
  # Input data. 
  tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, image_size * image_size))
  tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
  tf_valid_dataset = tf.constant(valid_dataset)
  tf_test_dataset = tf.constant(test_dataset)

  # hidden layer
  hidden_nodes = 1024
  hidden_weights = tf.Variable( tf.truncated_normal([image_size * image_size, hidden_nodes]) )
  hidden_biases = tf.Variable( tf.zeros([hidden_nodes]))
  hidden_layer = **tf.nn.relu**( tf.matmul( tf_train_dataset, hidden_weights) + hidden_biases)

  # Variables.
  weights = tf.Variable( tf.truncated_normal([hidden_nodes, num_labels])) 
  biases = tf.Variable(tf.zeros([num_labels]))

  # Training computation.
  logits = tf.matmul(hidden_layer, weights) + biases
  loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels) )

  # Optimizer.
  optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

  # Predictions for the training, validation, and test data.
  train_prediction = tf.nn.softmax(logits)
  valid_relu = **tf.nn.relu**(  tf.matmul(tf_valid_dataset, hidden_weights) + hidden_biases)
  valid_prediction = tf.nn.softmax( tf.matmul(valid_relu, weights) + biases) 

  test_relu = **tf.nn.relu**( tf.matmul( tf_test_dataset, hidden_weights) + hidden_biases)
  test_prediction = tf.nn.softmax(tf.matmul(test_relu, weights) + biases)

3 Answers 3


In addition to @Bhagyesh_Vikani:

  • Relu behaves close to a linear unit
  • Relu is like a switch for linearity. If you don't need it, you "switch" it off. If you need it, you "switch" it on. Thus, we get the linearity benefits but reserve ourself an option of not using it altogther.
  • The derivative is 1 when it's active. The second derivative of the function is 0 almost everywhere. Thus, it's a very simple function. That makes optimisation much easier.
  • The gradient is large whenever you want it be and never saturate

There are also generalisations of rectified linear units. Rectified linear units and its generalisations are based on the principle that linear models are easier to optimize.

Both sigmoid/softmax are discouraged (chapter 6: Ian Goodfellow) for vanilla feedforward implementation. They are more useful for recurrent networks, probabilistic models, and some autoencoders have additional requirements that rule out the use of piecewise linear activation functions.

If you have a simple NN (that's the question), Relu is your first preference.


Relu have its own pros and cons:

1. Does not saturate (in +ve region)
2. Computationally, it is very efficient
3. Generally models with relu neurons converge much faster than neurons with other activation functions, as described here

1. One issue with dealing with them is where they die, i.e. dead Relus. Because if activation of any relu neurons become zero then its gradients will be clipped to zero in back-propagation. This can be avoided if we are very careful with weights initialization and tuning learning rate.

For more details: Check this lecture-5 of CS231n

  • $\begingroup$ It's worth noting that ELU units trade computational efficiency for immortality -- immunity to dying. arxiv.org/abs/1511.07289 $\endgroup$
    – Sycorax
    Commented Feb 16, 2017 at 5:00
  • $\begingroup$ Thanks for incorporating that. Yes, ELU can be used but the reason why RELUs are still popular and used over ELU is because while using ELU, a new hyper-parameter is introduced. $\endgroup$ Commented Feb 16, 2017 at 5:12
  • 1
    $\begingroup$ There's essentially no reason you have to tune over that particular parameter. $\endgroup$
    – Sycorax
    Commented Feb 16, 2017 at 12:58
  • $\begingroup$ What is +ve region region? $\endgroup$
    – dzieciou
    Commented Sep 9, 2022 at 7:26
  • $\begingroup$ +ve region is where x > 0 $\endgroup$ Commented Sep 9, 2022 at 7:29



Sigmoids saturate and kill gradients. Sigmoid outputs are not zero-centered.


Like the sigmoid neuron, its activations saturate, but unlike the sigmoid neuron its output is zero-centered. Therefore, in practice the tanh non-linearity is always preferred to the sigmoid nonlinearity.


Use the ReLU non-linearity, be careful with your learning rates and possibly monitor the fraction of “dead” units in a network. If this concerns you, give Leaky ReLU or Maxout a try. Never use sigmoid. Try tanh, but expect it to work worse than ReLU/Maxout.

  • 3
    $\begingroup$ I don't understand how sigmoid and tanh are significantly different, when bias nodes can shift one up or down to look like the other. $\endgroup$
    – endolith
    Commented Aug 23, 2018 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.