Suppose that we train some Deep Neural Network and during the training (forward-backward passes) phase, we use Leaky ReLU as our activation function.

During the evaluation, when we show the network the test data and basically do a forward pass, do we need again to have exactly the same activations functions as in training phase? Would it make any sense to use ReLU instead of leaky ReLU which used during the training?


1 Answer 1


If your neural network is working as it was trained, why would you alter how it works? There is no reason to change the activation function after training - this might even break the network.

In your specific case, the network might be doing something useful with negative activation values. By switching to ReLU, all activations will be >0, which would harm te network (because there are no negative activation values anymore).

Why are you asking this? (i'm curious!)

  • $\begingroup$ If during training, some ReLU neurons become deactivated, or "die" it is because they are not needed. Discarding them is then the logical next step. This step is pruning, and, if one wanted to, one could add new neurons in place of the dead ones, while conserving the useful ones at their trained values, and retraining on the same training set. The point i am making here is that paths that are not contributory are also not useful. $\endgroup$
    – Carl
    Apr 30, 2017 at 18:15
  • $\begingroup$ @Carl what are you trying to say? I never said you shouldn't discard them, I never said you can't add new neurons. The point i'm making is: if you train a network to map a function, you can't expect it to do just as well if you suddenly swap the LReLU to ReLU. $\endgroup$ Apr 30, 2017 at 18:18
  • $\begingroup$ Thanks for the comments, the question, is, because in my network when i have trained it with LReLU but evaluate the test set with ReLU I am getting slightly .003ish boost $\endgroup$
    – M. Prk
    Apr 30, 2017 at 18:33
  • $\begingroup$ @ThomasW The OP question is not ridiculous. Also, my comment was a meant as a hint as to why the OP is getting the results he is getting. Using a LReLU is for training and a ReLu for testing is a bit like pruning. It should remove paths that are unlikely. $\endgroup$
    – Carl
    Apr 30, 2017 at 18:44
  • $\begingroup$ @Carl what are you on about? Nowhere do I say it is ridiculous? I'm actually really curious why he is switching to ReLU. It seems like you haven't even read my answer. And by the way, if you seem to think my answer is so wrong, i'm glad to see your own answer. $\endgroup$ Apr 30, 2017 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.