I have been told by a professor that it is not possible to combine different activation functions within a neural network. And I can't find any examples of anyone doing this. However, I cannot find any good explanation why.

Conceptually, it seems to make sense: suppose I have a set of data on some population. Imagine the population is divided into two types of people, A and B, but I don't actually have that categorical variable in my data. We could imagine a logistic model that maps my population data into the categorical variable.

Now suppose I'm training a neural net with one hidden layer. It seems to me that it makes perfect sense for one node in that hidden layer to have a sigmoid function as input representing the transformation of the input variables to this latent categorical variable (now as a probability, of course). Meanwhile, all the other nodes have a linear activation function. And then output would be a linear function of all the nodes in the hidden layer.

I have no reason to assume this would improve prediction error. This is what I want to know: is it possible to estimate such a model using standard approaches?

  • $\begingroup$ "it is not possible to combine different activation functions within a neural network" --> you mean that it would lead to a decrease in the network performance? $\endgroup$ Dec 8 '16 at 3:35
  • $\begingroup$ Thanks, I clarified the question: I mean is it possible to estimate such a model using the standard approach or would it be a fundamentally different kind of prediction model. $\endgroup$ Dec 8 '16 at 3:50
  • $\begingroup$ @samplesize1 This is really an empirical question. I'm not aware of any theoretical reason against using different activation functions in a neural net. But, think about the effect of having a sigmoid on one hidden cell and a Tanh on another. Sigmoid has range (0,1) while Tanh goes from (-1,1). So the first cell will stay positive no matter what the inputs, while the other cell could contribute to the output layer negatively. Would you ever want this type of NN? $\endgroup$
    – horaceT
    Dec 8 '16 at 6:34
  • $\begingroup$ @horaceT when you say it's empirical.. do you mean to answer it, one would have to build an algorithm to try to estimate such a model and see if it converges? Is there no theory that sets any requirements for what can be estimated using standard approaches? $\endgroup$ Dec 8 '16 at 15:36
  • $\begingroup$ Do you have any new thoughts about this problem or done some experiments to see whether there is any difference between the two conditions? I recently think about this problem and find your question. I also wondering. I think different layer use different activation functions may affect the convergence speed of the network, because different activation functions have different drawbacks such as relu will lead to "die units", sigmoid will lead to vanishing gradient and saturation. But since the purpose of activation function in network is in order to bring in non-linearity property, ... $\endgroup$
    – likesiwell
    Jun 29 '17 at 2:01

Clearly you can use different activations in a neural network. An MLP with any activation and a softmax readout layer is one example (for example, multi-class classification). An RNN with LSTM units has at least two activation functions (logistic, tanh and any activations used elsewhere). ReLU activations in the hidden layers and a linear function in the readout layer for a regression problem.


I believe what is meant by the question is: can we mix different activation functions in a single layer.

So imagine we have only one hidden layer with 3 nodes, can I set the first node to have sigmoid, second node to have ReLU, and third node to have tanh?

I just thought about this as well, and I believe this should be possible but with the cost of computation time, because then we cannot vectorize the computation for that hidden layer.

  • $\begingroup$ An LSTM is a single layer, and an LSTM also contains two activation functions. Or you an use a FFN that vectorizes the two or more parts of a hidden layer before concatenating them again for the next layer. Whether or not you think this is "vectorized," it's eminently implementable in standard libraries. $\endgroup$
    – Sycorax
    Mar 18 at 2:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.