I have been told by a professor that it is not possible to combine different activation functions within a neural network. And I can't find any examples of anyone doing this. However, I cannot find any good explanation why.

Conceptually, it seems to make sense: suppose I have a set of data on some population. Imagine the population is divided into two types of people, A and B, but I don't actually have that categorical variable in my data. We could imagine a logistic model that maps my population data into the categorical variable.

Now suppose I'm training a neural net with one hidden layer. It seems to me that it makes perfect sense for one node in that hidden layer to have a sigmoid function as input representing the transformation of the input variables to this latent categorical variable (now as a probability, of course). Meanwhile, all the other nodes have a linear activation function. And then output would be a linear function of all the nodes in the hidden layer.

I have no reason to assume this would improve prediction error. This is what I want to know: is it possible to estimate such a model using standard approaches?

  • $\begingroup$ "it is not possible to combine different activation functions within a neural network" --> you mean that it would lead to a decrease in the network performance? $\endgroup$ – Franck Dernoncourt Dec 8 '16 at 3:35
  • $\begingroup$ Thanks, I clarified the question: I mean is it possible to estimate such a model using the standard approach or would it be a fundamentally different kind of prediction model. $\endgroup$ – samplesize1 Dec 8 '16 at 3:50
  • $\begingroup$ @samplesize1 This is really an empirical question. I'm not aware of any theoretical reason against using different activation functions in a neural net. But, think about the effect of having a sigmoid on one hidden cell and a Tanh on another. Sigmoid has range (0,1) while Tanh goes from (-1,1). So the first cell will stay positive no matter what the inputs, while the other cell could contribute to the output layer negatively. Would you ever want this type of NN? $\endgroup$ – horaceT Dec 8 '16 at 6:34
  • $\begingroup$ @horaceT when you say it's empirical.. do you mean to answer it, one would have to build an algorithm to try to estimate such a model and see if it converges? Is there no theory that sets any requirements for what can be estimated using standard approaches? $\endgroup$ – samplesize1 Dec 8 '16 at 15:36
  • $\begingroup$ Do you have any new thoughts about this problem or done some experiments to see whether there is any difference between the two conditions? I recently think about this problem and find your question. I also wondering. I think different layer use different activation functions may affect the convergence speed of the network, because different activation functions have different drawbacks such as relu will lead to "die units", sigmoid will lead to vanishing gradient and saturation. But since the purpose of activation function in network is in order to bring in non-linearity property, ... $\endgroup$ – likesiwell Jun 29 '17 at 2:01

Clearly you can use different activations in a neural network. An MLP with any activation and a softmax readout layer is one example (for example, multi-class classification). An RNN with LSTM units has at least two activation functions (logistic, tanh and any activations used elsewhere). ReLU activations in the hidden layers and a linear function in the readout layer for a regression problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.