# Varied Activation Functions in Neural Networks

I have been told by a professor that it is not possible to combine different activation functions within a neural network. And I can't find any examples of anyone doing this. However, I cannot find any good explanation why.

Conceptually, it seems to make sense: suppose I have a set of data on some population. Imagine the population is divided into two types of people, A and B, but I don't actually have that categorical variable in my data. We could imagine a logistic model that maps my population data into the categorical variable.

Now suppose I'm training a neural net with one hidden layer. It seems to me that it makes perfect sense for one node in that hidden layer to have a sigmoid function as input representing the transformation of the input variables to this latent categorical variable (now as a probability, of course). Meanwhile, all the other nodes have a linear activation function. And then output would be a linear function of all the nodes in the hidden layer.

I have no reason to assume this would improve prediction error. This is what I want to know: is it possible to estimate such a model using standard approaches?

• "it is not possible to combine different activation functions within a neural network" --> you mean that it would lead to a decrease in the network performance? Dec 8, 2016 at 3:35
• Thanks, I clarified the question: I mean is it possible to estimate such a model using the standard approach or would it be a fundamentally different kind of prediction model. Dec 8, 2016 at 3:50
• @samplesize1 This is really an empirical question. I'm not aware of any theoretical reason against using different activation functions in a neural net. But, think about the effect of having a sigmoid on one hidden cell and a Tanh on another. Sigmoid has range (0,1) while Tanh goes from (-1,1). So the first cell will stay positive no matter what the inputs, while the other cell could contribute to the output layer negatively. Would you ever want this type of NN? Dec 8, 2016 at 6:34
• @horaceT when you say it's empirical.. do you mean to answer it, one would have to build an algorithm to try to estimate such a model and see if it converges? Is there no theory that sets any requirements for what can be estimated using standard approaches? Dec 8, 2016 at 15:36
• Do you have any new thoughts about this problem or done some experiments to see whether there is any difference between the two conditions? I recently think about this problem and find your question. I also wondering. I think different layer use different activation functions may affect the convergence speed of the network, because different activation functions have different drawbacks such as relu will lead to "die units", sigmoid will lead to vanishing gradient and saturation. But since the purpose of activation function in network is in order to bring in non-linearity property, ... Jun 29, 2017 at 2:01