Considering multi-layer fully connected feed-forward networks we specify the single layers together with an activation which is the same for each node (neuron) in that layer. However does the activation function have to be same for each neuron within a layer?
- Would it make sense to use different activation functions within single layers in order to modify the degree of non-linearity that the network reflects?
- If so, what would be possible use cases for which this type of architecture would be superior to homogeneously activated layers?
- Is there any research on this type of network architecture?