Neural network with heterogeneous activation within layers

Question

Considering multi-layer fully connected feed-forward networks we specify the single layers together with an activation which is the same for each node (neuron) in that layer. However does the activation function have to be same for each neuron within a layer?

1. Would it make sense to use different activation functions within single layers in order to modify the degree of non-linearity that the network reflects?
2. If so, what would be possible use cases for which this type of architecture would be superior to homogeneously activated layers?
3. Is there any research on this type of network architecture?

Answer 1

The Maxout activation is a parameterized activation function which can model convex piecewise linear functions -- relu, identity, and abs are three such functions.

Critically, the parameters of individual neurons are untied, so one neuron in a layer can use relu, while another uses abs, while a third uses something entirely different.

The paper shows that this sort of network is competitive for general image recognition / classification tasks, and can achieve better performance. Despite the fact, maxout has not been widely adopted because it heavily increases the parameter count and all the usual associated problems.