# Research on heterogeneous neural networks

Is there a sound theoretical reason why there's relatively little research on heterogeneous neural networks? By this I mean neural nets with non-homogeneous activation functions.

So far most of the papers I've read analyse neural networks with at most two types of activation function. One type for the hidden layers(ex. ReLU) and one for the output layer(ex. softmax).

Is there a sound theoretical reason for the relatively small number of papers on this subject?

• Just a guess: I think it's mostly due to practicalities: sigmoid units have vanishing gradient problems, which is ameliorated by ReLUs. Going half-and-half on activation functions only solves half the problem. – Sycorax Mar 22 '17 at 0:09
• @Sycorax What if the activation functions are learned? Ex: maxout networks. – Aidan Rocke Mar 22 '17 at 1:32
• I'm not sure. I remember reading the maxout paper a while back; it seemed like a lot of extra parameters for a marginal benefit, but maybe I'm misremembering. The problem remains, though: maxout+sigmoid still has gradient issues and maxout+relu might as well be maxout everywhere since relu is a special case of maxout. – Sycorax Mar 22 '17 at 1:57
• @Sycorax What I meant is that while training, the maxout can converge to any convex function. In that sense, the resulting neural network is heterogeneous. – Aidan Rocke Mar 22 '17 at 3:15
• It's perfectly fair to m write your own answer and get points on it. – Sycorax Mar 22 '17 at 13:05

After more reflection, I realised that maxout networks are probably the best example of a heterogeneous network that is learned[1]. I'll try to clarify this below.

Given an input $x \in \mathbb{R}^d$, a maxout hidden layer implements the function:

$$h_i = max_{j \in [1,k]} z_{ij}$$

where $z_{ij} = x^TW_{ij}+b_{ij}, W \in \mathbb{R}^{dxmxk}, b \in \mathbb{R}^{mxk}$. Each of the $m$ units has $k$ different affine transformation units(i.e. piecewise linear functions) as illustrated here.

Now, the Stone-Weierstrass approximation theorem gives us the desired result that any convex function can be approximated arbitrarily well by a sufficiently large number of maxout units. In fact, in Goodfellow's paper an existence proof is given which shows that two hidden maxout units are sufficient to approximate any continuous function provided that $k$ is sufficiently large. For this reason, it's reasonable to argue that after training a network for a supervised task the resulting trained network will be probably be heterogeneous in terms of its hidden units.

In fact, given that maxout networks are intended to be used with dropout regularisation it's not surprising that some hidden units among the "thinned" networks might be highly non-linear although they will all be locally linear almost everywhere. Clearly, the ReLU is a very special case of the maxout hidden unit. More importantly, these geometric properties allow maxout networks to take advantage of dropout's model averaging technique much better than sigmoid, tanh...and other activations that have significant curvature almost everywhere without reducing its ability to learn highly non-linear mappings.

Finally, it's important to note that the maxout network was tested on four benchmarks(CIFAR-10,MNIST,CIFAR-100,SVHN) and set the state of the art on all of them. I think that this important research sketches a path for more interesting work on 'emergent' heterogeneous networks.

References:

1. Goodfellow, Ian et al. "Maxout Networks" Arxiv. 20 Sep 2013
2. Srivastava, Nitish et al. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" Journal of Machine Learning Research. June 2014