Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Is there a sound theoretical reason why there's relatively little research on heterogeneous neural networks? By this I mean neural nets with non-homogeneous activation functions.
So far most of the papers I've read analyse neural networks with at most two types of activation function. One type for the hidden layers(ex. ReLU) and one for the output layer(ex. softmax).
Is there a sound theoretical reason for the relatively small number of papers on this subject?
After more reflection, I realised that maxout networks are probably the best example of a heterogeneous network that is learned[1]. I'll try to clarify this below.
Given an input $ x \in \mathbb{R}^d$, a maxout hidden layer implements the function:
\begin{equation} h_i = max_{j \in [1,k]} z_{ij} \end{equation}
where $z_{ij} = x^TW_{ij}+b_{ij}, W \in \mathbb{R}^{dxmxk}, b \in \mathbb{R}^{mxk}$. Each of the $m$ units has $k$ different affine transformation units(i.e. piecewise linear functions) as illustrated here.
Now, the Stone-Weierstrass approximation theorem gives us the desired result that any convex function can be approximated arbitrarily well by a sufficiently large number of maxout units. In fact, in Goodfellow's paper an existence proof is given which shows that two hidden maxout units are sufficient to approximate any continuous function provided that $k$ is sufficiently large. For this reason, it's reasonable to argue that after training a network for a supervised task the resulting trained network will be probably be heterogeneous in terms of its hidden units.
In fact, given that maxout networks are intended to be used with dropout regularisation it's not surprising that some hidden units among the "thinned" networks might be highly non-linear although they will all be locally linear almost everywhere. Clearly, the ReLU is a very special case of the maxout hidden unit. More importantly, these geometric properties allow maxout networks to take advantage of dropout's model averaging technique much better than sigmoid, tanh...and other activations that have significant curvature almost everywhere without reducing its ability to learn highly non-linear mappings.
Finally, it's important to note that the maxout network was tested on four benchmarks(CIFAR-10,MNIST,CIFAR-100,SVHN) and set the state of the art on all of them. I think that this important research sketches a path for more interesting work on 'emergent' heterogeneous networks.
References:
