Modern neural networks that build their own topology

Question

A limitations of standard neural net algorithms (like backprop) is that you have to make a design decision of how many hidden layers and neurons-per-layer you want. Usually, the learning rate and generalization is highly sensitive to these choices. This has been the reason, why neural net algorithms like cascade correlation have been generating interest. It starts with a minimal topology (just input and output unit) and recruit new hidden units as learning progresses.

The CC-NN algorithm was introduced by Fahlman in 1990, and the recurrent version in 1991. What are some more recent (post 1992) neural net algorithms that start with a minimal topology?

---

Related questions

CogSci.SE: Neural networks with biologically plausible accounts of neurogenesis

Answer 1

The implicit question here is how can you determine the topology/structure of a neural network or machine learning model so that the model is "of the right size" and not overfitting/underfitting.

Since cascade correlation back in 1990, there has been a whole host of methods for doing this now, many of them with much better statistical or computational properties:

• boosting: train a weak learner at a time, with each weak learner given a reweighted training set so that it learns things that past learners haven't learnt.
• sparsity inducing regularization like lasso or automatic relevance determination: start with a large model/network, and use a regularizer that encourages the unneeded units to get "turned off", leaving those that are useful active.
• Bayesian nonparametrics: forget trying to find the "right" model size. Just use one big model, and be careful with regularizing/being Bayesian, so you don't overfit. For example, a neural network with an infinite number of units and Gaussian priors can be derived to be a Gaussian process, which turns out to be much simpler to train.
• Deep learning: as noted in another answer, train a deep network one layer at a time. This doesn't actually solve the problem of determining the number of units per layer - often this is still set by hand or cross-validation.

Answer 2

How about NeuroEvolution of Augmenting Topologies (NEAT) http://www.cs.ucf.edu/~kstanley/neat.html

It seems to work for simple problems, but is INCREDIBLY slow to converge.

Answer 3

As I understand the top of the art today is" Unsupervised Feature Learning and Deep Learning". at the nutshell: the network is being trained in unsupervised manner, each layer at a time:

• http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial
• http://www.youtube.com/watch?v=ZmNOAtZIgIk&feature=player_embedded

Answer 4

There's already been a mention of NEAT (Neural Evolution with Augmenting Topologies). There are advances on this including speciation and HyperNEAT. HyperNEAT uses a 'meta' network to optimise the weighting of a fully connected phenotype. This gives a network 'spacial awareness' which is invaluable in image recognition and board game type problems. You aren't limited to 2D either. I'm using it in 1D for signal analysis and 2D upward is possible but gets heavy on processing requirement. Look for papers by Ken Stanley, and theres a group on Yahoo. If you have a problem that's tractable with a network, then NEAT and/or HyperNEAT may well be applicable.

Answer 5

There is a somewhat recent paper on this topic: R. P. Adams, H. Wallach, and Zoubin Ghahramani. Learning the structure of deep sparse graphical models. This is a bit outside the usual neural network community and more on the machine learning side. The paper uses non-parametric Bayesian inference on the network structure.