21
$\begingroup$

A limitations of standard neural net algorithms (like backprop) is that you have to make a design decision of how many hidden layers and neurons-per-layer you want. Usually, the learning rate and generalization is highly sensitive to these choices. This has been the reason, why neural net algorithms like cascade correlation have been generating interest. It starts with a minimal topology (just input and output unit) and recruit new hidden units as learning progresses.

The CC-NN algorithm was introduced by Fahlman in 1990, and the recurrent version in 1991. What are some more recent (post 1992) neural net algorithms that start with a minimal topology?


Related questions

CogSci.SE: Neural networks with biologically plausible accounts of neurogenesis

$\endgroup$
10
$\begingroup$

The implicit question here is how can you determine the topology/structure of a neural network or machine learning model so that the model is "of the right size" and not overfitting/underfitting.

Since cascade correlation back in 1990, there has been a whole host of methods for doing this now, many of them with much better statistical or computational properties:

  • boosting: train a weak learner at a time, with each weak learner given a reweighted training set so that it learns things that past learners haven't learnt.
  • sparsity inducing regularization like lasso or automatic relevance determination: start with a large model/network, and use a regularizer that encourages the unneeded units to get "turned off", leaving those that are useful active.
  • Bayesian nonparametrics: forget trying to find the "right" model size. Just use one big model, and be careful with regularizing/being Bayesian, so you don't overfit. For example, a neural network with an infinite number of units and Gaussian priors can be derived to be a Gaussian process, which turns out to be much simpler to train.
  • Deep learning: as noted in another answer, train a deep network one layer at a time. This doesn't actually solve the problem of determining the number of units per layer - often this is still set by hand or cross-validation.
$\endgroup$
9
$\begingroup$

How about NeuroEvolution of Augmenting Topologies (NEAT) http://www.cs.ucf.edu/~kstanley/neat.html

It seems to work for simple problems, but is INCREDIBLY slow to converge.

$\endgroup$
4
$\begingroup$

As I understand the top of the art today is" Unsupervised Feature Learning and Deep Learning". at the nutshell: the network is being trained in unsupervised manner, each layer at a time:

$\endgroup$
  • $\begingroup$ does it start with a minimal number of neurons (only input and output)? $\endgroup$ – Artem Kaznatcheev Feb 12 '12 at 18:18
  • $\begingroup$ deep learning contains a wide variety of methods for neural networks with multiple hidden layers. I'm not familiar with such a methods that determines the number of neurons, but maybe google scholar knows more... $\endgroup$ – Ran Feb 12 '12 at 19:46
  • $\begingroup$ Afaik the number is fixed beforehand in all competitive current methods. This is somewhat of a problem since this means there are a lot of hyper parameters. To deal with that, James Bergstra recently proposed the use of Gaussian Processes to find the best hyper paramter settings9http://people.fas.harvard.edu/~bergstra/files/pub/11_nips_hyperopt.pdf). But this is a sort of "outer loop" that tries many different settings in a smart way. $\endgroup$ – Andreas Mueller Feb 14 '12 at 8:26
4
$\begingroup$

There's already been a mention of NEAT (Neural Evolution with Augmenting Topologies). There are advances on this including speciation and HyperNEAT. HyperNEAT uses a 'meta' network to optimise the weighting of a fully connected phenotype. This gives a network 'spacial awareness' which is invaluable in image recognition and board game type problems. You aren't limited to 2D either. I'm using it in 1D for signal analysis and 2D upward is possible but gets heavy on processing requirement. Look for papers by Ken Stanley, and theres a group on Yahoo. If you have a problem that's tractable with a network, then NEAT and/or HyperNEAT may well be applicable.

$\endgroup$
3
$\begingroup$

There is a somewhat recent paper on this topic: R. P. Adams, H. Wallach, and Zoubin Ghahramani. Learning the structure of deep sparse graphical models. This is a bit outside the usual neural network community and more on the machine learning side. The paper uses non-parametric Bayesian inference on the network structure.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.