5
$\begingroup$

I am developing a neuroscience-inspired training algorithm for feed-forward neural networks. The natural benchmark for comparison is backpropagation. So I need to know to know what sort of classification problems backpropagation (and thus feed-forward NNs) is most suited for. Also, since I ultimately want to move beyond backprop, what non NN algorithms are the best alternatives to backprop, when it comes to the sort of problems that backprop is good at.

Or, in other words, what problems should I benchmark a feed-forward network on, and what are the algorithms to beat for those problems?

$\endgroup$
2
$\begingroup$

In general, feed-forward networks utilizing backpropagation are great for classification tasks in which you have a number of probabilistic cues which need to be integrated. They're obviously used for a great many other things, but this is an example of one driving many people to use them -- it's difficult to capture this sort of learning in a more Hebbian fashion.

In any case where you have multiple probabilistic cues which may be of limited, but above-chance use in classification, they can be integrated in a cognitively plausible manner (as in natural language acquisition by children) -- the integration of multiple cues, any one of which is a weak cue for classification in and of itself, can result in very robust classification performance.

For this sort of classification w/ multiple cues, AdaBoost would be an appropriate (and quite well-established) algorithm to use as a baseline.

An off-the-shelf support vector machine might rival AdaBoost's performance. If you're looking for baseline classifiers that people are already familiar with, you should definitely use an SVM for at least one of them. There are also approaches that combine AdaBoost with SVMs.

$\endgroup$
  • $\begingroup$ Cool! Is AdaBoost one of the best? or are there new algorithms that can outperform it? $\endgroup$ – Matt Munson Apr 3 '12 at 19:32
  • $\begingroup$ An off-the-shelf support vector machine might rival AdaBoost's performance. If you're looking for baseline classifiers that people are already familiar with, you should definitely use an SVM for at least one of them. There are also approaches that combine AdaBoost with SVMs. $\endgroup$ – Julie Apr 3 '12 at 20:02
  • $\begingroup$ "classification tasks in which you have a number of probabilistic cues which need to be integrated." Per your suggestion, I have been doing some preliminary tests in this area and am getting good results. I am wondering if you could give me some examples of practical problems in this domain, or of some benchmarck datasets. $\endgroup$ – Matt Munson Apr 12 '12 at 4:07
2
$\begingroup$

Edit

Originally this answer discussed alternate learning algorithms and topologies for Neural Nets. After the edit, the answer is divided into three parts:

  1. Uses and problems with Backpropogation;
  2. Alternate Neural Network Training Schemes; and
  3. Alternates to Neural Networks (newly added).

Part 1 Backpropogation algorithms can help to solve problems where there is a discriminant that can help to separate the positive inputs from the negative inputs, often in networks that have no connections that loop between neurons in their topology (feed forward neural networks as described in your question).

Backpropogation is typically slow and it is not guaranteed to converge and may converge to a local maximum (that is, it will converge up to a point were you get results that are better than the results that can be obtained by changing the weights slightly).

Part 2 Alternate neural network training algorithms include using a Genetic Algorithm to evolve the weights and simulated annealing. These may eliminate the problems related to slow convergence and problems with connections that loop between neurons.

Hebbian learning as used in recurrent neural networks, typically Hopfield networks, is another alternative. Hebbian learning is robust and Hopfield proved that Hebbian learning will converge for Hopfield networks.

You might also consider comparing your neural network to a modular neural network, such as, for example a Committee of Machines made up of several neural networks each with their own training algorithms or an Associative Neural Network.

Such modular neural networks will require more space and time to work with, but ultimately may perform better if their topologies are designed well for the particular problem at hand.

Part 3

Popular alternatives to consider include Support Vector Machines (which are widely used in Fraud Analysis and Network Intrusion Detection amongst other areas).

Genetic Algorithms and Simulated Annealing (as stand-alone algorithms and not in conjunction with Neural Nets).

A final group of algorithms to consider are Swarm Algorithms such as Ant Colony Optimisation. Some algorithms in this group such as continuous orthogonal ant colony optimisation can be very powerful for solving some problems.

You might consider using some sort of clustering algorithms. Depending on your problems, K-Medoids or Density Based Clustering might be useful.

Depending on your problem, Machine Vision algorithms might also be useful. For example, the algorithm by Milanfar, http://users.soe.ucsc.edu/~milanfar/research/computer-vision.html, based on local regression kernels might also be useful.

The exact algorithms to use might depend on the exact problem that you are facing. Some algorithms will work really well in some instances but really badly in another. Personally I do not believe that there is a single algorithm that is perfect for solving everything (if there were we would be using it), so you would need to decide on the algorithm to benchmark against depending on the scenario in which you will be using it.

$\endgroup$
  • $\begingroup$ I agree about the slowness and local minimal problems, but I think that NN are good in solving NON-linear discriminant problems. They also require a lot of experience designing them correctly for each problem. $\endgroup$ – Guy Apr 3 '12 at 20:20
  • $\begingroup$ "problems where there is a linear discriminant that can help to separate the positive inputs from the negative inputs". what does this mean? $\endgroup$ – Matt Munson Apr 3 '12 at 22:35
  • $\begingroup$ Since Matt wants to compare his NN implementation to state-of-the-art classifiers, I think he should definitely think about using a support vector machine as a baseline -- it could be just one of several baselines for comparison. $\endgroup$ – Julie Apr 4 '12 at 5:06
  • $\begingroup$ Guy - You might be able to solve NON-linear problems given the appropriate kernel, if this will transform the non-linear problem into a linear one. Also, rereading the problem thoroughly, I noticed that the querier is actually asking for NON-Neural Net alternative, whereas in this post above I described alternate Neural Net learning algorithms. Of course, Genetic Algorithms and Simulated Annealing might be used as algorithms on their own and not just training for NN. As Julie and others mentioned, SVMs are popular alternatives and Ant Colony / SWARM optimisation might also be interesting. $\endgroup$ – Andrew Apr 4 '12 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.