13
$\begingroup$

Pulsed or Spiking neural networks incorporate more of the membrane dynamics of biological neurons, where pulses carry information to the next layer. The neurons do not necessarily have to "fire" all at the same time, as they would in a backprop, for example.

Yet, there seem to be barriers against employing these models for machine learning problems. What specific issues stand in the way of machine learning practitioners using models that are more biologically realistic?

$\endgroup$
12
$\begingroup$

The major issue is that nobody knows how the brain works :)

Theory

As far as I can tell, there are three big steps in neural network research:

  1. Perceptron (or, threshold-gate) model, where any boolean function can be computed by some multi-layer perceptron with a single hidden layer.
  2. Neuron model - improved version of the previous one, where network elements use sigmoid activation function (continuous set of possible inpits and outputs). They also can compute any boolean function (after applying a threshold) and, additionally, can approximate any continuous function.
  3. Spiking neurons model, which uses "temporal coding" to pass information between network elements. It can do everything the previous model does, and often can do so with fewer neurons.

Basically, the evolution here is to get closer to how human brain works, and the last model has the best fidelity.

Practice

SNNs seem very promising and there's even a commersial product SpikeNET built on top of it (under "What SpikeNET can do" and "What SpikeNET cannot yet do" you can see the problems they faced).

I can't tell about specific issues with spiking networks - but in general I have an impression that the issues arise because people want SNNs work more and more like a human brain:

  • they want to choose how to code information - which can be done via delay coding (more highly stimulated neurons tend to spike more frequently and sooner), binary coding (information is represented by the number of spikes within a given time interval), time coding (information is the inverse of the granularity of the distinguishable time interval), rank order coding (first spikes received by a neuron are given influence and later ones are inhibited) and whatnot.
  • they emulate Hebbian plasticity which increases weights between neurons when both neurons are "on" (or both "off") at the same.
  • they apply self-organization where a group of neurons compete with the winning neuron inhibiting the response of other neurons. With a spiking neuron the winner can be computed quickly based only on single firing events.

Wikipedia has a link to the "Pulsed Neural Networks" book which has "Implementation Issues of Pulse-Coded Neural Networks" section but I'm not educated enough to comment on that.

As for introduction to the topic, I recommend this paper: Pulsed Neural Networks and their Application (pdf)

$\endgroup$
  • 1
    $\begingroup$ I agree that no one knows how the brain works, but there have been good, physiologically accurate membrane models since the 30s or so. Excellent information in the answer, but from what you've written, I would think practitioners would be encouraged to use them, rather than discouraged. $\endgroup$ – jonsca Feb 16 '12 at 21:03
  • $\begingroup$ Yep exactly - they are encouraged by those issues! :) $\endgroup$ – andreister Feb 17 '12 at 6:23
0
$\begingroup$

it seems that all neural network algorithms use some form of gradient descent in their training algorithms and even non ANN models use gradient descent. there seems to be no theory about how to apply gradient descent in a temporal fashion over SNNs. one possibility is a rise in neuromorphic computing which uses more biologically realistic models similiar to SNNs. but it seems like there are not strong machine-learning benchmarks/breakthroughs achieved in the neuromorphic field to date as with very definite benchmarks achieved with ANNs in many standard ML problems like handwriting recognition, speech recognition, object recognition, language translation, etc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.