6
$\begingroup$

I understand that backpropagation is good, but what are the main advantages (and disadvantaged) that it has over Hebbian learning?

I'm mostly wondering about contrastive Hebbian learning, though arguments against Hebbian learning in general are welcomed.

$\endgroup$
  • $\begingroup$ What do you imagine "Hebbian learning" entails? You can't just do "neurons that fire together wire together" or else all of the neurons would eventually wire together. $\endgroup$ – Neil G Jun 17 '17 at 1:56
  • $\begingroup$ Well there's contrastive Hebbian learning, Oja's rule, and I'm sure many other things that branch from Hebbian learning as a general concept, just as naive backprop may not work unless you have good architectures, learning rates, normalization, etc. I'm wondering why in general Hebbian learning hasn't been so popular. Is it just because no one has come up with a good way to avoid making all the neurons wire together? I really don't know, so I asking what the fundamental issues are. $\endgroup$ – Veech Jun 17 '17 at 8:10
  • 1
    $\begingroup$ Contrastive divergence and Oja's rule both prevent all of the neurons from "wiring together". This is a good question. $\endgroup$ – Neil G Jun 17 '17 at 17:03
4
$\begingroup$

I have been doing simple toy experiments for a while and I find it is rather difficult to make Hebbian rules work well. Fundamentally, Hebbian learning leans more towards unsupervised learning as a teacher signal at a deep layer cannot be efficiently propagated to lower levels as in backprop with ReLu. This in turn means that for Hebbian systems to be applied to real problems, the system has to find 'good' representations that can easily be 'fetched' by another supervised algorithm as in the following recent example: http://www.ece.ucsb.edu/wcsl/people/aseem/Aseem_stuff/hebbian_preprint.pdf

As for advantages of Hebbian learning I would list the following:

  • high online adaptivity to changes in the input distribution
  • (probably) higher suitability to super-deep architectures, especially RNNs
  • biological plausibility
  • easy interpretability of what the algorithm is doing layer-to-layer

I hope others with more knowledge will chime in for some comments as I'd like to know more about this as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.