24
$\begingroup$

I know my question/title is not very specific, so I will try to clearify it:

Artificial neural networks have relatively strict designs. Of course, generally, they are influenced by biology and try to build a mathematical model of real neural networks, but our understanding of real neural networks is insufficient for building exact models. Therefore, we can not conceive exact models or anything that comes "near" real neural networks.

As far as I know, all artificial neural networks are far away from real neural networks. Standard, classic fully-connected MLPs are not present in biology. Recurrent neural networks have a lack of real neuroplasticity, each neuron of a RNN has the same "feedback architecture" while real neurons save and share their information rather individually. Convolutional neural networks are effective and popular, but (for example) image processing in the human brain consists of only a few convolution layers while modern solutions (like GoogLeNet) already use tens of layers...and although they are producing great results for computers, they are not even close to human performance. Especially when we think of a "per-layer-performance", as we need a fairly high amount of layers and data reduction compared to real neural networks.

Additionally, to my knowledge, even modular, self-extending / self-restructuring artificial neural networks are rather "fixed and static" compared to the huge adaptability of real neural networks. The biological neuron normally has thousands of dendrites connecting the neuron to a huge variety of different areas and other neurons. Artificial neural networks are way more "straightforward".

So, is there anything we can learn about the human brain / real neural networks from artificial neural networks? Or is it just some attempt to create software that performs better than classic, static algorithms (or even do things where such algorithms fail)?

Can someone supply (preferably scientific) sources about this topic?

EDIT: More answers are highly appreciated (:

$\endgroup$
  • $\begingroup$ A lot has changed since this was asked. Deep networks trained on ImageNet look like they might be reasonably decent approximations for the visual system (or at least the feed-forward part), for example. $\endgroup$ – Matt Krause Aug 23 '18 at 0:59
13
$\begingroup$

As you mentioned, most neural networks are based on general simple abstractions of the brain. Not only are they lacking in mimicking characteristics like plasticity, but they do not take into account signals and timing as real neurons do.

There's a fairly recent interview, that I felt was appropriate for your specific question, Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts, and I quote:

But it’s true that with neuroscience, it’s going to require decades or even hundreds of years to understand the deep principles. There is progress at the very lowest levels of neuroscience. But for issues of higher cognition—how we perceive, how we remember, how we act—we have no idea how neurons are storing information, how they are computing, what the rules are, what the algorithms are, what the representations are, and the like. So we are not yet in an era in which we can be using an understanding of the brain to guide us in the construction of intelligent systems.

$\endgroup$
  • 5
    $\begingroup$ Is this not an answer to a different question from the OPs? The last line of the quote from the Jordan interview implies that he is addressing the question "what can we learn about (how to create) artificial intelligence from our knowledge of the brain?"--not the converse. "What can we learn from the brain from artificial intelligence?" Not much, which is why the field of cognitive science has imploded since its heyday in the '80s and '90s. $\endgroup$ – dodgethesteamroller Jun 29 '15 at 3:53
  • $\begingroup$ Yes - more or less...but it is a nice circumscription of the topic. It leads to the idea that our understanding of the brain is highly insufficient, hence we are not able to build accurate models or to learn much from currently successful models. $\endgroup$ – daniel451 Jun 29 '15 at 6:15
11
$\begingroup$

Not much --- arguably nothing --- has so far been learnt about brain functioning from artificial neural networks. [Clarification: I wrote this answer thinking about neural networks used in machine learning; @MattKrause (+1) is right that neural network models of some biological neural phenomena might have been helpful in many cases.] However, this is perhaps partially due to the fact the research into artificial neural networks in machine learning was more or less in stagnation until around 2006, when Geoffrey Hinton almost single-handedly rekindled the whole field which by now attracts billions of dollars.

In a 2012 lecture in Google called Brains, Sex, and Machine Learning (from 45:30), Hinton suggested that artificial neural networks can provide a hint into why [most] neurons communicate with spikes and not with analogue signals. Namely, he suggests to see spikes as a regularization strategy similar to dropout. Dropout is a recently developed way of preventing overfitting, when only a subset of weights is updated on any given gradient descent step (see Srivastava et al. 2014). Apparently it can work very well, and Hinton thinks that perhaps spikes (i.e. most neurons being silent at any given moment) serve the similar purpose.

I work in a neuroscience research institute and I don't know anybody here who is convinced by the Hinton's argument. The jury is still out (and is probably be going to be out for quite some time), but at least this is an example of something that artificial neural networks could potentially teach us about brain functioning.

$\endgroup$
  • $\begingroup$ This sounds interesting - comparing the technique of dropout vs. spikes in biological neural networks. Can you supply further sources? At least for the moment I did not find a good paper about this topic via some Google Scholar and IEEE searching... $\endgroup$ – daniel451 Jun 29 '15 at 16:48
  • 2
    $\begingroup$ I don't think this has ever been made into a scientific paper or published at all. Sounds more like a provoking idea and a vague intuition that Hinton came up with, but there is a long long way to demonstrate experimentally that it actually is (or is not) the case. $\endgroup$ – amoeba Jun 29 '15 at 16:52
  • 1
    $\begingroup$ Ok...too bad :( would have loved to have some quotable source for those ideas...they sound interesting for conclusions ;) $\endgroup$ – daniel451 Jun 29 '15 at 16:55
  • $\begingroup$ If you do not mind my asking: what is your actual topic of research? You sound familiar and experienced on those kind of issues. Could you recommend papers for further reading? $\endgroup$ – daniel451 Jun 29 '15 at 17:04
8
$\begingroup$

It is certainly not true that the human brain only uses "a few" convolutional layers. About 1/3 of the primate brain is somehow involved in processing visual information. This diagram, from Felleman and Van Essen is a rough outline of how visual information flows through the monkey brain, beginning in the eyes (RGC at the bottom) and ending up in the hippocampus, a memory area. Felleman and Van Essen

Each one of these boxes is a anatomically-defined area (more or less), which contains several processing stages (actual layers, in most cases). The diagram itself is 25 years old and if anything, we've learned that there are a few more boxes and a lot more lines.

It is true that a lot of the deep learning work is more "vaguely inspired by" the brain than based on some underlying neural truth. "Deep learning" also has the added advantage of sounding a lot sexier than "iterated logistic regression."

However, mathematical models of neural networks have also contributed a lot to our understanding of the brain. At one extreme, some models attempt to mimic the known biology and biophysics precisely. These typically include terms for individual ions and their flow. Some even use 3D reconstructions of real neurons to constrain their shape. If this interests you, ModelDB has a large collection of models and the associated publications. Many are implemented using the freely-available NEURONsoftware.

There are larger-scale models that attempt to mimic certain behavioral or neurophysiological effects, without worrying too much about the underlying biophysics. Connectionist or Parallel-Distributed-Processing models, which were particularly popular in the late 1980s and 1990s and used models similar to those you might find in a current machine learning application (e.g., no biophysics, simple activation functions and stereotyped connectivity) to explain various psychological processes. These have fallen a little out of vogue, though one wonders if they might make a comeback now that we have more powerful computers and better training strategies. (See edit below!)

Finally, there is a lot of work somewhere in the middle which includes some "phenomenology", plus some biological details (e.g., an explicitly inhibitory term with certain properties, but without fitting the exact distribution of chloride channels). A lot of current work fits into this category, e.g., work by Xiao Jing Wang (and many others....)

EDIT: Since I wrote this, there's been an explosion of work comparing the (real) visual system to deep neural networks trained on object recognition tasks. There are some surprising similarities. Kernels in the first layers of a neural network are very similar to the kernels/receptive fields in primary visual cortex and subsequent layers resemble the receptive fields in higher visual areas (see work by Nikolaus Kriegeskorte, for example). Retraining neural networks can cause similar changes to extensive behavioral training (Wenliang and Seitz, 2018). DNNs and humans sometimes--but not always--make similar patterns of errors too.

At the moment, it's still rather unclear whether this reflects similarity between real and artificial neural networks in general, something about images specifically[*], or the tendency for neural networks of all stripes to find patterns, even when they aren't there. Nevertheless, comparing the two has become an increasingly hot area of research and it seems likely that we'll learn something from it.

* For example, the representation used in the early visual system/first layers of a CNN is an optimal sparse basis for natural images.

$\endgroup$
  • $\begingroup$ Nice contribution. However, I feel that the OP was pretty specific: what did we learn about the brain from artificial neural network (NN) models? Of course there are zillions of papers in computational neuroscience about NNs, ranging from exploring conditions needed for some criticality patterns and neural avalanches, to what type of biologically plausible plasticity rules can drive learning, etc. And there are further zillions of papers that provide NN models of some neural phenomenon. All of that perhaps tells us something about NNs; but did we really learn anything new about the brain? $\endgroup$ – amoeba Jun 29 '15 at 18:51
  • 3
    $\begingroup$ That last bit is tough to answer concisely. David Heeger earned my undying love and affection for ending his 1992 paper with a list of five proposed experiments. Not many papers do that (and more should), but it highlights a key role for modeling: inspiring new experiments. Off the top of my head, I can't think of anything known only due to modeling, but I can think of several scenarios where we've noticed something interesting about a model and then designed experiments to examine it more closely. $\endgroup$ – Matt Krause Jun 29 '15 at 20:59
1
$\begingroup$

The one what we really learned is the use of sparse activation and the use of linear rectified activation functions. The later is basically one reason, why we saw an explosion in activity regarding so called neural network since using this kind of activation functions resulted in dramatic degrease of training affords for those artificial computational networks we use to call neural networks.

What we learned is why synapse and neurons are build this way and why it is preferable. These linear rectified activation (f(x) := x > a ? x : 0) results in sparse activation (only few of the 'neurons' (weights)) get activated.

So what we do while our knowledge extends towards biological functions, we understand why this was selected and preferred by evolution. We understand that those systems are sufficient enough but also stable in terms of error control during training and also preserve resources like energy and chemical/biological resources in a brain.

We simply understand why the brain is what it is. Also by training and looking at the strategies we understand about possible flows of information and the involved information processing helping us to construct and assess hypothesis about the very subjects.

For example something I can remember from a decade ago was training a system on learning natural spoken language and it the discovery made was the system showed similar problems that reassemble analogic behavior of babies learning speaking a language. Even the differences between learning different kind of languages were similar enough.

So by studying this approach and design, it was concluded that the human information processing during language learning is similar enough to draw training recommendations and treatment for language related problems, that it helped in aiding and understanding humans difficulties and developing more efficient treatment (what ever of it really made it in practice is another question).

A month ago I read an article about how the 3D navigation and remembering of rat brains really work and by creating computational models about every finding it was a great help to understand what is really going on. So the artificial model filled in the blanks of what was observed in the biological system.

It really amazed me when I learned that the neurological scientists used a language that assembled more that of an engineer than a biological person talking about circuits, flow of information and logical processing units.

So we are learning a lot from artificial neural networks since it presents us with empiric play grounds we can derive rules and assurance from when it comes to the why the architecture of the brain is what it is and also why evolution prefers this over alternative ways.

There are still lots of blanks but from what I read - I just got recently into CNN's etc. but had artificial AI, fuzzy logic and neural networks during university time in the early 2000's.

So I had catch up on a decade worth of development and discovery resulting in gratitude for all those scientists and practitioners of the neural network and AI field. Well done people, really well done!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.