What can we learn about the human brain from artificial neural networks?

Question

I know my question/title is not very specific, so I will try to clearify it:

Artificial neural networks have relatively strict designs. Of course, generally, they are influenced by biology and try to build a mathematical model of real neural networks, but our understanding of real neural networks is insufficient for building exact models. Therefore, we can not conceive exact models or anything that comes "near" real neural networks.

As far as I know, all artificial neural networks are far away from real neural networks. Standard, classic fully-connected MLPs are not present in biology. Recurrent neural networks have a lack of real neuroplasticity, each neuron of a RNN has the same "feedback architecture" while real neurons save and share their information rather individually. Convolutional neural networks are effective and popular, but (for example) image processing in the human brain consists of only a few convolution layers while modern solutions (like GoogLeNet) already use tens of layers...and although they are producing great results for computers, they are not even close to human performance. Especially when we think of a "per-layer-performance", as we need a fairly high amount of layers and data reduction compared to real neural networks.

Additionally, to my knowledge, even modular, self-extending / self-restructuring artificial neural networks are rather "fixed and static" compared to the huge adaptability of real neural networks. The biological neuron normally has thousands of dendrites connecting the neuron to a huge variety of different areas and other neurons. Artificial neural networks are way more "straightforward".

So, is there anything we can learn about the human brain / real neural networks from artificial neural networks? Or is it just some attempt to create software that performs better than classic, static algorithms (or even do things where such algorithms fail)?

Can someone supply (preferably scientific) sources about this topic?

EDIT: More answers are highly appreciated (:

Answer 1

As you mentioned, most neural networks are based on general simple abstractions of the brain. Not only are they lacking in mimicking characteristics like plasticity, but they do not take into account signals and timing as real neurons do.

There's a fairly recent interview, that I felt was appropriate for your specific question, Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts, and I quote:

But it’s true that with neuroscience, it’s going to require decades or even hundreds of years to understand the deep principles. There is progress at the very lowest levels of neuroscience. But for issues of higher cognition—how we perceive, how we remember, how we act—we have no idea how neurons are storing information, how they are computing, what the rules are, what the algorithms are, what the representations are, and the like. So we are not yet in an era in which we can be using an understanding of the brain to guide us in the construction of intelligent systems.

Answer 2

Not much --- arguably nothing --- has so far been learnt about brain functioning from artificial neural networks. [Clarification: I wrote this answer thinking about neural networks used in machine learning; @MattKrause (+1) is right that neural network models of some biological neural phenomena might have been helpful in many cases.] However, this is perhaps partially due to the fact the research into artificial neural networks in machine learning was more or less in stagnation until around 2006, when Geoffrey Hinton almost single-handedly rekindled the whole field which by now attracts billions of dollars.

In a 2012 lecture in Google called Brains, Sex, and Machine Learning (from 45:30), Hinton suggested that artificial neural networks can provide a hint into why [most] neurons communicate with spikes and not with analogue signals. Namely, he suggests to see spikes as a regularization strategy similar to dropout. Dropout is a recently developed way of preventing overfitting, when only a subset of weights is updated on any given gradient descent step (see Srivastava et al. 2014). Apparently it can work very well, and Hinton thinks that perhaps spikes (i.e. most neurons being silent at any given moment) serve the similar purpose.

I work in a neuroscience research institute and I don't know anybody here who is convinced by the Hinton's argument. The jury is still out (and is probably be going to be out for quite some time), but at least this is an example of something that artificial neural networks could potentially teach us about brain functioning.

Answer 3

It is certainly not true that the human brain only uses "a few" convolutional layers. About 1/3 of the primate brain is somehow involved in processing visual information. This diagram, from Felleman and Van Essen is a rough outline of how visual information flows through the monkey brain, beginning in the eyes (RGC at the bottom) and ending up in the hippocampus, a memory area.

Each one of these boxes is a anatomically-defined area (more or less), which contains several processing stages (actual layers, in most cases). The diagram itself is 25 years old and if anything, we've learned that there are a few more boxes and a lot more lines.

It is true that a lot of the deep learning work is more "vaguely inspired by" the brain than based on some underlying neural truth. "Deep learning" also has the added advantage of sounding a lot sexier than "iterated logistic regression."

However, mathematical models of neural networks have also contributed a lot to our understanding of the brain. At one extreme, some models attempt to mimic the known biology and biophysics precisely. These typically include terms for individual ions and their flow. Some even use 3D reconstructions of real neurons to constrain their shape. If this interests you, ModelDB has a large collection of models and the associated publications. Many are implemented using the freely-available NEURONsoftware.

There are larger-scale models that attempt to mimic certain behavioral or neurophysiological effects, without worrying too much about the underlying biophysics. Connectionist or Parallel-Distributed-Processing models, which were particularly popular in the late 1980s and 1990s and used models similar to those you might find in a current machine learning application (e.g., no biophysics, simple activation functions and stereotyped connectivity) to explain various psychological processes. These have fallen a little out of vogue, though one wonders if they might make a comeback now that we have more powerful computers and better training strategies. (See edit below!)

Finally, there is a lot of work somewhere in the middle which includes some "phenomenology", plus some biological details (e.g., an explicitly inhibitory term with certain properties, but without fitting the exact distribution of chloride channels). A lot of current work fits into this category, e.g., work by Xiao Jing Wang (and many others....)

EDIT: Since I wrote this, there's been an explosion of work comparing the (real) visual system to deep neural networks trained on object recognition tasks. There are some surprising similarities. Kernels in the first layers of a neural network are very similar to the kernels/receptive fields in primary visual cortex and subsequent layers resemble the receptive fields in higher visual areas (see work by Nikolaus Kriegeskorte, for example). Retraining neural networks can cause similar changes to extensive behavioral training (Wenliang and Seitz, 2018). DNNs and humans sometimes--but not always--make similar patterns of errors too.

At the moment, it's still rather unclear whether this reflects similarity between real and artificial neural networks in general, something about images specifically[*], or the tendency for neural networks of all stripes to find patterns, even when they aren't there. Nevertheless, comparing the two has become an increasingly hot area of research and it seems likely that we'll learn something from it.

* For example, the representation used in the early visual system/first layers of a CNN is an optimal sparse basis for natural images.

Answer 4

The one what we really learned is the use of sparse activation and the use of linear rectified activation functions. The later is basically one reason, why we saw an explosion in activity regarding so called neural network since using this kind of activation functions resulted in dramatic degrease of training affords for those artificial computational networks we use to call neural networks.

What we learned is why synapse and neurons are build this way and why it is preferable. These linear rectified activation (f(x) := x > a ? x : 0) results in sparse activation (only few of the 'neurons' (weights)) get activated.

So what we do while our knowledge extends towards biological functions, we understand why this was selected and preferred by evolution. We understand that those systems are sufficient enough but also stable in terms of error control during training and also preserve resources like energy and chemical/biological resources in a brain.

We simply understand why the brain is what it is. Also by training and looking at the strategies we understand about possible flows of information and the involved information processing helping us to construct and assess hypothesis about the very subjects.

For example something I can remember from a decade ago was training a system on learning natural spoken language and it the discovery made was the system showed similar problems that reassemble analogic behavior of babies learning speaking a language. Even the differences between learning different kind of languages were similar enough.

So by studying this approach and design, it was concluded that the human information processing during language learning is similar enough to draw training recommendations and treatment for language related problems, that it helped in aiding and understanding humans difficulties and developing more efficient treatment (what ever of it really made it in practice is another question).

A month ago I read an article about how the 3D navigation and remembering of rat brains really work and by creating computational models about every finding it was a great help to understand what is really going on. So the artificial model filled in the blanks of what was observed in the biological system.

It really amazed me when I learned that the neurological scientists used a language that assembled more that of an engineer than a biological person talking about circuits, flow of information and logical processing units.

So we are learning a lot from artificial neural networks since it presents us with empiric play grounds we can derive rules and assurance from when it comes to the why the architecture of the brain is what it is and also why evolution prefers this over alternative ways.

There are still lots of blanks but from what I read - I just got recently into CNN's etc. but had artificial AI, fuzzy logic and neural networks during university time in the early 2000's.

So I had catch up on a decade worth of development and discovery resulting in gratitude for all those scientists and practitioners of the neural network and AI field. Well done people, really well done!