# Magic of neural networks?

I've read a bit on neural networks and wonder why they are actually so popular at the moment. For example, a feed-forward network is a quite simple mathematical object: An input vector $\mathbf{x}\in \mathbb{R}^n$ is mapped to some output vector $\mathbf{y}\in\mathbb{R}^k$ via linear and non-linear mappings, e.g., for a two-layer NN:

$$\mathbf{y}=f(\mathbf{W}_1,\mathbf{W}_2)=h(\mathbf{W}_2g(\mathbf{W}_1\mathbf{x})).$$

Learning amounts to estimating the matrices $\mathbf{W}_1,\mathbf{W}_2$. So, asking a bit heretically: Most mathematicians would find this a very simple problem, for which well-known techniques exist. So, what's the issue about neural networks and why are they so powerful and why is there so much mystery surrounding them?

• I think the only mystery surrounding them is by those who don't possess the necessary mathematical and statistical background to understand how they "learn." – StatsStudent Jan 25 '16 at 0:19
• And the label "neural" evokes a connotation that the algorihm would somehow mimic how the brain operates, which indeed - for most human beings - works mysteriously well. – Christoph Hanck Jan 25 '16 at 5:32
• because there are infinite amount of non-linear functions, and neural networks can help to find a good one. – dontloo Jan 25 '16 at 7:27

The reason there is such a fuss about them now is because of 'big data' in particular image and speech data (high dimensional and large number of samples, cf imagenet) . Essentially the issue is computational rather than mathematical :neural networks using (gradient descent) learn in O(n) whereas most other statistical methods are of higher order and cannot be applied efficiently to such large datasets. Furthermore GPU devices allow fast computation of the simple matrix algebra underlying neural nets.

The third computational aspect is to do with convolutional neural networks (which basically mirror adaptive nonlinear filters): in standard image processing, you manually develop a set of (non linear) image features (feature engineering) that you believe best captures the problem, then pass those into a classifier. And iterate (adding new features..) this manual intervention is very time consuming and may not yield very good results. With convolutional Nns this manual step is removed and you have a fully automated pipeline that you can leave the computer to optimise over (much as a chess computer would work as opposed to person)... So then the magic is developing scripts running gpus on loads of machines to evaluate different combinations of filters /filter sizes etc.

• Several problems here: first is that the computational complexity is $O(nk)$, where $k$ is the number of predictors. Second, gradient descent can be a applied to virtually any problem, so this is by no means an advantage over other estimators. Gradient descent is often a sub-optimal algorithm to use as well. Third is that the strength of neural networks is definitely not the computational cost, but rather the flexibility of the final function. – Cliff AB Jun 6 '16 at 0:28

I think there are several lines of thought here:

1. The perceptron is closely involved in what was later named the Artificial Intelligence Winter. It gives ANN an aura of mystery (they were thought gone and resurrected in the 80s).

2. They are inspired by biological systems. They could give us insights into how we work, move and live ourselves. They help us reflect on our brains, our nervous systems, our reactions and movements (there is a big research area modeling how biological neurons work). We speak here of Cognitive Science.

3. They give amazingly good results in many practical applications. When people learn how they work and how some problems can be posed, ANN actually give better results than other techniques (assuming that there are enough data to support the problem). This apparent easiness of use may lead to think that they do magic...

4. There is people claiming that science and technology is drifting paradigms, from a mechanistic way of viewing the world, to a more connectionist one. We often tend to speak in terms of networks, connections, paths or nodes (where other people would have spoken about structures or mechanisms a while ago). This is an issue on philosophy of science.

Neural networks are more than a set of equations. They may have a different meaning for different areas of study as well as practical applications. Actually, they give philosophers and sociologist an area to study about such deep issues such as interdisciplinary scientific communication.