Magic of neural networks?

Question

I've read a bit on neural networks and wonder why they are actually so popular at the moment. For example, a feed-forward network is a quite simple mathematical object: An input vector $\mathbf{x}\in \mathbb{R}^n$ is mapped to some output vector $\mathbf{y}\in\mathbb{R}^k$ via linear and non-linear mappings, e.g., for a two-layer NN:

$$\mathbf{y}=f(\mathbf{W}_1,\mathbf{W}_2)=h(\mathbf{W}_2g(\mathbf{W}_1\mathbf{x})).$$

Learning amounts to estimating the matrices $\mathbf{W}_1,\mathbf{W}_2$. So, asking a bit heretically: Most mathematicians would find this a very simple problem, for which well-known techniques exist. So, what's the issue about neural networks and why are they so powerful and why is there so much mystery surrounding them?

Answer 1

The reason there is such a fuss about them now is because of 'big data' in particular image and speech data (high dimensional and large number of samples, cf imagenet) . Essentially the issue is computational rather than mathematical :neural networks using (gradient descent) learn in O(n) whereas most other statistical methods are of higher order and cannot be applied efficiently to such large datasets. Furthermore GPU devices allow fast computation of the simple matrix algebra underlying neural nets.

The third computational aspect is to do with convolutional neural networks (which basically mirror adaptive nonlinear filters): in standard image processing, you manually develop a set of (non linear) image features (feature engineering) that you believe best captures the problem, then pass those into a classifier. And iterate (adding new features..) this manual intervention is very time consuming and may not yield very good results. With convolutional Nns this manual step is removed and you have a fully automated pipeline that you can leave the computer to optimise over (much as a chess computer would work as opposed to person)... So then the magic is developing scripts running gpus on loads of machines to evaluate different combinations of filters /filter sizes etc.

Answer 2

I think there are several lines of thought here:

1. The perceptron is closely involved in what was later named the Artificial Intelligence Winter. It gives ANN an aura of mystery (they were thought gone and resurrected in the 80s).

2. They are inspired by biological systems. They could give us insights into how we work, move and live ourselves. They help us reflect on our brains, our nervous systems, our reactions and movements (there is a big research area modeling how biological neurons work). We speak here of Cognitive Science.

3. They give amazingly good results in many practical applications. When people learn how they work and how some problems can be posed, ANN actually give better results than other techniques (assuming that there are enough data to support the problem). This apparent easiness of use may lead to think that they do magic...

4. There is people claiming that science and technology is drifting paradigms, from a mechanistic way of viewing the world, to a more connectionist one. We often tend to speak in terms of networks, connections, paths or nodes (where other people would have spoken about structures or mechanisms a while ago). This is an issue on philosophy of science.

Neural networks are more than a set of equations. They may have a different meaning for different areas of study as well as practical applications. Actually, they give philosophers and sociologist an area to study about such deep issues such as interdisciplinary scientific communication.