# Meaning of a neural network as a black-box?

I often hear people talking about neural networks as something as a black-box that you don't understand what it does or what they mean. I actually I can't understand what they mean by that! If you understand how back-propagation works, then how is it a black-box?

Do they mean that we don't understand how the weights that were computed or what?

-

A neural network is a black box in the sense that while it can approximate any function, studying its structure won't give you any insights on the structure of the function being approximated.

As an example, one common use of neural networks on the banking business is to classify loaners on "good payers" and "bad payers". You have a matrix of input characteristics C (sex, age, income, etc) and a vector of results R ("defaulted", "not defaulted", etc). When you model this using a neural network, you are supposing that there is a function f(C)=R, in the proper sense of a mathematical function. This function f can be arbitrarily complex, and might change according to the evolution of the business, so you can't derive it by hand.

Then you use the Neural Network to build an approximation of f that has a error rate that is acceptable to your application. This works, and the precision can be arbitrarily small - you can expand the network, fine tune its training parameters and get more data until the precision hits your goals.

The black box issue is: The approximation given by the neural network will not give you any insight on the form of f. There is no simple link between the weights and the function being approximated. Even the analysis of which input characteristic is irrelevant is a open problem (see this link).

Plus, from a traditional statistics viewpoint, a neural network is a non-identifiable model: Given a dataset and network topology, there can be two neural networks with different weights and same result. This makes the analysis very hard.

As an example of "non-black box models", or "interpretable models", you have regression equations and decision trees. The first one gives you a closed form approximation of f where the importance of each element is explicit, the second one is a graphical description of some relative risks\odds ratios.

-

Maybe this would help: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

This article tries to uncover the underlying mechanism of neural networks from a topological perspective, it offers a lot of brilliant insights to explain the performance of neural networks.

-