Why are neural networks described as black-box models?

Question

I often hear people talking about neural networks as something as a black-box that you don't understand what it does or what they mean. I actually I can't understand what they mean by that! If you understand how back-propagation works, then how is it a black-box?

Do they mean that we don't understand how the weights that were computed or what?

Answer 1

A neural network is a black box in the sense that while it can approximate any function, studying its structure won't give you any insights on the structure of the function being approximated.

As an example, one common use of neural networks on the banking business is to classify loaners on "good payers" and "bad payers". You have a matrix of input characteristics $C$ (sex, age, income, etc) and a vector of results $R$ ("defaulted", "not defaulted", etc). When you model this using a neural network, you are supposing that there is a function $f(C)=R$, in the proper sense of a mathematical function. This function f can be arbitrarily complex, and might change according to the evolution of the business, so you can't derive it by hand.

Then you use the Neural Network to build an approximation of $f$ that has a error rate that is acceptable to your application. This works, and the precision can be arbitrarily small - you can expand the network, fine tune its training parameters and get more data until the precision hits your goals.

The black box issue is: The approximation given by the neural network will not give you any insight on the form of f. There is no simple link between the weights and the function being approximated. Even the analysis of which input characteristic is irrelevant is a open problem (see this link).

Plus, from a traditional statistics viewpoint, a neural network is a non-identifiable model: Given a dataset and network topology, there can be two neural networks with different weights but exactly the same result. This makes the analysis very hard.

As an example of "non-black box models", or "interpretable models", you have regression equations and decision trees. The first one gives you a closed form approximation of f where the importance of each element is explicit, the second one is a graphical description of some relative risks\odds ratios.

Answer 2

Google has published Inception-v3. It's a Neural Network (NN) for image classification algorithm (telling a cat from a dog).

In the paper they talk about the current state of image classification

For example, GoogleNet employed only 5 million parameters, which represented a 12x reduction with respect to its predecessor AlexNet, which used 60 million parameters.Furthermore, VGGNet employed about 3x more parameters than AlexNet

and that is basically why we call NN for black boxes. If I train an image classification model - with 10 million parameters - and hand it to you. What can you do with it?

You can certainly run it and classify images. It will work great! But you can not answer any of the following questions by studying all the weights, biases and network structure.

• Can this network tell a Husky from a Poodle?
• Which objects are easy to classify for the algorithm, which are difficult?
• Which part of a dog is the most important for being able to classify it correctly? The tail or the foot?
• If I photoshop a cats head on a dog, what happens, and why?

You can maybe answer the questions by just running the NN and see the result (black box), but you have no change of understanding why it is behaving the way it does in edge cases.