# Why are neural networks described as black-box models?

I often hear people talking about neural networks as something as a black-box that you don't understand what it does or what they mean. I actually I can't understand what they mean by that! If you understand how back-propagation works, then how is it a black-box?

Do they mean that we don't understand how the weights that were computed or what?

• Maybe this would help: colah.github.io/posts/2014-03-NN-Manifolds-Topology This article tries to uncover the underlying mechanism of neural networks from a topological perspective, it offers a lot of brilliant insights to explain the performance of neural networks. – Sol Apr 16 '14 at 17:37
• I like to add point to Jack, when we look at MLP in machine learning point of view, neural networks are not black box anymore. With simple sigmoid function we shall be able to interpret input and out relation with an equation. – user131276 Sep 16 '16 at 6:23
• It depends on the complexity of the model. You can have simple neural networks that can be considered interpretable models. Usually, in practical applications, they are black-boxes because, as said by others, it's pretty impossible to get any insight about the learnt function from the parameters. The large number of parameters and the typical non-linearity of the activation functions are the main reasons why this task is practically impossible. A black box model is a model that is extremely hard or practically impossible to interpret. – user289381 Jul 29 '20 at 20:22

A neural network is a black box in the sense that while it can approximate any function, studying its structure won't give you any insights on the structure of the function being approximated.

As an example, one common use of neural networks on the banking business is to classify loaners on "good payers" and "bad payers". You have a matrix of input characteristics $$C$$ (sex, age, income, etc) and a vector of results $$R$$ ("defaulted", "not defaulted", etc). When you model this using a neural network, you are supposing that there is a function $$f(C)=R$$, in the proper sense of a mathematical function. This function f can be arbitrarily complex, and might change according to the evolution of the business, so you can't derive it by hand.

Then you use the Neural Network to build an approximation of $$f$$ that has a error rate that is acceptable to your application. This works, and the precision can be arbitrarily small - you can expand the network, fine tune its training parameters and get more data until the precision hits your goals.

The black box issue is: The approximation given by the neural network will not give you any insight on the form of f. There is no simple link between the weights and the function being approximated. Even the analysis of which input characteristic is irrelevant is a open problem (see this link).

Plus, from a traditional statistics viewpoint, a neural network is a non-identifiable model: Given a dataset and network topology, there can be two neural networks with different weights but exactly the same result. This makes the analysis very hard.

As an example of "non-black box models", or "interpretable models", you have regression equations and decision trees. The first one gives you a closed form approximation of f where the importance of each element is explicit, the second one is a graphical description of some relative risks\odds ratios.

• As this is an old answer, it may be useful to some to provide some newly developed tools: "The approximation given by the neural network will not give you any insight on the form of f" - I'd say that SHAP now does a great job of model explanation, even for neural nets. "Even the analysis of which input characteristic is irrelevant is a open problem" - mothods like permutation importance, as well as SHAP, now tackle this problem pretty well. – Bobson Dugnutt Sep 3 '19 at 15:44

Google has published Inception-v3. It's a Neural Network (NN) for image classification algorithm (telling a cat from a dog).

In the paper they talk about the current state of image classification

For example, GoogleNet employed only 5 million parameters, which represented a 12x reduction with respect to its predecessor AlexNet, which used 60 million parameters.Furthermore, VGGNet employed about 3x more parameters than AlexNet

and that is basically why we call NN for black boxes. If I train an image classification model - with 10 million parameters - and hand it to you. What can you do with it?

You can certainly run it and classify images. It will work great! But you can not answer any of the following questions by studying all the weights, biases and network structure.

• Can this network tell a Husky from a Poodle?
• Which objects are easy to classify for the algorithm, which are difficult?
• Which part of a dog is the most important for being able to classify it correctly? The tail or the foot?
• If I photoshop a cats head on a dog, what happens, and why?

You can maybe answer the questions by just running the NN and see the result (black box), but you have no change of understanding why it is behaving the way it does in edge cases.

• I think at least one of the questions ('Which part of a dog is the most important for being able to classify it correctly? The tail or the foot?') is quite answerable, if you look at Matt Zeiler's article and video on deconvolutitonal networks – Alex Nov 8 '17 at 21:03