For statistical and machine learning models, there are multiple levels of interpretability: 1) the algorithm as a whole, 2) parts of the algorithm in general 3) parts of the algorithm on particular inputs, and these three levels split into two parts each, one for training and one for function eval. The last two parts are much closer than to the first. I'm asking about #2, which usually leads to better understanding of #3). (if those are not what 'interpretability' means then what should I be thinking?)

As far as interpretability goes, logistic regression is one of the easiest to interpret. Why did this instance pass the threshold? Because that instance had this particular positive feature and it has a larger coefficient in the model. It's so obvious!

A neural network is the classic example of a model that is difficult to interpret. What do all those coefficients mean? They all add up in such complicated crazy ways that it is hard to say what any particular coefficient is really doing.

But with all the deep neural nets coming out, it feels like things are becoming clearer. The DL models (for say vision) seem to capture things like edges or orientation in early layers, and in later layers it seems like some nodes are actually semantic (like the proverbial 'grandmother cell'). For example:

enter image description here

(from 'Learning About Deep Learning')

This is a graphic (of many out there) created by hand for presentation so I am very skeptical. But it is evidence that somebody thinks that is how it works.

Maybe in the past there just weren't enough layers for us to find recognizable features; the models were successful, just not easy to post-hoc analyze particular ones.

But maybe the graphic is just wishful thinking. Maybe NNs are truly inscrutable.

But the many graphics with their nodes labeled with pictures are also really compelling.

Do DL nodes really correspond to features?

  • 6
    $\begingroup$ I don't see the premise of this question. That neural nets have become more complex and give better predictions doesn't make them any more interpretable. The opposite is usually true: complexity/better prediction <-> simplicity/better interpretation. $\endgroup$
    – AdamO
    Commented May 4, 2018 at 22:30
  • 4
    $\begingroup$ @AdamO is exactly correct. Because of that, regression trees (recursive partitioning) are only intepretable because the results are wrong. They are wrong because they are volatile; get a new sample and the tree can be arbitrarily different. And single trees are not competitive with respect to predictive discrimination. Parsimony is often the enemy of predictive discrimination. And to the original question, in the biomedical field, AI/ML results have not been interpretable. $\endgroup$ Commented May 5, 2018 at 10:57
  • 2
    $\begingroup$ See this article AI researchers allege that machine learning is alchemy sciencemag.org/news/2018/05/… $\endgroup$
    – user78229
    Commented May 5, 2018 at 13:10
  • 4
    $\begingroup$ The bold question in the body and the question in your title are very different. It looks like all of the answers, including mine, are addressing the question in the title. Perhaps you could ask the narrower question about nodes and features in its own thread? But before you do that, consider that you've already linked to a paper that answers your bold question in the affirmative, so consider what, precisely, you'd like to learn in an answer before asking. $\endgroup$
    – Sycorax
    Commented May 5, 2018 at 13:32
  • $\begingroup$ @Sycorax The link I just added is to a blog post, not a paper, and so I am very skeptical of the affirmative view. The variety of interpretability that I ask about DL in the title I consider to be the one in bold in the text. $\endgroup$
    – Mitch
    Commented May 5, 2018 at 13:39

3 Answers 3


Interpretation of deep models is still challenging.

  1. Your post only mentions CNNs for computer vision applications, but (deep or shallow) feed-forward networks and recurrent networks remain challenging to understand.

  2. Even in the case of CNNs which have obvious "feature detector" structures, such as edges and orientation of pixel patches, it's not completely obvious how these lower-level features are aggregated upwards, or what, precisely, is going on when these vision features are aggregated in a fully-connected layer.

  3. Adversarial examples show how interpretation of the network is difficult. An adversarial example has some tiny modification made to it, but results in a dramatic shift in the decision made by the model. In the context of image classification, a tiny amount of noise added to an image can change an image of a lizard to have a highly confident classification as another animal, like a (species of) dog.

This is related to interpretability in the sense that there is a strong, unpredictable relationship between the (small) amount of noise and the (large) shift in the classification decision. Thinking about how these networks operate, it makes some sense: computations at previous layers are propagated forward, so that a number of errors -- small, unimportant errors to a human -- are magnified and accumulate as more and more computations are performed using the "corrupted" inputs.

On the other hand, the existence of adversarial examples shows that the interpretation of any node as a particular feature or class is difficult, since the fact that the node is activated might have little to do with the actual content of the original image, and that this relationship is not really predictable in terms of the original image. But in the example images below, no humans are deceived about the content of the images: you wouldn't confuse the flag pole for a dog. How can we interpret these decisions, either in aggregate (a small noise pattern "transmutes" a lizard into dog, or a flagpole into a dog) or in smaller pieces (that several feature detectors are more sensitive to the noise pattern than the actual image content)?

HAAM is a promising new method to generate adversarial images using harmonic functions. ("Harmonic Adversarial Attack Method" Wen Heng, Shuchang Zhou, Tingting Jiang.) Images generated using this method can be used to emulate lighting/shadow effects and are generally even more challenging for humans to detect as having been altered.

As an example, see this image, taken from "Universal adversarial perturbations", by Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. I chose this image just because it was one of the first adversarial images I came across. This image establishes that a particular noise pattern has a strange effect on the image classification decision, specifically that you can make a small modification to an input image and make the classifier think the result is a dog. Note that the underlying, original image is still obvious: in all cases, a human would not be confused into thinking that any of the non-dog images are dogs. adversaria

Here's a second example from a more canonical paper, "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES" by Ian J. Goodfellow, Jonathon Shlens & Christian Szegedy. The added noise is completely indistinguishable in the resulting image, yet the result is very confidently classified as the wrong result, a gibbon instead of a panda. In this case, at least, there is at least a passing similarity between the two classes, since gibbons and pandas are at least somewhat biologically and aesthetically similar in the broadest sense. panda

This third example is taken from "Generalizable Adversarial Examples Detection Based on Bi-model Decision Mismatch" by João Monteiro, Zahid Akhtar and Tiago H. Falk. It establishes that the noise pattern can be indistinguishable to a human yet still confuse the classifier. indistinguishable

For reference, a mudpuppy is a dark-colored animal with four limbs and a tail, so it does not really have much resemblance to a goldfish. mudpuppy

  1. I just found this paper today. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus. "Intriguing properties of neural networks". The abstract includes this intriguing quotation:

First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks.

So, rather than having 'feature detectors' at the higher levels, the nodes merely represent coordinates in a feature space which the network uses to model the data.

  • 3
    $\begingroup$ Good points. But note that even in the simplest models (logistic, decision trees) it is obscure why any particular coefficient /threshild is what it is (but that's not the same interpretability I was asking about). Not totally unrelatedly, are there any good adversarial studies/examples for language/RNNs/LSTMs? $\endgroup$
    – Mitch
    Commented May 4, 2018 at 17:23
  • 7
    $\begingroup$ I would also point out that even simple models such as logistic regression are vulnerable to adversarial attacks. In fact, Goodfellow et. al. points out that it is shallow models which lack the capability to resist such attacks. Yet we still claim to be able to interpret logistic regression. $\endgroup$
    – shimao
    Commented May 4, 2018 at 17:47
  • 2
    $\begingroup$ good answer, except for point 3 which is moot for two reasons. 1) It's very seldom the case that an adversarial image is ", to a human, indistinguishable from an unmodified image", unless that human has serious visual impairments. Nearly always you can notice that the image has some noise pattern added to it, especially in the background, aesthetically resembling so-called JPEG-noise (only visually: the actual statistical properties of the perturbation are different). What's surprising is not that the classifier is uncertain whether it's a bona fide cat rather than, say, a corrupted 1/ $\endgroup$
    – DeltaIV
    Commented May 5, 2018 at 10:12
  • 2
    $\begingroup$ 2/ image, but that's it's nearly certain that it's a bus. 2) How are adversarial examples related to interpretability? Linear models, generalized linear models and also decision trees are susceptible to adversarial examples. It's actually easier to find an adversarial example which fools logistic regression, rather than one which fools ResNet. Notwithstanding this, we usually consider (G)LM to be interpretable model, so I wouldn't associate the existence of adversarial examples with the interpretability of a model. $\endgroup$
    – DeltaIV
    Commented May 5, 2018 at 10:15
  • 3
    $\begingroup$ @DeltaIV the point is not that you can't notice the noise. Every jpeg that's been degraded too much has noise. The point is that the noise can be manipulated to make the DNN do crazy things, things which make no sense to a human observer even if the noise itself can be seen. $\endgroup$
    – Hong Ooi
    Commented May 5, 2018 at 20:30

Layers don't map onto successively more abstract features as cleanly as we'd like. A good way to see this is to compare two very popular architectures.

VGG16 consists of many convolutional layers stacked on top of each other with the occasional pooling layer -- a very traditional architecture.

Since then, people have moved on to designing residual architectures, where each layer is connected to not only the previous layer, but also one (or possibly more) layers farther down in the model. ResNet was one of the first to do this, and has around 100 layers, depending on which variant you use.

While VGG16 and similar networks do have layers act in a more or less interpretable manner -- learning higher and higher level features, ResNets do not do this. Instead, people have proposed that they either keep refining features to make them more accurate or that they're just a bunch of shallow networks in disguise, neither of which matches the "traditional views" on what deep models learn.

While ResNet and similar architectures handily outperform VGG in image classification and object detection, there seem to be some applications for which the simple bottom-up feature hierarchy of VGG is very important. See here for a good discussion.

So given that more modern architectures don't seem to fit into the picture anymore, I would say that we can't quite say CNNs are interpretable yet.

  • $\begingroup$ Presumably the entirely unengineered/undesigned topology of a DL network would be a large random partial ordered set, input the sensors and output the desired function (that is, no attempt at layering at all, let the training figure it out). The nodes here would be very inscrutable. But doesn't that sort of imply that the more designed a topology is, the more likely it does have some interpretability? $\endgroup$
    – Mitch
    Commented May 6, 2018 at 16:44
  • 4
    $\begingroup$ @Mitch Some recent architectures such as Densenet seem to be slowly creeping towards the limit of having every layer connected to every other layer -- much like your "undesigned network". But surely, ResNet and Densenet have a more sophisticated design than VGG16, yet one could say they are less interpretable -- so no, I don't think more design means more interpretable. Possible, sparser connections means more interpretable. $\endgroup$
    – shimao
    Commented May 6, 2018 at 16:54

The subject of my Ph.D dissertation was to reveal the black-box properties of neural networks, specifically feed-forward neural networks, with one or two hidden layers.

I will take up the challenge to explain to everyone what the weights and bias terms mean, in a one-layer feed-forward neural network. Two different perspectives will be addressed: a parametric one and a probabilistic one.

In the following, I assume that the input values provided to each input neuron have all been normalized to the interval (0,1), by linear scaling ($x_{input}=\alpha \cdot x + \beta$), where the two coefficients $\alpha$ and $\beta$ are chosen per input variable, such that $x_{input} \in (0,1)$. I make a distinction between real-numbered variables, and enumerated variables (with a boolean variable as a special case enumerated variable):

  • A real-numbered variable is provided as a decimal number between $0$ and $1$, after linear scaling.
  • An enumerated variable, take the days of the week (monday, tuesday, etc.) are represented by $v$ input nodes, with $v$, being the number of enurable outcomes, i.e. $7$ for the number of days in a week.

Such a representation of your input data is required in order to be able to interpret the (absolute value) size of the weights in the input layer.

Parametric meaning:

  • the larger the absolute value of the weight is between an input neuron and a hidden neuron, the more important that variable is, for the 'fireing' of that particular hidden node. Weights close to $0$ indicate that an input value is as good as irelevant.
  • the weight from a hidden node to an output node indicates that the weighted amplification of the input variables that are in absolute sense most amplified by that hidden neuron, that they promote or dampen the particular output node. The sign of the weight indicates promotion (positive) or inhibition (negative).
  • the third part not explicitly represented in the parameters of the neural network is the multivariate distribution of the input variables. That is, how often does it occur that the value $1$ is provided to input node $3$ - with the really large weight to hidden node $2$ ?
  • a bias term is just a translation constant that shifts the average of a hidden (or output) neuron. It acts like the shift $\beta$, presented above.

Reasoning back from an output neuron: which hidden neurons have the highest absolute weight values, on their connections to the output neurons? How often does the activation of each hidden node become close to $1$ (assuming sigmoid activation functions). I'm talking about frequencies, measured over the training set. To be precise: what is the frequency with which the hidden nodes $i$ and $l$, with large weights to the input variables $t$ and $s$, that these hidden nodes $i$ and $l$ are close to $1$? Each hidden node propagates a weighted average of its input values, by definition. Which input variables does each hidden node primarily promote - or inhibit? Also the $\Delta_{j,k}=\mid w_{i,j} - w_{i,k}\mid$ explains much, the absolute difference in weights between the weights that fan out from hidden node $i$ to the two output nodes $j$ and $k$.

The more important hidden nodes are for an output node (talking in frequencies, over the training set), which 'input weights times input frequencies' are most important? Then we close in on the significance of the parameters of feed-forward neural networks.

Probabilistic interpretation:

The probabilistic perspective means to regard a classification neural network as a Bayes classifier (the optimal classifier, with the theoretically defined lowest error-rate). Which input variables have influence on the outcome of the neural network - and how often? Regard this as a probabilistic sensitivithy analysis. How often can varying one input variable lead to a different classification? How often does input neuron $x_{input}$ have potential influence on which classification outcome becomes the most likely, implying that the corresponding output neuron achieves the highest value?

Individual case - pattern

When varying a real-numbered input neuron $x_{input}$ can cause the most likely classification to change, we say that this variable has potential influence. When varying the outcome of an enumerated variable (changing weekday from monday $[1,0,0,0,0,0,0]$ to tuesday $[0,1,0,0,0,0,0]$, or any other weekday), and the most likely outcome changes, then that enumerated variable has potential influence on the outcome of the classification.

When we now take the likelihood of that change into account, then we talk out expected influence. What is the probability of observing a changing input variable $x_{input}$ such that a the input case changes outcome, given the values of all the other inputs? Expected influence refers to expected value, of $x_{input}$, namely $E(x_{input} \mid {\bf x}_{-input})$. Here ${\bf x}_{-input}$ is the vector of all input values, except from input $x_{input}$. Keep in mind that an enumerated variable is represented by a number of input neurons. These possible outcomes are here regarded as one variable.

Deep leaning - and the meaning of the NN parameters

When applied to computer vision, neural networks have shown remarkable progress in the last decade. The convolutional neural networks introduced by LeCunn in 1989 have turned out to eventually perform really well in terms of image recognition. It has been reported that they can outperform most other computer-based recognition approaches.

Interesting emergent properties appear when convolutional neural networks are being trained for object recognition. The first layer of hidden nodes represents low-level feature detectors, similar to the scale-space operators T. Lindeberg, Feature Detection with Automatic Scale Selection, 1998. These scale-space operators detect

  • lines,
  • corners,
  • T-junctions

and some other basic image features.

Even more interesting is the fact that perceptual neurons in mammal brains have been shown to resemble this way of working in the first steps of (biological) image processing. So with CNNs, the scientific community is closing in on what makes human perception so phenomenal. This makes it very worthwhile to pursue this line of research further.

  • $\begingroup$ This is interesting - doesn't sound like it would provide much interpretability in the case of correlated features? $\endgroup$
    – khol
    Commented May 5, 2018 at 13:13
  • $\begingroup$ The expected vallue E(.) is also known as the average of the conditional distribution, x_input given x_-input, all the other variables. Hence, correlations are fully incorporated into this expected influence concept. Note that probabilistic independence has a wider definition than 'correlation' - the latter primarily being defined for Gaussian distributed data. $\endgroup$ Commented May 5, 2018 at 23:36
  • $\begingroup$ Nice. Is this sort of a generalization of an interpretation of logistic regression to a set of stacked regression models, one feeding into the next? $\endgroup$
    – Mitch
    Commented May 6, 2018 at 16:46
  • $\begingroup$ A subset of hidden nodes can act as a logical 'OR' for an output neuron, or more like a logical 'AND'. OR occurs when one hidden node activation is enough to cause the output neuron to become close to 1. AND occurs when only a sum of hidden node activations can cause the output node activation to become close to 1. Whether more 'OR' or more 'AND', that depends on the trained weight vector of the 'fan in', into the output node. $\endgroup$ Commented May 6, 2018 at 18:15
  • $\begingroup$ @MatchMakerEE Very nice explanation! With regards to human perception, one thing that I was thinking is the following. Suppose, we see an image of a human. We can immediately detect the presence of a face (output 1 or 0). However, a hidden unit in a neural network (assume sigmoid activation) can only output a continuous value for the presence of a face. E.g. it can output 0.8 or 0.9. This discrepancy of course boils down to the substitution of step activation with sigmoid, tanh etc. Can we assign any interpretation to this continuous output? Why a certain face activates more the detector? $\endgroup$
    – ado sar
    Commented Apr 22 at 14:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.