Can't deep learning models now be said to be interpretable? Are nodes features?

Question

For statistical and machine learning models, there are multiple levels of interpretability: 1) the algorithm as a whole, 2) parts of the algorithm in general 3) parts of the algorithm on particular inputs, and these three levels split into two parts each, one for training and one for function eval. The last two parts are much closer than to the first. I'm asking about #2, which usually leads to better understanding of #3). (if those are not what 'interpretability' means then what should I be thinking?)

As far as interpretability goes, logistic regression is one of the easiest to interpret. Why did this instance pass the threshold? Because that instance had this particular positive feature and it has a larger coefficient in the model. It's so obvious!

A neural network is the classic example of a model that is difficult to interpret. What do all those coefficients mean? They all add up in such complicated crazy ways that it is hard to say what any particular coefficient is really doing.

But with all the deep neural nets coming out, it feels like things are becoming clearer. The DL models (for say vision) seem to capture things like edges or orientation in early layers, and in later layers it seems like some nodes are actually semantic (like the proverbial 'grandmother cell'). For example:

(from 'Learning About Deep Learning')

This is a graphic (of many out there) created by hand for presentation so I am very skeptical. But it is evidence that somebody thinks that is how it works.

Maybe in the past there just weren't enough layers for us to find recognizable features; the models were successful, just not easy to post-hoc analyze particular ones.

But maybe the graphic is just wishful thinking. Maybe NNs are truly inscrutable.

But the many graphics with their nodes labeled with pictures are also really compelling.

Do DL nodes really correspond to features?

Answer 1

Interpretation of deep models is still challenging.

1. Your post only mentions CNNs for computer vision applications, but (deep or shallow) feed-forward networks and recurrent networks remain challenging to understand.

2. Even in the case of CNNs which have obvious "feature detector" structures, such as edges and orientation of pixel patches, it's not completely obvious how these lower-level features are aggregated upwards, or what, precisely, is going on when these vision features are aggregated in a fully-connected layer.

3. Adversarial examples show how interpretation of the network is difficult. An adversarial example has some tiny modification made to it, but results in a dramatic shift in the decision made by the model. In the context of image classification, a tiny amount of noise added to an image can change an image of a lizard to have a highly confident classification as another animal, like a (species of) dog.

This is related to interpretability in the sense that there is a strong, unpredictable relationship between the (small) amount of noise and the (large) shift in the classification decision. Thinking about how these networks operate, it makes some sense: computations at previous layers are propagated forward, so that a number of errors -- small, unimportant errors to a human -- are magnified and accumulate as more and more computations are performed using the "corrupted" inputs.

On the other hand, the existence of adversarial examples shows that the interpretation of any node as a particular feature or class is difficult, since the fact that the node is activated might have little to do with the actual content of the original image, and that this relationship is not really predictable in terms of the original image. But in the example images below, no humans are deceived about the content of the images: you wouldn't confuse the flag pole for a dog. How can we interpret these decisions, either in aggregate (a small noise pattern "transmutes" a lizard into dog, or a flagpole into a dog) or in smaller pieces (that several feature detectors are more sensitive to the noise pattern than the actual image content)?

HAAM is a promising new method to generate adversarial images using harmonic functions. ("Harmonic Adversarial Attack Method" Wen Heng, Shuchang Zhou, Tingting Jiang.) Images generated using this method can be used to emulate lighting/shadow effects and are generally even more challenging for humans to detect as having been altered.

As an example, see this image, taken from "Universal adversarial perturbations", by Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. I chose this image just because it was one of the first adversarial images I came across. This image establishes that a particular noise pattern has a strange effect on the image classification decision, specifically that you can make a small modification to an input image and make the classifier think the result is a dog. Note that the underlying, original image is still obvious: in all cases, a human would not be confused into thinking that any of the non-dog images are dogs.

Here's a second example from a more canonical paper, "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES" by Ian J. Goodfellow, Jonathon Shlens & Christian Szegedy. The added noise is completely indistinguishable in the resulting image, yet the result is very confidently classified as the wrong result, a gibbon instead of a panda. In this case, at least, there is at least a passing similarity between the two classes, since gibbons and pandas are at least somewhat biologically and aesthetically similar in the broadest sense.

This third example is taken from "Generalizable Adversarial Examples Detection Based on Bi-model Decision Mismatch" by João Monteiro, Zahid Akhtar and Tiago H. Falk. It establishes that the noise pattern can be indistinguishable to a human yet still confuse the classifier.

For reference, a mudpuppy is a dark-colored animal with four limbs and a tail, so it does not really have much resemblance to a goldfish.

4. I just found this paper today. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus. "Intriguing properties of neural networks". The abstract includes this intriguing quotation:

First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks.

So, rather than having 'feature detectors' at the higher levels, the nodes merely represent coordinates in a feature space which the network uses to model the data.

Answer 2

Layers don't map onto successively more abstract features as cleanly as we'd like. A good way to see this is to compare two very popular architectures.

VGG16 consists of many convolutional layers stacked on top of each other with the occasional pooling layer -- a very traditional architecture.

Since then, people have moved on to designing residual architectures, where each layer is connected to not only the previous layer, but also one (or possibly more) layers farther down in the model. ResNet was one of the first to do this, and has around 100 layers, depending on which variant you use.

While VGG16 and similar networks do have layers act in a more or less interpretable manner -- learning higher and higher level features, ResNets do not do this. Instead, people have proposed that they either keep refining features to make them more accurate or that they're just a bunch of shallow networks in disguise, neither of which matches the "traditional views" on what deep models learn.

While ResNet and similar architectures handily outperform VGG in image classification and object detection, there seem to be some applications for which the simple bottom-up feature hierarchy of VGG is very important. See here for a good discussion.

So given that more modern architectures don't seem to fit into the picture anymore, I would say that we can't quite say CNNs are interpretable yet.

Answer 3

The subject of my Ph.D dissertation was to reveal the black-box properties of neural networks, specifically feed-forward neural networks, with one or two hidden layers.

I will take up the challenge to explain to everyone what the weights and bias terms mean, in a one-layer feed-forward neural network. Two different perspectives will be addressed: a parametric one and a probabilistic one.

In the following, I assume that the input values provided to each input neuron have all been normalized to the interval (0,1), by linear scaling ($x_{input}=\alpha \cdot x + \beta$), where the two coefficients $\alpha$ and $\beta$ are chosen per input variable, such that $x_{input} \in (0,1)$. I make a distinction between real-numbered variables, and enumerated variables (with a boolean variable as a special case enumerated variable):

• A real-numbered variable is provided as a decimal number between $0$ and $1$, after linear scaling.
• An enumerated variable, take the days of the week (monday, tuesday, etc.) are represented by $v$ input nodes, with $v$, being the number of enurable outcomes, i.e. $7$ for the number of days in a week.

Such a representation of your input data is required in order to be able to interpret the (absolute value) size of the weights in the input layer.

Parametric meaning:

• the larger the absolute value of the weight is between an input neuron and a hidden neuron, the more important that variable is, for the 'fireing' of that particular hidden node. Weights close to $0$ indicate that an input value is as good as irelevant.
• the weight from a hidden node to an output node indicates that the weighted amplification of the input variables that are in absolute sense most amplified by that hidden neuron, that they promote or dampen the particular output node. The sign of the weight indicates promotion (positive) or inhibition (negative).
• the third part not explicitly represented in the parameters of the neural network is the multivariate distribution of the input variables. That is, how often does it occur that the value $1$ is provided to input node $3$ - with the really large weight to hidden node $2$ ?
• a bias term is just a translation constant that shifts the average of a hidden (or output) neuron. It acts like the shift $\beta$, presented above.

Reasoning back from an output neuron: which hidden neurons have the highest absolute weight values, on their connections to the output neurons? How often does the activation of each hidden node become close to $1$ (assuming sigmoid activation functions). I'm talking about frequencies, measured over the training set. To be precise: what is the frequency with which the hidden nodes $i$ and $l$, with large weights to the input variables $t$ and $s$, that these hidden nodes $i$ and $l$ are close to $1$? Each hidden node propagates a weighted average of its input values, by definition. Which input variables does each hidden node primarily promote - or inhibit? Also the $\Delta_{j,k}=\mid w_{i,j} - w_{i,k}\mid$ explains much, the absolute difference in weights between the weights that fan out from hidden node $i$ to the two output nodes $j$ and $k$.

The more important hidden nodes are for an output node (talking in frequencies, over the training set), which 'input weights times input frequencies' are most important? Then we close in on the significance of the parameters of feed-forward neural networks.

Probabilistic interpretation:

The probabilistic perspective means to regard a classification neural network as a Bayes classifier (the optimal classifier, with the theoretically defined lowest error-rate). Which input variables have influence on the outcome of the neural network - and how often? Regard this as a probabilistic sensitivithy analysis. How often can varying one input variable lead to a different classification? How often does input neuron $x_{input}$ have potential influence on which classification outcome becomes the most likely, implying that the corresponding output neuron achieves the highest value?

Individual case - pattern

When varying a real-numbered input neuron $x_{input}$ can cause the most likely classification to change, we say that this variable has potential influence. When varying the outcome of an enumerated variable (changing weekday from monday $[1,0,0,0,0,0,0]$ to tuesday $[0,1,0,0,0,0,0]$, or any other weekday), and the most likely outcome changes, then that enumerated variable has potential influence on the outcome of the classification.

When we now take the likelihood of that change into account, then we talk out expected influence. What is the probability of observing a changing input variable $x_{input}$ such that a the input case changes outcome, given the values of all the other inputs? Expected influence refers to expected value, of $x_{input}$, namely $E(x_{input} \mid {\bf x}_{-input})$. Here ${\bf x}_{-input}$ is the vector of all input values, except from input $x_{input}$. Keep in mind that an enumerated variable is represented by a number of input neurons. These possible outcomes are here regarded as one variable.

Deep leaning - and the meaning of the NN parameters

When applied to computer vision, neural networks have shown remarkable progress in the last decade. The convolutional neural networks introduced by LeCunn in 1989 have turned out to eventually perform really well in terms of image recognition. It has been reported that they can outperform most other computer-based recognition approaches.

Interesting emergent properties appear when convolutional neural networks are being trained for object recognition. The first layer of hidden nodes represents low-level feature detectors, similar to the scale-space operators T. Lindeberg, Feature Detection with Automatic Scale Selection, 1998. These scale-space operators detect

• lines,
• corners,
• T-junctions

and some other basic image features.

Even more interesting is the fact that perceptual neurons in mammal brains have been shown to resemble this way of working in the first steps of (biological) image processing. So with CNNs, the scientific community is closing in on what makes human perception so phenomenal. This makes it very worthwhile to pursue this line of research further.