# What did my neural network just learn? What features does it care about and why?

A neural net learns features of a data set as a means of achieving some goal. When it is done, we may want to know what the neural net learned. What were the features and why did it care about those. Can someone give some references on the body of work that concerns this problem?

It is true that it's hard to understand what a neural network is learning but there has been a lot of work on that front. We definitely can get some idea of what our network is looking for.

Let's consider the case of a convolutional neural net for images. We have the interpretation for our first layer that we are sliding $K$ filters over the image, so our first hidden layer corresponds to the agreement between small chunks of the image and our various filters. We can visualize these filters to see what our first layer of representation is:

This picture is of the first layer of filters from an AlexNet and is taken from this wonderful tutorial: http://cs231n.github.io/understanding-cnn/. This lets us interpret the first hidden layer as learning to represent the image, consisting of raw pixels, as a tensor where each coordinate is the agreement of a filter with a small region of the image. The next layer then is working with these filter activations.

It's not so hard to understand the first hidden layer because we can just look at the filters to see how they behave, because they're directly applied to an input image. E.g. let's say you're working with a black and white image (so our filters are 2D rather than 3D) and you have a filter that's something like $$\begin{bmatrix}0 & 1 & 0 \\ 1 & -4 & 1 \\ 0 & 1 & 0\end{bmatrix}.$$

Imagine applying this to a 3x3 region of an image (ignoring the bias term). If every pixel was the same color then you'd get $0$ since they'd cancel out. But if the upper half is different from the lower half, say, then you'll get a potentially large value. This filter, in fact, is an edge detector, and we can figure that out by actually just applying it to images and seeing what happens. But it's a lot harder to understand the deeper layers because the whole problem is we don't know how to interpret what we're applying the filters to.

This paper by Erhan et al (2009) agrees with this: they say that first hidden layer visualizations are common (and that was back in 2009) but visualizing the deeper layers is the hard part. From that paper:

The main experimental finding of this investigation is very surprising: the response of an internal unit to input images, as a function in image space, appears to be unimodal, or at least that the maximum is found reliably and consistently for all the random initializations tested. This is interesting because finding this dominant mode is relatively easy, and displaying it then provides a good characterization of what the unit does.

Chris Olah et al (https://distill.pub/2017/feature-visualization/) build on this and discuss how in general you can (1) generate images that lead to large activations in order to get a sense of what the network is looking for; or (2) take actual input images and see how different parts of the image activate the network. That post focuses on (1).

In the image below, taken from that linked article by Olah et al., the authors discuss the different aspects of the network that you can inspect. The left-most image shows the result of optimizing the activation of a particular neuron over the input image space, and so on.

I would highly recommend reading that article in its entirety if you want a deeper understanding of this, and by reading its references you should have a great grasp of what's been done with this.

Now of course this was all just for images where we as humans can make sense of the inputs. If you're working with something harder to interpret, like just a big vector of numbers, then you may not be able to make such cool visualizations, but in principle you could still consider these techniques for assessing the various neurons, layers, and etc.

• +1 especially for the final paragraph. I personally think vision people are over doing visualizations on these large number of parameters. – Haitao Du Jan 11 '18 at 19:20
• of course +1 for the well written long answer with credible references. – Haitao Du Jan 11 '18 at 19:21
• @hxd1011 thank you, and I think Olah et al make a great point in that article that it's not really fair to look at just what maximizes a single neuron since really the layer is like a basis and it's more about the space the layer represents than the exact basis vectors. So while fun to look at, a lot of those visualizations may not be super insightful – jld Jan 11 '18 at 19:27
• Also a common misconception is that when taking the first layer filters, a patch of image that will maximalize the output of a given neuron looks exactly like that filter. Actually not, a patch maximalizing the output of a neuron looks like nothing resembling the numerical values of filter, since its convolution, not template matching. That's where I believe adversarial examples reside. – Lugi Jan 11 '18 at 21:38
• Olah et al. blog post is nice, but they make a mistake - they "optimize for maximum activation" (not exactly that, but close), find a picture which looks nothing like pictures from the training set and conclude that CNNs "see" differently from us. While the conclusion is obviously true, the reasoning is flawed - the mode of a high-dimensional distribution doesn't look anything like a typical sample from that distribution. In other words, it's precisely because CNNs need to classify cats, that the mode of the distribution must not be a cat, but a "nightmare cat". – DeltaIV Jan 12 '18 at 18:06

Neural Network is one of the black box models that would not give "easy to understand" rules / or what has been learned.

Specifically, what has been learned are the parameters in the model, but the parameters can be large: hundreds of thousands of the parameters is very normal.

In addition, it is also not clear on the important features learned, you can understand the model uses all the features, with many complicated operations to derive the results, where not easy to say in plain English how the model transform each feature use it.

In fact, the one layer neural network (without hidden layer) with logistic function as activation function is identical to logistic regression. Logistic regression is very rich in interpretations. Here is one example. But with complex neural network / more hidden layers, such interpretation will not apply.