# Neural networks - how can I interpret what a hidden layer is doing to my data?

So following the example at the end of the chapter here, I generated a neural network for digit recognition which is (surprisingly) accurate. It's a 784->100->10 feedforward network with one input layer, one hidden layer, and one output layer.

So for an example, here is an input image (reshaped to be a 28x28 image): After running through the first layer, which has a sigmoid activation function:

first_layer_output = sigmoid(np.dot(weights,input_image)+biases)

I get the following result (reduced to 100 pixels, shown as a 10x10 image): Then, I generate the final output with:

last_layer_output = sigmoid(np.dot(weights,first_layer_output)+biases)

Which has the following output: Basically it makes the correct guess. How can I interpret visually what is happening to the input image? Is there a way to visually represent what each neuron is "looking for"? I've used the Tensorflow Playground, and the layers make intuitive sense. For example, something simple like this image, I can see how the first two neurons of the hidden layer are basically just the positive and negative averages of the first two input neurons: In a similar attempt, I thought about seeing how the weights to a given neuron were altering my image so I did an element-by-element multiplication of the pixels by the weights for the most positively weighed neuron in my output layer (weights has the weights for the output layer corresponding to the number 3):

pos_n = weights.argsort()[-1]

e_by_e = np.multiply(input_image.reshape((28,28)),weights[pos_n].reshape((28,28)))

plt.imshow(e_by_e) Is the correct way to interpret the image is that this neuron is weighing positively the pixels in red (red is high according to that colormap) and conversely decreasing the weights of pixels in blue? So you can imagine the top of the '3' is negatively weighted, while the "arch" along the top half of the '3' is positively weighted?

Here's the pos_n neuron alone: Edit: Per @Lagerbaer 's suggestion below, I trained a simpler network. This network is a feedforward 784->1->2 network with one input, one hidden, and one output layer to distinguish only between 0s and 1s in the MNIST dataset. It's >99% accurate and here is an image of the weights: • Can you rewrite this so that it's a single, self-contained narrative? It's not immediately obvious which parts are obsolete and which parts we should pay attention to.
– Sycorax
Jan 10, 2017 at 22:06
• Sorry about that -- just went through and cleaned it up. Thanks Jan 10, 2017 at 22:47
• For your last imagine: It would make more sense to show weights[pos_n] instead of already multiplying it with the input. That way, you would see which parts of the input image that particular neuron "scans" for. Jan 10, 2017 at 22:53
• @Lagerbaer added the pos_n neuron above -- thanks Jan 10, 2017 at 22:57

From my own understanding, it is notoriously difficult to infer useful things from the weights of a complex neural network.

However, with 10 digits, you could play around a bit.

Don't look just at the hidden-layer neuron with the strongest weight for 3, look at the top few.

Then look at those hidden-layer neuron's input weights, but without multiplying with the input image already. That shows you which regions of the input image the neuron "scans" for.

Now if you think heuristically about how such a neural network will work: It'll have to recognize broad features that make a 3 a 3 and discourage features that make it not a 3. So, positive weight for the arches, negative weight for a closed top loop (because that could be a hastily written 9 instead).

The problem is that it's getting quite complicated quite fast. As a simpler exercise, you could try to build a smaller neural network that just distinguishes between two relatively dissimilar shapes and build your intuition from there.

• Thanks - I tried your suggestion and trained a network to discriminate between 0s and 1s via a 784 -> 1 -> 2 feedforward network. I updated my question to include an image of the weights for the hidden neuron expecting it to look somewhat more obvious.. somewhat surprisingly, it really doesn't look like anything to me (the network is >99% accurate at picking between 0s and 1s). Neural networks are something else entirely it seems. Jan 11, 2017 at 0:37