# Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:

• standard Backpropagation
• incremental (or stochastic) Backpropagation
• resilient Backpropagation (RPROP)

The network performs well with all of the algorithms above, using one hidden layer with 10 neurons and outputting just one number between 0 and 1 (~0.9 if the test image contains the face of the person the network was trained to recognize; ~0.1 if not) in the output layer.

After training the network, I then read the weights from the input layer to one neuron and encode them to a .png image (for each neuron in the hidden layer). Now, what I get with the RPROP algorithm is 10 (one for each neuron) 128x128 images, each representing some blurred human face, very similar to the face of the target person the network had to recognize. Nice! But what I get with standard and stochastic Backpropagation is just 10 images with random noise. I even tried initializing the weights to zero but all I got was, again, random noise. Can anyone tell me why is there such a difference between RPROP and the other algorithms, when it comes to interpreting the learned weights that go from the input to the hidden layer? I'm just curious.

EDIT:

The training set consisted of 457 .png pictures; each picture was 128x128 RGB pixels, so I encoded the pictures in 457 text files containing 128x128x3 integers in the range [0,255] (3 is because each RGB pixel is 3 colors: Red, Green, Blue). The input pixels for the ANN were scaled proportionally from [0,255] to [-1,1] (and the back to [0,255] when I encoded the hidden layer neurons to .png's).

The training set contains pictures of 6 different persons (but all in the same pose), approximately 70 for each one. The ANN has 128*128*3 = 49152 input units, one hidden layer with 10 neurons (but it was almost the same with 8 neurons) and only one output neuron (but my problem persisted even with 6 distinct output units). And yes, the network is fully connected; I'll try the sparse-connected option too, maybe with some more hidden layer.

With RPROP and standard Backpropagation I set the desired error to 0.006; usually, the number of training epochs was not more than 300 before the network could converge. With every algorithm, the perceptron unit was the sigmoid function and the learning rate was set to 0.7.

The weights were randomly initialised between -0.1 and 0.1; I only tried to monitor the weight updates when I initialized them to zero for incremental Backpropagation, but the images I obtained by encoding the hidden layer neurons to .png's were again just random noise.

• I din't use the library you mentioned, but I surprized the success of the structure! What I see is you didn't use sparse connection but rather fully connected network with single hidden layer. How can that network learn features efficiently? As the input dimention is too high for the network structure. Could you provide more information about the training set, accuracy of each algorithm, etc? I don't expect any of the method could lead to succesfull result (or features). Could you monitor updates of weights? Or maybe the initialization is simply average of the training examples? – yasin.yazici Oct 5 '15 at 21:02
• @yasin.yazici Thank you for the reply. Well, I thought that one hidden layer can still be very expressive (I read it in "Machine Learning" by T. Mitchell, but maybe it's an old book). I edited my post with some information about the training set and the algorithms. – Giorgio Oct 7 '15 at 16:56
• Yes, even single hidden layer can be expressive if you supply enough data, but you have a small data set. I guess the low error rate you mention about is training error rate? That is not surprising as there is a little data and lots of parameters to learn. So the network prone to over-fitting. But I still didn't understand why learnt features are good at this condition. – yasin.yazici Oct 7 '15 at 17:21
• Maybe he could accomplish the task because the faces in the training data were quite different between each other: one was a woman with long hair, another had a huge beard, etcetera. So maybe he just learned to search for some darker spot in the image (e.g. a beard). But I'm just guessing. But actually I was just wondering why the hidden representations were so different between RPROP and the other two algorithms. – Giorgio Oct 7 '15 at 17:34
• Yeah, it can just find some simple distinctive features between classes and exploit it, but it doesn't generalize well on test time. RPROP may converge faster than the other two but I don't expect it to learn good features with the current settings. Do the features look like eigenfaces? – yasin.yazici Oct 7 '15 at 17:44