I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:
- standard Backpropagation
- incremental (or stochastic) Backpropagation
- resilient Backpropagation (RPROP)
The network performs well with all of the algorithms above, using one hidden layer with 10 neurons and outputting just one number between 0 and 1 (~0.9 if the test image contains the face of the person the network was trained to recognize; ~0.1 if not) in the output layer.
After training the network, I then read the weights from the input layer to one neuron and encode them to a .png image (for each neuron in the hidden layer). Now, what I get with the RPROP algorithm is 10 (one for each neuron) 128x128 images, each representing some blurred human face, very similar to the face of the target person the network had to recognize. Nice! But what I get with standard and stochastic Backpropagation is just 10 images with random noise. I even tried initializing the weights to zero but all I got was, again, random noise. Can anyone tell me why is there such a difference between RPROP and the other algorithms, when it comes to interpreting the learned weights that go from the input to the hidden layer? I'm just curious.
The training set consisted of 457 .png pictures; each picture was 128x128 RGB pixels, so I encoded the pictures in 457 text files containing 128x128x3 integers in the range [0,255] (3 is because each RGB pixel is 3 colors: Red, Green, Blue). The input pixels for the ANN were scaled proportionally from [0,255] to [-1,1] (and the back to [0,255] when I encoded the hidden layer neurons to .png's).
The training set contains pictures of 6 different persons (but all in the same pose), approximately 70 for each one. The ANN has 128*128*3 = 49152 input units, one hidden layer with 10 neurons (but it was almost the same with 8 neurons) and only one output neuron (but my problem persisted even with 6 distinct output units). And yes, the network is fully connected; I'll try the sparse-connected option too, maybe with some more hidden layer.
With RPROP and standard Backpropagation I set the desired error to 0.006; usually, the number of training epochs was not more than 300 before the network could converge. With every algorithm, the perceptron unit was the sigmoid function and the learning rate was set to 0.7.
The weights were randomly initialised between -0.1 and 0.1; I only tried to monitor the weight updates when I initialized them to zero for incremental Backpropagation, but the images I obtained by encoding the hidden layer neurons to .png's were again just random noise.