I've been implementing a Feed-forward Neural Network in C++ and CUDA. It is a basic Multi-layered Feed Forward ANN, using various activation functions (sigmoid bipolar, tanh, tanh scaled, and soft-sign).
The only training method I've implemented so far is back-propagation, using the Mean Squared Error. When trying it with a simple XOR network, it learns and the MSE is an accurate indicator.
However, when using it with other datasets (all taken from lib FANN) such as abelone or diabetes, the networks fail to learn. The MSE simply goes up and and down, with MSE values ranging between 1.1 and 5.0.
The way I've implemented MSE, is as dictated by Jeff Heaton's YouTube videos and in specific this video. In fact I used all his videos to implement that library.
The actual formula is:
$\frac{1}{n}\sum (i-a)^2$
Where i is the ideal output and a is the actual output, and n is the amount of samples (or patterns).
Algorithmically therefore, I simply calculate for each output node, then the difference between the actual output value and the ideal output value, square it (note for each output, and for each pattern) and then sum all squared differences. Finally, I divide the sum by the amount of patterns/samples. However, in order to scale it for networks with more than one output, my formula is essentially:
$\frac{1}{(n * o)}\sum_{o = nodes}^{n=samples} (ideal-actual)^2$
Where o is the network's output nodes.
I understand that back-propagation can get stuck in local minima, and that figuring out the learning rate $\epsilon$ and momentum $\alpha$ can be a black art.
An example of my current problem is shown in the plot below: the Diabetes data-set MSE from 90000 Epochs, $\epsilon = 0.2$ and $\alpha =0.9$. Activation function is tanh.
I do also understand that the network parameters greatly influence the networks, I've used the upper-bound rule for hidden neurons, and this post as a guide to initialize random weights.
My XOR toy-problem works fine, but any other network fails to learn.
In my todo list, I intend to implement the resilient back-propagation algorithm, but for now, I have four simple questions.
Please note, I am not very good with mathematics and formulas.
- Is this formula the correct way to calculate MSE?
- Why is my MSE fluctuating so much? Why is it going up and down to extreme values? If MSE = 0.25 means 25% error, then when I get 1.30 MSE does it mean that my network has %130 error (is that even possible?)
- Should I be investing more time in Resilient Back-Propagation and ignore Back-Prop?
- Should I be investing more time in other Error functions, such as Average Cross Entropy?
I would greatly appreciate any advice or help!