Trouble training Neural Network I'm trying to use Encog to define an artificial neural network in order to process this dataset (6 inputs, 2 yes/no outputs), but I can't get any lower than ~65% error. The NN is feedforward with backprop and sigmoid for activation.
My steps were:


*

*Normalize the dataset: temperatures from 35-42 to [-1,1], no to -1, yes to 1. Pick 80/120 random entries to use for training (the rest was supposed to be used for cross-validation - pun intended - later).

*Use a NN with 6 input neurons, 2 outputs, bias, and backpropagation.

*Start playing with learning rate, momentum and hidden layers/neurons count. The "best" result was achieved with 6 hidden neurons, 0.1 learning rate and 0.7 momentum. Tried all the common heuristics (half the input neurons, somewhere between the input and output count, using a large number of hidden units to facilitate the finding of a minimum, even if it raises generalization error, etc). Also, tried combinations of high/low learning-rate/momentum, but with no success.
Funny enough, I did find a paper where the author seems to have solved this problem rather well, but is splitting the dataset by the 2 outputs a valid choice? I mean, the author used Levenberg Marquardt, which can only be used for 1 output neuron, so how is it supposed to encode the result in this way? The only way I can think of is training the NN to recognize output disease 1, and then do the same for another NN for output disease 2.
I hope this was not too confusing, but I'm just getting started in this field of study.
 A: I think that's the right thing to do - splitting the problem into two separate networks, each contrating on just one output.
Furthermore, you should consider switching the activation from sigmoid to tanh, for tanh can also have negative values (-1 .. 1), while sigmoid values are limited to 0 .. 1 values.
Another thing, but I'm not sure if it'll help - try changing the outputs from either -1 or 1 to a milder -0.8 - 0.8.
A: It sounds like a good idea to split it into two networks. There is nothing wrong with doing that and it's exactly what the author did in the paper you linked to. I wouldn't trust everything that paper says about neural networks though. Figures 4 & 6 make me feel like the guy doesn't know what he's talking about.
A: If you encode yes/no with 1/-1, you should use tanh rather than sigmoid.
A: Although parameters of the learning are important, tuning them usually cannot warrant significant improvement on the test set error. I suggest increasing the capacity of the network - number of hidden neurons and/or a number of layers. This will lead to increase of the number of parameters so you have to be careful not to overfit. I am not familiar with this particular dataset but if its size is small you can augment it by applying artificial transformations (think flipping an image or something similar).
