I am having some issues with using neural network. I am using a non linear activation function for the hidden layer and a linear function for the output layer. Adding more neurons in the hidden layer should have increased the capability of the NN and made it fit to the training data more/have less error on training data.

However, I am seeing a different phenomena. Adding more neurons is decreasing the accuracy of the neural network even on the training set. enter image description here

Here is the graph of the mean absolute error with increasing number of neurons. The accuracy on the training data is decreasing. What could be the cause of this?

Is it that the nntool that I am using of matlab splits the data randomly into training,test and validation set for checking generalization instead of using cross validation.

Also I could see lots of -ve output values adding neurons while my targets are supposed to be positives. Could it be another issues?

I am not able to explain the behavior of NN here. Any suggestions?

Here is my code

targets = dayofyear_targets(:, i+1);
net = newfit(train_data', targets', 4);
net = init(net);
net.performFcn = 'mae';
net.layers{2}.transferFcn = 'purelin';
net.trainParam.max_fail = 10;
net.layers{1}.transferFcn = 'tansig';
net = train(net, train_data', targets');
results = sim(net, train_data')';
diff = abs(results-targets);
mae = sum(diff(:))/ num_samples
  • $\begingroup$ Are you sure that this plot represents training set errors, but not validation or test set? Check that point $\endgroup$ – O_Devinyak Sep 4 '13 at 17:54
  • $\begingroup$ @O_Devinyak. Yes I am sure it's training set errors $\endgroup$ – user34790 Sep 4 '13 at 18:03
  • $\begingroup$ Than I have no glue besides possible errors in code. Maybe your neural network is performing standardization, so you are comparing standardized output with non-standardized actual values? $\endgroup$ – O_Devinyak Sep 4 '13 at 18:27
  • $\begingroup$ @O_Devinyak. I have added my code. Yeah, the NN tool is performing minmax normalization to make the output in the range [-1 1]. However, I am using nn's sim function to predict and compare it with the actual values. So I guess it should have been fine $\endgroup$ – user34790 Sep 4 '13 at 18:34
  • $\begingroup$ Hm. I don't see any flaws (besides redundant quotation marks - are they placed intentionally?). Maybe it is worth to try the same with default activation functions. $\endgroup$ – O_Devinyak Sep 4 '13 at 19:04

The axis of that graph is in 10 to the power 7, thats kind of high. I think you might want to check what learning rate you are using, and reduce it. I think the network is probably diverging rather than converging.


I'm not sure about your code,

But do you keep the same number of epoch ? If it's the case, it's totally normal. The more neurons you have, the more time it need to learn. It can be that your problem is to large for the Gradient descent to make its job.


I recommend training the simplest network first to confirm that you can find a solution that converges. As Hugh Perkins suggested, reducing learning rate might help here. Once such solution is in the bag you can experiment by adding more layers. Your graph shows that optimization is not converging which might signal a variety of issues ranging from incorrect data or improper optimization procedure.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.