I implemented an artificial neural network using scikit neuralnetwork. As default configuration for my classification task I am using

10730 Datsets x 115 Features
1 Hidden Layer with 61 neurons
7 Outputs,

finding the minimum (~ 0.72) after 16-17 iterations

Now I tried to optimize the architecture in order to lower the mean categorical cross-entropy. At first I iterated the number of hidden neurons from 1 - 150, expecting the test-error to decrease until finding a minimum and then raising again.

But actually the error stayed constantly on the same level, oscillating around ~ 0.75 the best error for each architecture (1-150 hidden units) with 50 iterations each

(Note: For this plot I used the best/lowest error from all iterations for each of the 150 architectures. For another plot I used the mean error of all 50 iterations, which gave me the expected output (error decreases -> reaches minimum -> error rises), but I think it's the wrong calculation methodology, isn't it? )

So no matter how many hidden units I used, the error stayed on the same level. So I supposed, that the data is linearly separable and I don't need a Mult-Layer Perceptron.

I constructed a SLP, but here the NN doesn't learn at all: training- and validation-error for SLP, 200 iterations

I am confused whether the data is linearly separable or not. What am I missing? Why does the error not rise with additional hidden units, if it's not linearly separable?

Any help is welcome!


1 Answer 1


Just to jump from the one plot you have to the fact that the data is linearly separable is a bit quick and in this case even your MLP should find the global optima. The easiest way to check this, by the way, might be an LDA.

It is not unheard of that neural networks behave like this. One option here is that you are stuck in a badly generalizing local minima and you just move deeper into it. Dropout, a larger learning rate and smaller batch sizes might hhelp in this case. Another, more likely one, is that your regularization is not properly tuned, hurting generalization performance. How do you do regularization on the net?

I think what also might be happening here is that the test-set can simply not obtain a better performance. This happens in case some output units "give up" and decide rather not to model anything but reduce their weight magnitude to suite regularization. You can figure that out by taking a look at the distribution of the predictions.

Last but not least: It could simply be a misfit of the loss function. Try something easier, the squared loss for example.

  • $\begingroup$ Hey, thank you very much for your answer and sorry for my late feedback! This was not the only plot I had, I tested several parameters such as learningrate, batchsize (also online learning), weight decay and others, but the curve always had the same form. For regularization I use L2, I also did feature selection and a grid search for the best parameters, but the error stayed the same. But I think you might be right about the data-set: It's very unbalanced and the quality is not the best, so it seems it simply cannot generalize well. SVM and DecisionTress showed the same results. Thanks! $\endgroup$
    – Jonas M.
    Commented Feb 11, 2016 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.