SLP vs. MLP: Is my data linearly separable?

Question

I implemented an artificial neural network using scikit neuralnetwork. As default configuration for my classification task I am using

10730 Datsets x 115 Features
1 Hidden Layer with 61 neurons
7 Outputs,

finding the minimum (~ 0.72) after 16-17 iterations

Now I tried to optimize the architecture in order to lower the mean categorical cross-entropy. At first I iterated the number of hidden neurons from 1 - 150, expecting the test-error to decrease until finding a minimum and then raising again.

But actually the error stayed constantly on the same level, oscillating around ~ 0.75

(Note: For this plot I used the best/lowest error from all iterations for each of the 150 architectures. For another plot I used the mean error of all 50 iterations, which gave me the expected output (error decreases -> reaches minimum -> error rises), but I think it's the wrong calculation methodology, isn't it? )

So no matter how many hidden units I used, the error stayed on the same level. So I supposed, that the data is linearly separable and I don't need a Mult-Layer Perceptron.

I constructed a SLP, but here the NN doesn't learn at all:

I am confused whether the data is linearly separable or not. What am I missing? Why does the error not rise with additional hidden units, if it's not linearly separable?

Any help is welcome!

Answer 1

Just to jump from the one plot you have to the fact that the data is linearly separable is a bit quick and in this case even your MLP should find the global optima. The easiest way to check this, by the way, might be an LDA.

It is not unheard of that neural networks behave like this. One option here is that you are stuck in a badly generalizing local minima and you just move deeper into it. Dropout, a larger learning rate and smaller batch sizes might hhelp in this case. Another, more likely one, is that your regularization is not properly tuned, hurting generalization performance. How do you do regularization on the net?

I think what also might be happening here is that the test-set can simply not obtain a better performance. This happens in case some output units "give up" and decide rather not to model anything but reduce their weight magnitude to suite regularization. You can figure that out by taking a look at the distribution of the predictions.

Last but not least: It could simply be a misfit of the loss function. Try something easier, the squared loss for example.