I'm training a Binary classifier for pattern recognition problem. I have a feedforward neural network of 4 inputs, one hidden layer and I have many examples. I use cross-entropy as cost function. The data isn't linearly separable.
Only for visualization of the data, the next images is the result of PCA, holding the 90% of the variance.
I use the four features in the network. The network performs well for an arbitrary number of units in the hidden layer (~7% of error), but when I test the network for different number of hidden units (between 1 and 3000) , the cost in the train and validations sets are constant, like in the next figure.
The same happens if I increase the size of the examples set, plotting the learning curve, putting a constant number of hidden layer units.
But, if I variate the regularization parameter, putting a constant number of hidden layer units and using all the examples set, I can get an optimum value for the regularization parameter that minimizes the CV cost, like in the image.
I don't know what is happening or what to do. I think that perhaps the neural network needs a lot of more data or a lot of more complexity to fit the data. I'm using matlab for the network.
This is my learning curve:
The dataset is $\#train + \# cv + \# test$.