Problem training neural network for binary classification

Question

I'm training a Binary classifier for pattern recognition problem. I have a feedforward neural network of 4 inputs, one hidden layer and I have many examples. I use cross-entropy as cost function. The data isn't linearly separable.

Only for visualization of the data, the next images is the result of PCA, holding the 90% of the variance.

I use the four features in the network. The network performs well for an arbitrary number of units in the hidden layer (~7% of error), but when I test the network for different number of hidden units (between 1 and 3000) , the cost in the train and validations sets are constant, like in the next figure.

The same happens if I increase the size of the examples set, plotting the learning curve, putting a constant number of hidden layer units.

But, if I variate the regularization parameter, putting a constant number of hidden layer units and using all the examples set, I can get an optimum value for the regularization parameter that minimizes the CV cost, like in the image.

I don't know what is happening or what to do. I think that perhaps the neural network needs a lot of more data or a lot of more complexity to fit the data. I'm using matlab for the network.

EDIT:

This is my learning curve:

The dataset is $\#train + \# cv + \# test$.

Answer 1

There seems to be a big gap between the error on the training-set and the error on the validation-set. This points to overfitting. That would also explain the increase in error on the validation set when you increase the number of nodes in the hidden layer after a certain point. After that point, you could just be making the overfitting worse. My advice would be:

1. try some regularization techniques to see if that helps with the overfitting.
2. Find more data if possible. The real pattern might be complex and very difficult to find with your current number of training-cases.
3. Add more features that capture new aspects of the data, to make the pattern separating the cases more obvious.
4. Accept that the real pattern is difficult to find and take the optimal number of hidden-units as the best you can do. There might be some randomness involved that is impossible to capture.