What are some ways to improve performance of a neural network binary classifier? I'm beginning to learn about neural networks, and I'm doing a beginner's project on classifying whether an email is spam or not spam. I have a few thousand data points, each has a few dozen features (frequency of letters, certain statistics, etc.)
I created a simple neural network with 1 fully connected layer and a sigmoid activation function at the end. It was able to achieve about a 91% accuracy.
I want to make it better, but I'm stuck. I tried to normalize my data and create deeper networks (with more fully connected layers), but these didn't increase my accuracy. 
I should note: When I added more fully connected layers, I was able to overfit my training data (with 99% accuracy) but the validation accuracy just hovers around 90%. When I added holdup layers, both training and validation accuracy reduced again to around 91%.
Since I'm a beginner, I think I just don't know all the things I should try. Can you suggest some ways to improve a neural network binary classifier?
 A: Normalizing your data may help for faster convergence. Take a look to this paper:
http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
In your case for faster convergence you should probably take these basic steps:


*

*normalize your data

*use hyperbolic tangent as activation function (and set the values of the outputs to [-1,1])

*be careful with correlation of your inputs
Anyway, these will only make that the training of your neural network is faster, they will not give you better results. From your description, you seem to have a problem with variance (overfitting). When you make your network more complex by adding more layers to it, you are addressing bias, not variance. You could try these basic steps for variance:


*

*use regularization

*try with a smaller subset of your features
When you fix your variance problems, then you can try to make your neural network more complex, in particular if you apply regularization. For doing that effectively, you will need to take care of convergence as well.
Another thing to take into account, although you don't mention it and maybe you're doing it already: since it is a classification problem, use cross-entropy for the error (not MSE).
Hope it is clear and it helps.
