I'm beginning to learn about neural networks, and I'm doing a beginner's project on classifying whether an email is spam or not spam. I have a few thousand data points, each has a few dozen features (frequency of letters, certain statistics, etc.)

I created a simple neural network with 1 fully connected layer and a sigmoid activation function at the end. It was able to achieve about a 91% accuracy.

I want to make it better, but I'm stuck. I tried to normalize my data and create deeper networks (with more fully connected layers), but these didn't increase my accuracy.

I should note: When I added more fully connected layers, I was able to overfit my training data (with 99% accuracy) but the validation accuracy just hovers around 90%. When I added holdup layers, both training and validation accuracy reduced again to around 91%.

Since I'm a beginner, I think I just don't know all the things I should try. Can you suggest some ways to improve a neural network binary classifier?


1 Answer 1


Normalizing your data may help for faster convergence. Take a look to this paper: http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf

In your case for faster convergence you should probably take these basic steps:

  • normalize your data

  • use hyperbolic tangent as activation function (and set the values of the outputs to [-1,1])

  • be careful with correlation of your inputs

Anyway, these will only make that the training of your neural network is faster, they will not give you better results. From your description, you seem to have a problem with variance (overfitting). When you make your network more complex by adding more layers to it, you are addressing bias, not variance. You could try these basic steps for variance:

  • use regularization

  • try with a smaller subset of your features

When you fix your variance problems, then you can try to make your neural network more complex, in particular if you apply regularization. For doing that effectively, you will need to take care of convergence as well.

Another thing to take into account, although you don't mention it and maybe you're doing it already: since it is a classification problem, use cross-entropy for the error (not MSE).

Hope it is clear and it helps.

  • $\begingroup$ Thank you, this is really helpful. Can you explain (or point me to something) why correlated features are bad? how is that related to variance? and why does adding more layers only address bias? $\endgroup$
    – foobar
    Commented Jan 28, 2017 at 15:25
  • $\begingroup$ You're more than welcome. If the features are correlated, this is bad for convergence of the neural network and more in general to stability of the model (i.e. for any model, e.g. in regression contexts you could look e.g. at "Introduction to Statistical Learning", But it does not really related so much to the variance problem, not necessarily. I edit the post since I understand it was confusing. $\endgroup$
    – lrnzcig
    Commented Jan 28, 2017 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.