I'm beginning to learn about neural networks, and I'm doing a beginner's project on classifying whether an email is spam or not spam. I have a few thousand data points, each has a few dozen features (frequency of letters, certain statistics, etc.)
I created a simple neural network with 1 fully connected layer and a sigmoid activation function at the end. It was able to achieve about a 91% accuracy.
I want to make it better, but I'm stuck. I tried to normalize my data and create deeper networks (with more fully connected layers), but these didn't increase my accuracy.
I should note: When I added more fully connected layers, I was able to overfit my training data (with 99% accuracy) but the validation accuracy just hovers around 90%. When I added holdup layers, both training and validation accuracy reduced again to around 91%.
Since I'm a beginner, I think I just don't know all the things I should try. Can you suggest some ways to improve a neural network binary classifier?