I would like to classify a mail (spam = 1/ham = 0), using logistic regression. My implementation is similar to this implementation and using tensorflow.
A mail is represented as a bag-of-words vector, with each number in the vector representing how often a term appeared in a mail. The idea is to multiply that with a vector, and use the sign-function to turn regression into classification. $$y_{predicted} = \sigma(x_i^T\theta) $$, with $\sigma = \frac{1}{1 + e^{-x}}$. To calculate the loss, I am using the l2-loss (squared loss). Since I have a lot of trainig data, regularization seems not necessary (training and testing accuracy is always very close). Still I only get a max accuracy of about 90% (both training and testing). How can I improve this?
I already tried the following:
Use regularization, L1, L2 with different strength (seems not necessary)
Use different learning rates
Use gradient descent, stochastic gradient descent and batch gradient descent (the hope is to avoid local minima in the loss-function, by introducing more variance with stochastic/batch gradient descent)
create more training data (classes were disbalanced 80/20 spam/ham), using SMOTE
Things that I could still try:
- use a different loss function
Any other suggestions?