How to improve the binary classification model for text (News Articles) of Recurrent Neural Net with word emmbeding?

Question

I am trying to do binary classification of news articles using Recurrent Neural Net with word embedding. Following are the parameters of the model:

Data:
    8000 labelled news articles (Sports:Non-sports::15:85)

Parameters:
    embedding size = 128
    vocabulary size = 100000
    No. of LSTM cell in each layer = 128
    No. of hidden layers = 2
    batch size = 16
    epochs = 10000

Result:
    AUC on training set = 0.60
    AUC on testing set = 0.55

As the both training and testing error is high model is underfitting and require more data. So I have couple of doubts here:

1. What would be the optimum data size required?
2. Can we change the parameters to improve AUC. By decreasing, embedding size or No. of neurons we can minimize degree of freedom.

Answer 1

1. What would be the optimum data size required?

The more the better. From my experience in text classification, 8000 samples should yield good results in most cases. AUC on testing set = 0.55 means your classifier is almost random: I would expect better.

See How to get the data set size required for neural network training? or How few training examples is too few when training a neural network?.

2. Can we change the parameters to improve AUC. By decreasing, embedding size or No. of neurons we can minimize degree of freedom.

You have to experiment. Your hyperparameters look reasonable to me, but you may want to try CNN as CNN tend to outperform RNN for text classification when texts are not short (> 100 words).

I did some experiments on (sequential) short-text classification:

• Franck Dernoncourt, Ji Young Lee Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.
• Ji Young Lee, Franck Dernoncourt Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks NAACL 2016.

FYI:

• Neural architectures: data-informed automatic design
• What are the state-of-the-art methods to determine parameters in CNN, NN, RNN, or any deep learning models
• Guideline to select the hyperparameters in Deep Learning