13
$\begingroup$

I'm modeling 15000 tweets for sentiment prediction using a single layer LSTM with 128 hidden units using a word2vec-like representation with 80 dimensions. I get a descent accuracy (38% with random = 20%) after 1 epoch. More training makes the validation accuracy start declining as the training accuracy starts climbing - a clear sign of overfitting.

I'm therefore thinking of ways to do regularization. I'd prefer not to reduce the number of hidden units (128 seems a bit low already). I currently use dropout with a probability 50%, but this could perhaps be increased. The optimizer is Adam with the default parameters for Keras (http://keras.io/optimizers/#adam).

What are some effective ways of reducing overfitting for this model on my dataset?

$\endgroup$
  • $\begingroup$ I'm having the exact same problem. How did you finally manage to regularize your LSTM? My LSTM's validation accuracy is 41%. My input shape is (200,) and I have 1 LSTM layer with 64 units, followed by 2 Dense layers with 0.4 dropout. $\endgroup$ – Nirvan Anjirbag Oct 30 '18 at 9:55
6
$\begingroup$

You could try:

  • Reduce the number of hidden units, I know you said it already seems low, but given that the input layer only has 80 features, it actually can be that 128 is too much. A rule of thumb is to have the number of hidden units be in-between the number of input units (80) and output classes (5);
  • Alternatively, you could increase the dimension of the input representation space to be more than 80 (however this may overfit as well if the representation is already too narrow for any given word).

A good way to fit a network is too begin with an overfitting network and then reduce capacity (hidden units and embedding space) until it no longer overfits.

$\endgroup$
  • 1
    $\begingroup$ Did you try the l1 and l2 regularization? Does it actually work? This answer suggests that you shouldn't do this in general $\endgroup$ – Jakub Bartczuk Nov 1 '17 at 17:50
  • $\begingroup$ I wasn't aware of this property of RNNs, I will delete that point of the answer $\endgroup$ – Miguel Nov 2 '17 at 9:14
  • $\begingroup$ Hello, I was wondering how you got the rule of thumb that states "to have the number of hidden units be in-between the number of input units and output classes". Is there a paper that I can refer to ? $\endgroup$ – Kong May 7 '18 at 15:13
  • $\begingroup$ That's the thing about rules of thumb, I don't know where I got it from... $\endgroup$ – Miguel May 7 '18 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.