22
$\begingroup$

I'm modeling 15000 tweets for sentiment prediction using a single layer LSTM with 128 hidden units using a word2vec-like representation with 80 dimensions. I get a descent accuracy (38% with random = 20%) after 1 epoch. More training makes the validation accuracy start declining as the training accuracy starts climbing - a clear sign of overfitting.

I'm therefore thinking of ways to do regularization. I'd prefer not to reduce the number of hidden units (128 seems a bit low already). I currently use dropout with a probability 50%, but this could perhaps be increased. The optimizer is Adam with the default parameters for Keras (http://keras.io/optimizers/#adam).

What are some effective ways of reducing overfitting for this model on my dataset?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ I'm having the exact same problem. How did you finally manage to regularize your LSTM? My LSTM's validation accuracy is 41%. My input shape is (200,) and I have 1 LSTM layer with 64 units, followed by 2 Dense layers with 0.4 dropout. $\endgroup$
    – mjsxbo
    Oct 30, 2018 at 9:55
  • $\begingroup$ So in my case I am trying to train LSTM autoencoder on (800, 97) data input. What do you recommend please? $\endgroup$
    – Avv
    Jul 9, 2021 at 20:50

1 Answer 1

Reset to default
13
$\begingroup$

You could try:

  • Reduce the number of hidden units, I know you said it already seems low, but given that the input layer only has 80 features, it actually can be that 128 is too much. A rule of thumb is to have the number of hidden units be in-between the number of input units (80) and output classes (5);
  • Alternatively, you could increase the dimension of the input representation space to be more than 80 (however this may overfit as well if the representation is already too narrow for any given word).

A good way to fit a network is too begin with an overfitting network and then reduce capacity (hidden units and embedding space) until it no longer overfits.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Did you try the l1 and l2 regularization? Does it actually work? This answer suggests that you shouldn't do this in general $\endgroup$
    – Jakub Bartczuk
    Nov 1, 2017 at 17:50
  • $\begingroup$ I wasn't aware of this property of RNNs, I will delete that point of the answer $\endgroup$
    – Miguel
    Nov 2, 2017 at 9:14
  • $\begingroup$ Hello, I was wondering how you got the rule of thumb that states "to have the number of hidden units be in-between the number of input units and output classes". Is there a paper that I can refer to ? $\endgroup$
    – Kong
    May 7, 2018 at 15:13
  • $\begingroup$ That's the thing about rules of thumb, I don't know where I got it from... $\endgroup$
    – Miguel
    May 7, 2018 at 15:33
  • 1
    $\begingroup$ @Kong , rule of thumb in the book Introduction to Neural Networks with Java by Jeff Heaton, where I will quote the exact words as " The number of hidden neurons should be between the size of the input layer and the size of the output layer. " Miguel probably used that as reference. I know its an old post, But I thought I should let you know. $\endgroup$
    – stitchesguy90
    May 28, 2021 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.