This question already has an answer here:

I am using Keras RNN Cell to perform parts of speech tagging. The architecture is as follows(I cannot put the code because of privacy reasons) :

  1. An embedding layer of of 40 units of shape (batch_Size, max_sentence_length, 40)
  2. tf.keras.layers.SimpleRNNCell(state_size=number_of_tags_in_dataset+20, dropout=0.2,recurrent_dropout=0.0, activation='tanh')
  3. tf.contrib.layers.fully_connected(units=state_size)
  4. tf.contrib.layers.dropout(keep_prob=0.6)
  5. tf.contrib.layers.fully_connected(units=number_of_tags)
  6. tf.keras.layers.BatchNormalization()
  7. tf.keras.activations.relu()
  8. Using tf.contrib.seq2seq.sequence_loss() with AdamOptimizer and gradient clipping of 0.5
  9. batch_size=32, learn_rate=0.01

The result for ~14 epochs are as follows (due to resource constraint, thats the maximum number of epochs I can run for)

accuracy 0.0172309
accuracy 0.800888
accuracy 0.866243
accuracy 0.893743
accuracy 0.896006
accuracy 0.901575
accuracy 0.899487
accuracy 0.898529
accuracy 0.900531
accuracy 0.902532
accuracy 0.899051
accuracy 0.903055
accuracy 0.901053
accuracy 0.898703
accuracy 0.898703

I have noticed this trend where no matter what hyperparameter changes I make, the accuracy is getting stuck around 89-90%. Can you provide some suggestions to boost the accuracy? I am fairly new to Deep Learning so I have been struggling a lot to optimize my model. I have also tried building Bidirectional LSTMs for the same but they are too slow for the resource constraint and the maximum accuracy that I can achieve there is around 92%.


marked as duplicate by Stephan Kolassa, mdewey, Ferdi, Siong Thye Goh, Carl May 4 at 14:52

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ Why do you say stuck? It looks like it is converging to 90%. Why would you expect to be able to get higher accuracy than that? $\endgroup$ – Michael Chernick Apr 28 at 23:35
  • $\begingroup$ I say so because I have noticed some of my peers getting an accuracy of around 95% on the same dataset, with a similar architecture. So, I feel there are some shortcomings with my model or hyperparameters. I am fairly new to Deep Learning and I have been struggling a lot to optimize my model. I have also tried building Bidirectional LSTMs for the same but they are too slow for the resource constraint. $\endgroup$ – Reshu Bisht Apr 28 at 23:41
  • $\begingroup$ 1. don't use batch norm before or after the last layer 2. do you have a good reason to use relu as an output activation? $\endgroup$ – shimao Apr 29 at 0:17
  • $\begingroup$ At or around which layer can I benefit from batch norm? All my layers have Tanh activation expect the last one. Putting tanh in place of relu at the end, was reducing the accuracy. $\endgroup$ – Reshu Bisht Apr 29 at 0:21
  • $\begingroup$ removing batch norm from the very end and trying to place it somewhere before the first fully connected layer, etc. is reducing the accuracy. Also, relu is performing the best(have tried softmax, tanh,etc.). Any other ideas ? $\endgroup$ – Reshu Bisht Apr 29 at 3:36