I am using Keras RNN Cell to perform parts of speech tagging. The architecture is as follows(I cannot put the code because of privacy reasons) :
- An embedding layer of of 40 units of shape (batch_Size, max_sentence_length, 40)
- tf.keras.layers.SimpleRNNCell(state_size=number_of_tags_in_dataset+20, dropout=0.2,recurrent_dropout=0.0, activation='tanh')
- Using tf.contrib.seq2seq.sequence_loss() with AdamOptimizer and gradient clipping of 0.5
- batch_size=32, learn_rate=0.01
The result for ~14 epochs are as follows (due to resource constraint, thats the maximum number of epochs I can run for)
I have noticed this trend where no matter what hyperparameter changes I make, the accuracy is getting stuck around 89-90%. Can you provide some suggestions to boost the accuracy? I am fairly new to Deep Learning so I have been struggling a lot to optimize my model. I have also tried building Bidirectional LSTMs for the same but they are too slow for the resource constraint and the maximum accuracy that I can achieve there is around 92%.