I have been working in a text generator with LSTM cells inspired by this code http://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/ (I am adding words one-hot encoded). FYI I am just trying to learn with it :D
I want to run it in the cloud, but first I want to make sure I am doing the things right. The problem is that I have a fast decay in the loss function in the early steps of the optimization, and after, the gradient seems to vanish (very slow decrease of my loss function). You can check my code here https://github.com/Pasquinell/MLM-KERAS/blob/master/text_gen_lst_nltk.ipynb. In this particular case, I just ran a few steps because I was hurrying, but I have run 20 epochs other days... achieving awful results (loss: 5.2095 - acc: 0.0720). My question is... do you think I am just being impatient or I have other problem that I am not seeing (e.g.: bad feature engineering).
Thank you for your help!!!