I am working on a sequence to sequence chatbot model based on the Tensorflow NMT tutorial for a project. I have a database of about 15 million replies and around 3 million individual words. It is an encoder-decoder model with 2 recurrent layers and the Bahdanau attention mechanism (the layers of the encoder are bidirectional). I built this model in order to be able to change all the parameters quite easily, this being only what I have tried so far. The problem is that after 30 hours of training the model has not learned anything. The loss was initially decreasing, but after the first epoch, nothing significant happened.

I have no idea whether the problem lies in my model architecture or in the lack of time and computing power (I know that these models are extremely hard to train).

When testing with a Vocabulary of 1000 - 5000 words, the maximum sequence length of 10-15 and 1000 replies as the training database the model eventually converges to a global minimum. This makes me believe that there is nothing fundamentally wrong with de model's architecture, but I am still not sure.

I am using Google Bigquery to store the training database. I know that giving other people access to it is against best practices but I have backed everything up. The project is on this Github repository and it contains a Colab notebook for testing.

As a student, I do not have a lot of money to spend on cloud computing therefore I try to use Google Colab research but there are some obvious limitations.

Does anyone have any ideas on what might be wrong or know any relatively cheap cloud computing platforms? I really want to get this project done in order to participate in a contest but right now I am stuck... Is it even feasible to train such a network in less than a week?

  • $\begingroup$ It seems to me hat you either some very unreasonable hyperparameters or parameters of the optimizer. What are you hyperparamters? With 15M examples, a week is a reasonable estimate. $\endgroup$ – Jindřich Jan 21 '20 at 9:22
  • $\begingroup$ If time is your concern, you might also consider Transformer-based architectures. At training time, GPUs can parallelize more than when using RNNs and the training is faster. $\endgroup$ – Jindřich Jan 21 '20 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.