1
$\begingroup$

I'm looking for something like the scikit-learn sample generators for generating small machine translation datasets that are quick to learn. My goal is to debug a Transformer model and have a quick feedback loop. Ie. I want to make a change, train for thirty seconds, infer for two seconds, look at the result and see what I can do next. With ‘debug’ I mean find mistakes in the code I've written, not determining why neurons are dying.

My current idea is to come up with two simple grammars that have a one-on-one mapping between them, generate examples from them and train the Transformer on these examples. But there should be people who have done this before and know what kinds of grammars, alphabets etc. are good.

$\endgroup$
1
$\begingroup$

I would not go for an artificial dataset because they are too easy for a network to learn and you might not discover some bugs. It happened to me that I had a code that was able to do trivial tasks like capitalization but failed to scale to more complex tasks because of a bug in the model architecture.

An easy sequence-to-sequence task is transliteration. Google recently released a dataset for transliteration between Arabic and English.

A good small machine translation dataset is Multi30k. It contains of 30k image description which are pretty simple sentences (such as: "A man is running in a park."). The original language is English, they were translated into German, French, and Czech (although Czech is pretty low quality).

$\endgroup$
1
  • $\begingroup$ Transliteration is a good idea. I hadn't thought of it. By the way, I don't mind if I overlook some bugs. I just wanted to make quick progress on a particular bug that I was hunting. $\endgroup$ Jan 19 '20 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.