I'm currently working on implementing the Transformer model for machine translation. I'm taking a look at the data that was used in the actual paper and also used by many other implementations available publicly. The dataset is an English-German dataset (specifically the IWSLT 16 En-De dataset) and it's apparently supposed to be a parallel corpus.

What I don't understand is that doesn't "parallel" in this context mean that for every English sentence there is one German sentence? The dataset contains 233,213 English sentences and 196884 German sentences. Doesn't this mean for some German sentences there will be more than one corresponding English translation?

I'm asking this question because I'm trying to preprocess the data and create batches to feed into a neural network model. The problem is that right now the code I've written simply slices the English dataset up to the length of the German dataset (so that they have the same number of sentences), but my concern is that the data won't actually be parallel.

  • $\begingroup$ This might be a question for the German Language site, but in translating German scholarly writing to English, I believe it is often natural to use two English sentences to translate one long German sentence. A ratio of about 1.2 English sentences to 1 German sentence does not seem out of line. $\endgroup$ – BruceET Aug 5 '19 at 6:54
  • $\begingroup$ @BruceET Hello, thanks for the comment. Don't take this the wrong way, but are you serious? I've actually had no idea that any two languages worked that way, that's really interesting and cool and would make sense for my specific case. $\endgroup$ – Seankala Aug 5 '19 at 7:21
  • 1
    $\begingroup$ Well, just as an example, a good copy editor might have suggested splitting the first English sentence of my previous comment into two. "This ... site. However, my opinion is ... sentence." // I have seen German scholarly articles with sentences running as long as half a page. $\endgroup$ – BruceET Aug 5 '19 at 7:29
  • $\begingroup$ @BruceET Ah that makes a lot of sense. If that indeed is the case for my specific example, I'm not so sure if it could be called a "parallel" corpus, though. It seems like that would be a completely different dataset. $\endgroup$ – Seankala Aug 5 '19 at 7:33
  • $\begingroup$ Perhaps see this Q&A from German Stack Exchange. // If you need some help with parts written in German, you can translate the whole page into English. $\endgroup$ – BruceET Aug 5 '19 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.