I'm currently reading the paper for the attention model (Neural Machine Translation By Jointly Learning to Align and Translate (Bahdanau et al., 2015)) and had a question regarding a statement made in the paper. I'm sure this concept also applies to factors outside of deep learning and machine translation, but I just wanted to provide some context as to what prompted me to ask this question.

In the second paragraph of section 2, the authors state that "Recently, a number of papers have proposed the use of neural networks to directly learn this conditional distribution." What they're referring to is that the concept of machine translation is basically to maximize the conditional probability

$$P(\mathbf{y}\ |\ \mathbf{x})$$

where $\mathbf{y}$ is the target sentence and $\mathbf{x}$ is the source sentence. My assumption is that they're basically saying that the authors used neural networks to "learn" the features automatically rather than having to do so manually as was the case for previous translation models.

My question is, what does it exactly mean to "directly learn" a distribution?


Gone through the paper and their methodology, they are using a technique called 'word embeddings' and 'word2vec'. For more detail, here is an example below:

Suppose we have a sentence at the encoder like:

I will have an apple juice.

For this sentence, each word will be associated with index id in the dictionary. Encoder network will learn the importance of each vector by creating a table with all the input words. On the base of the importance, it will now learn the to predict the input sentences. Suppose, another sentence came like

I will have an orange ______.

Now, network will compare with the table and find that orange and apple have the same importance. So it will automatically fill the blanks with the 'juice' word.

This is what they are saying according to conditional probability, that if already they have an input vector x and y is the predicted output sentence. Then P(y|x), probability of a output sentence 'y' will be calculated directly looking upon 'x' which is previously enter in the network.

Furthermore, they are using Trigrams technique which is predicting the ith word in a sentence with the help of its neighbor words (i.e. i-1 and i+1). This is a very useful technique in machine translation applications where the input source data is full of grammatical mistakes. This can learn the words through the neighborhood and then provide the context of the source sentence.

You can go through this link for further details on this:


  • $\begingroup$ you can upvote the answer if you find it helpful @Seankala $\endgroup$ Aug 28 '19 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.