Questions tagged [machine-translation]
Machine translation (MT) is a sub-field of computational linguistics that investigates the use of software to translate text or speech from one language to another.
46 questions
3
votes
1
answer
2k
views
How does inference work on a Transformer?
(Let's say I trained a transformer for translation.)
In the training, the output sentences are given and fed into the decoder as a whole. However, with inference, only a start-of-sentence (SOS) token ...
1
vote
1
answer
489
views
Why are K and V the same in the second attention layer of a Transformer's decoder?
(For this example, let's say we're using a Transformer to translate from English to French.)
For the decoder in a Transformer, the second attention layer takes K and V from the encoder and then takes ...
0
votes
2
answers
204
views
Which metric to use for language translation?
So I am using a pre-trained model to do the language translation
Eg:
Input = "Good morning"
Output = "Bonjour"
I would like to see if the ...
- 63
0
votes
1
answer
700
views
Number of weights to be learnt in the encoder decoder attention in the transformer model
I have a doubt about the number of weights to be learned in the encoder-decoder attention layer in the transformer model (attention is all you need). Some blogs articles say the $K$ and $V$ (key and ...
- 55
2
votes
1
answer
59
views
In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?
So attention and transformer models can be used for machine translation.
Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
- 395
3
votes
1
answer
1k
views
Why is Bahdanau's attention sometimes called concat attention?
I am learning the intuition behind the attention mechanism from
https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
https://lilianweng.github....
- 165
8
votes
1
answer
573
views
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states:
Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
- 1,347
3
votes
1
answer
897
views
Teacher Forcing in RNNs
I'm reading about teacher forcing for neural translation applications here and here , but I am a little confused on the method. Why does teacher forcing speed up training? Also why in the Kaggle link ...
- 211
1
vote
1
answer
37
views
Neural Text Simplification with no appropriate dataset
I'm currently starting a research project focused on NLP.
One of the steps involved in this project will be the development of a text simplification system, probably using a neural encoder-decoder ...
- 11
3
votes
1
answer
45
views
How does recall help precision overcome "length-related problems?"
I'm reading the paper on the Bilingual Evaluation Understudy (BLEU) metric BLEU: A Method for Automatic Evaluation of Machine Translation (Papineni et al., 2002) and had a question regarding a quote ...
- 4,177
0
votes
1
answer
160
views
Is statistical machine translation similar in many ways to Hidden Markov Model? How can we justify it?
Came across this question but didn't know the correct answer, if anyone could clear this out would be of great help.
'If we say we Statistical machine translation is similar in many ways to Hidden ...
1
vote
1
answer
59
views
Why can't we calculate p(e|f) directly from corpus?
In SMT,
a document is translated according to the probability distribution $p(e|f)$ that a string $e$ in the target language (for example, English) is the translation of a string $f$ in the source ...
- 113
1
vote
1
answer
30
views
Seq2Seq Machine Translation Question
I'm reading through Pytorch's NLP from Scratch: Translation with a Sequence to Sequence Network and Attention, and I am a bit confused on the Preparing Training Data section, particularly:
...
- 211
1
vote
1
answer
254
views
Why do we feed the target sentence to the model during transformers training?
Going through the famous notebook: Annotated transformers to be found here, I noticed that during the training, the model is fed the target sentence.
If the model is fed the target sentence from the ...
- 111
2
votes
1
answer
113
views
What feature space is used in transformer networks for machine translation?
Title is the question.
The papers I've read, e.g. "Attention is All You Need" fail to specify exactly what word embeddings are used in these machine translation networks. In most cases ...
- 3,269
0
votes
0
answers
400
views
Training and testing transformer model from scratch
As you know, transformers are one of the strongest model in the field of NLP and machine translation.
I know there are many resources, but I still could not find a good tutorial teaching how to use ...
- 395
0
votes
1
answer
69
views
What metric should I use for evaluating a reading order of text tokens given the correct ordering?
Given an ordering of tokens extracted from a document with a ground truth ordering available.
What would be the correct way to evaluate the ordering?
I took a look at some Machine Translation ...
1
vote
1
answer
490
views
Attention Mechanisms and Alignment Models in Machine Translation
From the paper that introduced attention mechanisms (Bahdanau et al 2014: Neural Machine Translation by Jointly Learning to Align and Translate), it seems that the translating part is the regular RNN/...
- 1,046
1
vote
1
answer
880
views
How to compute classification accuracy for machine translation?
I looked online and am unable to find the proper classification metric for machine translation problems.
Let's the predicted output is: "My name John" and the ground truth is "My name is John"
I ...
- 1,932
1
vote
1
answer
1k
views
How does Transformer use BPE?
I'm trying to re-implement Transformer on my own but I'm struggling on the input encoding.
It seems that the original paper(Attention is All You Need) mentioned that they use Byte-pair Encoding to ...
- 135
0
votes
1
answer
100
views
Is there any way to train a multi-label machine translation model?
Generally, machine translation is translating a sentence from an original language to a target language. However, for a specific origin sentence, the target sentence is not unique. Now, I have ...
- 209
1
vote
1
answer
1k
views
Matrices K and V in the decoder part in the transformer model?
There is something I do not get in the illustrated transformer article from Jay Alamar (http://jalammar.github.io/illustrated-transformer/). In the decoder side paragraph, he said
The encoder start ...
- 55
0
votes
0
answers
34
views
Sequence to Sequence model not training
I am working on a sequence to sequence chatbot model based on the Tensorflow NMT tutorial for a project. I have a database of about 15 million replies and around 3 million individual words. It is an ...
- 1
2
votes
1
answer
68
views
Machine Translation: with sufficient parallel data, can we improve even further the performance of the system with the use of monolingual data?
I am trying to find scientific literature that studies if, in a situation in which we already have enough parallel data, the addition of monolingual data can further improve performance.
I have not ...
- 23
1
vote
1
answer
174
views
What toy dataset can be used/made for debugging a neural machine translation system?
I'm looking for something like the scikit-learn sample generators for generating small machine translation datasets that are quick to learn. My goal is to debug a Transformer model and have a quick ...
- 113
2
votes
1
answer
314
views
How do RNNs used in Machine Translation have the right output length?
For machine translation the length of input and output sequences is mostly different. Typically considering an encoder-decoder architecture is used, how does the output come out to be the right length ...
1
vote
0
answers
66
views
What is currently the best way to add a custom dictionary to a neural machine translator that uses the transformer architecture?
It's common to add a custom dictionary to a machine translator to ensure that terminology from a specific domain is correctly translated. For example, the term server should be translated differently ...
- 161
312
votes
11
answers
271k
views
What exactly are keys, queries, and values in attention mechanisms?
How should one understand the keys, queries, and values that are often mentioned in attention mechanisms?
I've tried searching online, but all the resources I find only speak of them as if the reader ...
- 4,177
0
votes
0
answers
93
views
Machine translation dataset is actually not parallel
I'm currently working on implementing the Transformer model for machine translation. I'm taking a look at the data that was used in the actual paper and also used by many other implementations ...
- 4,177
7
votes
1
answer
174
views
What exactly does the word "align" mean in the attention model?
I'm reading the famous attention model paper Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho, & Bengio, 2014) and I'm having trouble what the exact meaning of "...
- 4,177
3
votes
2
answers
1k
views
Multi-Head attention mechanism in transformer and need of feed forward neural network
After reading the paper, "Attention is all you need," I have two questions.
1) What is the need of multi-head attention mechanism? Paper says that "Multi-head attention allows the model to jointly ...
- 173
3
votes
0
answers
35
views
From a deep learning point of view, is there a lower limit on the number of hours of speech needed to train a neural net
From a deep learning practitioner's point of view, is there a lower limit on the number of hours of speech needed to train a neural net to translate speech to text? An estimate from CMU is 3000-5000 ...
- 409
0
votes
1
answer
104
views
How does SGD come in the picture for Sequence to Sequence models?
I was learning that seq2seq models (from the deeplearning.ai course) try to maximize:
$$ \max_{y} P_{\theta}(y_1 \dots y_{T'} \mid x_1 \dots x_T ) $$
I learned that one way they do it is via beam ...
- 7,074
12
votes
1
answer
2k
views
What is the intuition behind the positional cosine encoding in the transformer network?
I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence.
...
- 1,373
2
votes
1
answer
1k
views
Is Length Normalization used in each step of Beam Search?
In Andrew Ng's lesson on refining Beam Search, it seems that Length Normalization is used ONLY AFTER LAST STEP of Beam Search, that is, when the B most probable sequences have been generated. My ...
- 383
1
vote
0
answers
97
views
How should I choose the Encoder hyperparameters to make its memory state suitable for the Decoder in a Bidirectional Neural Network?
I'm trying to implement Neural Machine Translation following the tutorial on the Tensorflow website here https://www.tensorflow.org/versions/r1.8/tutorials/seq2seq and I was able to build a ...
- 11
0
votes
1
answer
108
views
Adapt speech recognition for Shakespeare english
We need to be able to search the works of Shakespeare by voice. The way I see it, the goal is if I quote into the microphone:
"Yet but three come one more.
Two of both kinds make up four.
Ere ...
- 155
35
votes
5
answers
47k
views
What are "residual connections" in RNNs?
In Google's paper Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, it is stated
Our LSTM RNNs have $8$ layers, with residual connections between ...
user82135
0
votes
1
answer
696
views
Word representation in sequence-to-sequence learning with LSTM
I understand that in case of seq2seq learning for machine translation or question-answer systems one must encode words of sentences. Most approaches do this via word2vec representation where each word ...
- 233
1
vote
1
answer
348
views
Highest value of machine translation metrics
Does anyone knows the highest and lowest values of the common machine translation metrics $BLEU_{1-4}$, $METEOR$, $ROUGE_{L}$, and $CIDEr$ and $CIDEr-D$?
I know that for BLEU is 1, correct? Does the ...
- 212
2
votes
2
answers
3k
views
Why use the cosine distance for machine translation (Mikolov paper)?
I am currently reading the paper "Exploiting Similarities among Languages for Machine Translation" by Mikolov et al. (available here : https://static.googleusercontent.com/media/research.google.com/en/...
- 120
2
votes
1
answer
45
views
In phrase-based machine translators, how does the program recognize phrases in the corpus text?
I know that a phrase-based statistical machine translator finds the probability of a correct translation by analyzing a bilingual corpus text, and it maps phrases from the one language to phrases in ...
- 133
2
votes
0
answers
63
views
Was BaiduTrans the first scalable deployment of Neural Machine Translation?
I read these two tweets written on 2016-12-15 by Andrew Ng:
Strong desire for global content made China 1st to develop Neural Machine Translation. US lucky to have so much english content
[…]
@...
- 47.6k
1
vote
2
answers
1k
views
NLTK: odd outputs from bleu_score
For machine translation purposes I use bleu score, which seems to be the validation mechanism of choice (used in the sutskever 2014 sequence-to-sequence).
The purpose is to get as high bleu as ...
3
votes
0
answers
209
views
Hessian-Free instead of LSTM for Recurrent Net Machine Translation
Last year, Ilya Sutskever and collaborators came out with a paper about a recurrent LSTM net that learns sequence to sequence mappings for machine translation. It's somewhat surprising that the ...
- 143
2
votes
2
answers
625
views
Removing human evaluator bias
I am working on machine translation evaluation, and am looking at ratings given by humans who are judging the quality of sentences produced by machine translation systems. The evaluators give fluency ...
- 121