1
$\begingroup$

I'm reading through Pytorch's NLP from Scratch: Translation with a Sequence to Sequence Network and Attention, and I am a bit confused on the Preparing Training Data section, particularly:

def indexesFromSentence(lang, sentence):
    return [lang.word2index[word] for word in sentence.split(' ')]


def tensorFromSentence(lang, sentence):
    indexes = indexesFromSentence(lang, sentence)
    indexes.append(EOS_token)
    return torch.tensor(indexes, dtype=torch.long, device=device).view(-1, 1)


def tensorsFromPair(pair):
    input_tensor = tensorFromSentence(input_lang, pair[0])
    target_tensor = tensorFromSentence(output_lang, pair[1])
    return (input_tensor, target_tensor)

Why do they add an EOS token to the end but not a SOS in the beginning?

$\endgroup$
1
$\begingroup$

It is not really important in the encoder, but it plays a crucial role in the decoder.

At the training time, you use the target sentence in the following way (with a 5-token sentence $w_1, \ldots, w_5$):

[BOS]  w₁   w₂   w₃   w₄   w₆
  ↓    ↓    ↓    ↓    ↓    ↓
┌─────────────────────────────┐
│           DECODER           │
└─────────────────────────────┘
  ↓    ↓    ↓    ↓    ↓    ↓   
  w₁   w₂   w₃   w₄   w₅ [EOS]

This means, the decoder needs to be provided with the [BOS] token to generate the first real token. At inference time, you need to know when to stop generating new tokens. You stop when you generate the [EOS] token.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.