0
$\begingroup$

I am a bit confused as to how to use different sets of source and target vocabularies in deep learning for NLP tasks.

  • What are the implications of using separate source and target vocabularies (for example, in machine translation, or in general)?
  • Should the indices for corresponding tokens match in the word embeddings?
  • How should their sizes be relative to each other?
  • Do you have to modify the architecture in any way to accommodate this, or could you simply change the path to vocabulary?

Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

You choose vocab size for each language separately and simplest approach is just to preserve all words but usually you skip most common stop words ('a', 'an'...) and those very rare (e.g. occuring only once). So you end with two vocabs with different sizes and they don't need to be the same / similar. And effectively you have two vocabs, e.g. input, target, the architectures/solution are/should support that.

When you create embeddings (for each language separately), they have features that are not one-hot encoded words - these are 'latent features' that encode meanings of specific words in a new space of selected dimensionality, much lower than size of vocab, e.g. 100 or 300.

Have a look e.g. here: https://towardsdatascience.com/seq2seq-model-in-tensorflow-ec0c557e560f

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.