0
$\begingroup$

In the context of word-based embeddings, the Negative Sampling algorithm chooses negative samples (k) from the most frequent words in the corpora which usually present less meaningful information than rare words [1]. How the algorithm chooses the negative samples to train character-based embeddings since the corpora consist of characters only? What is the selection criterion of (k) when it comes to characters?

[1] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems, pp. 3111–3119, 2013.

Improve this question
$\endgroup$
6
  • $\begingroup$ What do you mean by 'character-based embeddings'? (Word2vec only works on word-tokens.) $\endgroup$
    – gojomo
    Commented May 3, 2021 at 23:05
  • $\begingroup$ No, it was built for word-based embeddings, however you can build character-based embeddings for certain tasks. You can check github.com/dhwajraj/deep-siamese-text-similarity $\endgroup$
    – MManahi
    Commented May 4, 2021 at 9:19
  • 1
    $\begingroup$ Yes, deep neural networks can be used that way, but I'm not sure such 'deep siamese networks' even do the same sort of negative-word sampling as word2vec.. 'Word2vec' usually means the word-**based approach using a **shallow network. If you feed it characters-as-words, it'll use the exact same negative-sampling approach as with words - and not do anything interesting. (It's a very common rookie error, especially in Python, where strings behave like lists-of-one-character-strings.) $\endgroup$
    – gojomo
    Commented May 4, 2021 at 13:10
  • 1
    $\begingroup$ So again, what exactly does your question mean by mentioning "character-based embeddings"? Are you interested in what the word-based word2vec algorithm typically does, with words? Or what some other different algorithm does with characters? $\endgroup$
    – gojomo
    Commented May 4, 2021 at 13:11
  • 1
    $\begingroup$ If you're using pre-trained word2vec embeddings, there's no character aspect at all. Each word is known by an int index. Characters are irrelevant. If you're using some other algorithm, maybe characters are relevant, but you'll have to be specific about what other non-word2vec-algorithms are in play. $\endgroup$
    – gojomo
    Commented May 4, 2021 at 18:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.