All Questions
Tagged with word2vec loss-functions
4 questions
0
votes
0
answers
40
views
How does the chain-rule look for the gradient of a loss function?
When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like:
$...
- 1
1
vote
0
answers
173
views
When training word2vec, why is a new negative sampling process formulated instead of simply downsampling?
(For background, see The Skip-Gram Model.1 This question does not exactly use their notation, but you should be able to follow along.)
The original skip-gram log-likelihood of a single word, $w$, ...
- 1,625
7
votes
1
answer
2k
views
Do we use maximum likelihood or cross entropy Loss for training skip-gram model?
In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where
$$J(\theta) = -\frac{1}{T}\sum_{t=1}^...
- 1,116
1
vote
1
answer
312
views
Skip-gram model multiplicative constant in the objective function?
I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $...
- 271