0
$\begingroup$

I am referring to this question: Derivative of cross entropy loss in word2vec

$CE(w, \hat{w}) = -\sum_{k}^{|V|} w_klog(\frac{\exp(w_k^T \hat{r})}{\sum_{j}^{|V|}exp(w_j^T\hat{r})})$

$= -\sum_{k}^{|V|} w_klog(\exp(w_k^T \hat{r}) + w_klog(\sum_{j}^{|V|}exp(w_j^T\hat{r}))$

How is the $ \sum_{k}^{|V|} $ disappearing in the second term? I think it has something to do with the one hot vector, but I cant figure it out?

$\endgroup$

1 Answer 1

1
$\begingroup$

Only the term under the log has been split. The $ \Sigma_k^{|V|}w_k $ term is still there, being common among the $\log(a/b) = \log(a) - \log(b)$ terms.

Later on, the first term is $\Sigma_k^{|V|}w_k \log \exp(w_k^T \hat r)$ only has $w_k$ dependent on k, so at the ith label remains, and then the partial derivative w.r.t. r will be $w_i$.

The second term will have two sum terms, each with their own index and that will yield a probability term w.r.t. the summation and the corresponding $w_x$ term.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.