0
$\begingroup$

Recently, I studied NNLM and I saw the derivation of softmax using MLE:

\begin{align} & \frac{\partial\log P(w_t\mid h)}{\partial\theta} \\[8pt] = {} & \frac{\partial \log \exp(s_\theta(w_t,h)) - \log\sum_{w'\in V} \exp(s_\theta (w', h))}{\partial\theta} \end{align} For the second, the log that inconveniently precedes the sum (known as a log-sum-exp) complicates the calculation of the derivative, but it can be approached using two inverse identities to rewrite the second term into a more conveinent form:
\begin{align} & \frac{\partial}{\partial\theta}\log\sum_{w'\in V} \exp(s_\theta (w', h)) \\[8pt] = {} & \frac1{\sum_{w'\in V}\exp(s_\theta (w', h))}\sum_{w'\in V}\frac{\partial}{\partial\theta} \exp(s_\theta (w', h)) \\[8pt] = {} & \frac1{Z_\theta(h)}\sum_{w'\in V}\exp(s_\theta (w', h))\frac{\partial}{\partial\theta}\log \exp(s_\theta (w_t, h)) \\[8pt] = {} & \sum_{w'\in V} \frac{\exp(s_\theta (w', h))}{Z_\theta(h)}\, \frac{\partial}{\partial\theta}s_\theta(w',h) \end{align} where the $w_t$ is the target word, $h$ is history which means the contexts of the target word, and $V$ is vocabulary.

In the derivation, I don't understand why $w'$ changed to $w_t$. Can you please explain it to me?

Thanks for reading.

$\endgroup$
1
$\begingroup$

Your likelihood is $$ P(w_t\mid h) = \frac{\exp(s_\theta(w_t,h))}{\sum_{w'\in V} \exp(s_\theta (w', h))}. $$

$w_t$ is one specific value, while $w'$ is the index of the sum, and it's taking on all values in $V$. Then you take the log of this, then the derivative.

In the second part of your derivation, you're only looking at the second term, which came from the denominator of the above expression. The first line is chain rule, and the fact that the derivative of the sum is the sum of the derivatives. I'll mark that line with a (1). \begin{align} & \frac{\partial}{\partial\theta}\log\sum_{w'\in V} \exp(s_\theta (w', h)) \\[8pt] = {} & \frac1{\sum_{w'\in V}\exp(s_\theta (w', h))}\sum_{w'\in V}\frac{\partial}{\partial\theta} \exp(s_\theta (w', h)) \tag{1}\\[8pt] = {} & \frac1{Z_\theta(h)}\sum_{w'\in V}\exp(s_\theta (w', h))\frac{\partial}{\partial\theta}\log \exp(s_\theta (w_t, h)) \tag{2} \\[8pt] = {} & \sum_{w'\in V} \frac{\exp(s_\theta (w', h))}{Z_\theta(h)}\, \frac{\partial}{\partial\theta}s_\theta(w',h) \end{align}

Proceeding on, if you look at $$ \sum_{w'\in V}\frac{\partial}{\partial\theta} \exp(s_\theta (w', h)) $$ this equals $$ \sum_{w'\in V} \exp(s_\theta (w', h))\frac{\partial}{\partial\theta}s_\theta (w', h) $$ by the chain rule again. (2) just rewrites stuff using the following identity: $$ s_\theta (w', h) = \exp[\log(s_\theta (w', h))]. $$ But yes, that appears to be a typo in line (2) -- this term is written after replacing $w'$ with $w_t$. It should be what I wrote in the display directly above.

Also, the use of this identity was pretty superfluous because in the next line they just undo it. They also move the $Z_{\theta}(h)$ inside the sum.

$\endgroup$
  • $\begingroup$ I don't understand why your answer is relative to my question. So why $w'$ is changed to $w_t$? $\endgroup$ – Yoo Inhyeok Jul 12 '19 at 6:23
  • $\begingroup$ @YooInhyeok You're right---it looks like a typo. Just edited it to make it more clear. $\endgroup$ – Taylor Jul 12 '19 at 12:50
  • $\begingroup$ Oh so you think it's just a typo? It could be. $\endgroup$ – Yoo Inhyeok Jul 15 '19 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.