1

Your likelihood is $$ P(w_t\mid h) = \frac{\exp(s_\theta(w_t,h))}{\sum_{w'\in V} \exp(s_\theta (w', h))}. $$ $w_t$ is one specific value, while $w'$ is the index of the sum, and it's taking on all values in $V$. Then you take the log of this, then the derivative. In the second part of your derivation, you're only looking at the second term, which came from ...


1

But arent the layers at the top non linear combinations of the layers at the bottom? Indeed, this is the case, but that is kind of the point. Although the comparison is not perfect, think of a a large convolutional neural network. The bottom layers learn to distinguish very detailed local features, while the higher level layers learn more abstract ...


Only top voted, non community-wiki answers of a minimum length are eligible