0
$\begingroup$

I've been reading the following link: http://cs224d.stanford.edu/lecture_notes/notes1.pdf, with the relevant content starting on page 6.

Some background on Continuous Bag of Words

The continuous bag of words (CBOW) model takes a context (e.g. a sentence with a word missing), and predicts a word that should fit into the context (e.g. the missing word).

It starts by considering one-hot vector representations for each word in the input sentence: $x^{c-m},...,x^{c-1}, x^{c+1}, ..., x^{c+m}$, where $m$ is some integer context distance from the word to be found.

Vector representations can then be found by doing $v_{c-m}=Vx^{c-m}$, and the same for every other one-hot vector, producing $2m$ results. These vectors are then averaged, to give $\hat{v}$.

The score vector is $z=U\hat{v}$, and this is turned into a probability using softmax.

Cost function

The cost function to be minimised is: $$J=-\log{P(w_c|w_{c-m},...w_{c-1}, w_{c+1},...,w_{c+m}})=-\log{P(u_c|\hat{v})}.$$ Up to here makes sense to me. Obviously, the higher the probability of $w_c$, the lower the cost function is going to be, and $u_i$ is a vector of the $i$-th column in $U$.

I'm struggling to understand the next part: $$J=-\log{\frac{\exp(u_c^T\hat{v})}{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}}=-u_c^T\hat{v}+\log{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}.$$

Could someone please break down this step to help me understand where it comes from?

Am I missing some fundamental understanding of linear algebra, or do I lack in understanding of the word vector representations?

$\endgroup$
  • $\begingroup$ a) P(u) is given by softmax function $\endgroup$ – seanv507 Apr 1 '18 at 16:27
  • $\begingroup$ B) log(exp (X)) =X, log X/y = log(X) - log(y) $\endgroup$ – seanv507 Apr 1 '18 at 16:29
  • $\begingroup$ @seanv507 Thanks, I guess I wasn't using my brain! Posted an answer below now. $\endgroup$ – quanty Apr 3 '18 at 13:19
0
$\begingroup$

Understanding these steps requires a basic understanding of

  1. The softmax function.

  2. Rules of logarithms.

Problem 1

$$-\log{P(u_c|\hat{v})}=-\log{\frac{\exp(u_c^T\hat{v})}{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}}$$

The softmax function is defined (as can be seen on https://en.wikipedia.org/wiki/Softmax_function) as $$P(y=j|x)=\frac{\exp(x^Tw_j)}{\sum_{k=1}^K\exp(x^Tw_k)},$$ where $x\longmapsto x^Tw_j \text{ }\forall j\in [1, K]$.

Problem 2

$$-\log{\frac{\exp(u_c^T\hat{v})}{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}}=-u_c^T\hat{v}+\log\sum_{j=1}^{|V|}\exp(u_j^T\hat{v}).$$

Consider the following three log rules:

  1. $\log(\frac{a}{b})=\log(a)-\log(b)$

  2. $\log(a^n)=n\log(a)$

  3. $\ln(e)=1$

(In the case in the question, we are dealing with natural logarithms, so $\log = \ln$.)

Applying rule 1, we get $$-\log{\frac{\exp(u_c^T\hat{v})}{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}} = -\log(e^{u_c^T\hat{v}})+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big).$$

Applying rule 2 gives

$$-\log(e^{u_c^T\hat{v}})+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big)=-u_c^T\hat{v}(\log(e))+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big).$$

Finally, applying rule 3 we get

$$u_c^T\hat{v}(\log(e))+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big)=u_c^T\hat{v}+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big),$$ which is the final answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.