# Deriving the cost function for a continuous bag of words model

I've been reading the following link: http://cs224d.stanford.edu/lecture_notes/notes1.pdf, with the relevant content starting on page 6.

Some background on Continuous Bag of Words

The continuous bag of words (CBOW) model takes a context (e.g. a sentence with a word missing), and predicts a word that should fit into the context (e.g. the missing word).

It starts by considering one-hot vector representations for each word in the input sentence: $x^{c-m},...,x^{c-1}, x^{c+1}, ..., x^{c+m}$, where $m$ is some integer context distance from the word to be found.

Vector representations can then be found by doing $v_{c-m}=Vx^{c-m}$, and the same for every other one-hot vector, producing $2m$ results. These vectors are then averaged, to give $\hat{v}$.

The score vector is $z=U\hat{v}$, and this is turned into a probability using softmax.

Cost function

The cost function to be minimised is: $$J=-\log{P(w_c|w_{c-m},...w_{c-1}, w_{c+1},...,w_{c+m}})=-\log{P(u_c|\hat{v})}.$$ Up to here makes sense to me. Obviously, the higher the probability of $w_c$, the lower the cost function is going to be, and $u_i$ is a vector of the $i$-th column in $U$.

I'm struggling to understand the next part: $$J=-\log{\frac{\exp(u_c^T\hat{v})}{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}}=-u_c^T\hat{v}+\log{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}.$$

Could someone please break down this step to help me understand where it comes from?

Am I missing some fundamental understanding of linear algebra, or do I lack in understanding of the word vector representations?

• a) P(u) is given by softmax function – seanv507 Apr 1 '18 at 16:27
• B) log(exp (X)) =X, log X/y = log(X) - log(y) – seanv507 Apr 1 '18 at 16:29
• @seanv507 Thanks, I guess I wasn't using my brain! Posted an answer below now. – quanty Apr 3 '18 at 13:19

Understanding these steps requires a basic understanding of

1. The softmax function.

2. Rules of logarithms.

Problem 1

$$-\log{P(u_c|\hat{v})}=-\log{\frac{\exp(u_c^T\hat{v})}{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}}$$

The softmax function is defined (as can be seen on https://en.wikipedia.org/wiki/Softmax_function) as $$P(y=j|x)=\frac{\exp(x^Tw_j)}{\sum_{k=1}^K\exp(x^Tw_k)},$$ where $x\longmapsto x^Tw_j \text{ }\forall j\in [1, K]$.

Problem 2

$$-\log{\frac{\exp(u_c^T\hat{v})}{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}}=-u_c^T\hat{v}+\log\sum_{j=1}^{|V|}\exp(u_j^T\hat{v}).$$

Consider the following three log rules:

1. $\log(\frac{a}{b})=\log(a)-\log(b)$

2. $\log(a^n)=n\log(a)$

3. $\ln(e)=1$

(In the case in the question, we are dealing with natural logarithms, so $\log = \ln$.)

Applying rule 1, we get $$-\log{\frac{\exp(u_c^T\hat{v})}{\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})}} = -\log(e^{u_c^T\hat{v}})+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big).$$

Applying rule 2 gives

$$-\log(e^{u_c^T\hat{v}})+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big)=-u_c^T\hat{v}(\log(e))+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big).$$

Finally, applying rule 3 we get

$$u_c^T\hat{v}(\log(e))+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big)=u_c^T\hat{v}+\log\Big(\sum_{j=1}^{|V|}\exp(u_j^T\hat{v})\Big),$$ which is the final answer.