Considering the following functions I'm having a tough time finding the appropriate gradient function for the log-likelihood as defined below:

$a_k(x)=\sum_{i=1}^D w_{ki}\cdot x_i$

$P(y_k|x) = {\exp\{a_k(x)\}}\big/{\sum_{k'=1}^K \exp\{a_{k'}(x)\}}$

$L(w)=\sum_{n=1}^N\sum_{k=1}^Ky_{nk}\cdot \ln(P(y_k|x_n))$

where

• $x$ is a vector of inputs defined by 8x8 binary pixels (0 or 1)
• $y_{nk} = 1$ iff the label of sample $n$ is $y_k$ (otherwise 0)
• $N$ is the number of training examples
• $K$ is the number of class labels
• $D := \left\{\left(y_n,x_n\right) \right\}_{n=1}^{N}$

The partial derivatives of the gradient for each weight $w_{k,i}$ should look like this:

$\left<\frac{\delta}{\delta w_{1,1}}L,...,\frac{\delta}{\delta w_{k,i}}L,...,\frac{\delta}{\delta w_{K,D}}L \right>$

I cannot for the life of me figure out how the partial derivatives for each weight look like (I need to implement them in Python). I'm hoping that somebody of you can help me out on this or at least point me in the right direction.

• $P(y_k|x) = \text{softmax}_k(a_k(x))$. The derivative of the softmax can be found here. That and the chain rule of calculus should be all you need. Dec 15, 2014 at 14:25
• Alright, I'll see what I can do with it. Thank you very much! Dec 15, 2014 at 17:01
• Are you new to calculus in general? If so I can provide a more complete answer. Dec 15, 2014 at 22:18
• Well, to be honest, I should be able to do it on my own, but I'm still struggling because it's been quite a while since the last time I did math of this complexity... I'm a little rusty. So, yes, I'd be really grateful if you would provide me (and others maybe) with a more complete and actual answer. I would like to give you reputation by putting a green hook on it. :-) Dec 16, 2014 at 8:24
• I finally found my mistake this morning. Infernce and likelihood functions were working with the input data directly whereas the gradient was using a vector of incompatible feature data. Now, using this feature data in all three functions, everything works as expected. Still, I'd love to see a complete answer because I still need to fill some gaps in my understanding of how the gradient works. Thanks a lot! Dec 16, 2014 at 11:35

So if we construct a matrix $W$ by vertically stacking the vectors $w^T_{k^\prime}$, we can write the objective as

$$L(w) = \sum_{n,k} y_{nk} \ln \text{softmax}_k(Wx)$$

Then

$$\frac{\partial}{\partial w_{ij}} L(w) = \sum_{n,k} y_{nk} \frac{1}{\text{softmax}_k(Wx)} \times \frac{\partial}{\partial w_{ij}}\text{softmax}_k(Wx)$$

Now the derivative of the softmax function is

$$\frac{\partial}{\partial z_l}\text{softmax}_k(z) = \text{softmax}_k(z)(\delta_{kl} - \text{softmax}_l(z))$$

and if $z = Wx$ it follows by the chain rule that

\begin{align} \frac{\partial}{\partial w_{ij}}\text{softmax}_k(z) & = \sum_l \text{softmax}_k(z)(\delta_{kl} - \text{softmax}_l(z)) \times \frac{\partial z_l}{\partial w_{ij}} \\% & = \text{softmax}_k(z)(\delta_{ki} - \text{softmax}_i(z)) \times x_j \end{align}

So the whole thing is

\begin{align} \frac{\partial}{\partial w_{ij}} L(w) & = \sum_{n,k} y_{nk} \frac{1}{\text{softmax}_k(Wx)} \times \text{softmax}_k(z)(\delta_{ki} - \text{softmax}_i(z)) \times x_j \\ & = \sum_{n,k} y_{nk} (\delta_{ki} - \text{softmax}_i(Wx)) \times x_j \end{align}

Now, having wrote all that I realise my calculus isn't as smooth as it once was either! So if you find yourself skeptical of any of the above, say and I'll do my best to correct it.