1
$\begingroup$

I'm trying to find the derivative of the log-likelihood function in softmax regression. I have (with $\Theta$ being the parameters, and $x^{(i)}$ being the $i$th training example, and $s_j$ representing the softmax function),

$$\ell(\Theta) = \sum_{i=1}^m \sum_{j=1}^k \log \left( \frac{e^{\Theta_j^T x^{(i)}}}{\sum_{l=1}^k e^{\Theta_l^T x^{(i)}}} \right)^{I\{y^{(i)}=j\}}$$ I got the derivative of the softmax function itself as
$$\frac{\partial}{\partial \Theta_p} \left( \frac{e^{\Theta^T_j x^{(i)}}}{\sum_{l=1}^k e^{\Theta^T_l x^{(i)}}} \right)=s_j(\delta_{pj}-s_p)x^{(i)}$$
On using this to find the derivative of the log-likelihood, I get $$\begin{aligned}\frac{\partial}{\partial \Theta_p}\ell(\Theta) &= \sum_{i=1}^m \sum_{j=1}^k I\{y^{(i)}=j\} \cdot \frac{s_j(\delta_{pj}-s_p)x^{(i)}}{s_j} \\ &= \sum_{i=1}^m \sum_{j=1}^k I\{y^{(i)}=j\} \cdot (\delta_{pj}-s_p)x^{(i)} \\ &= \sum_{i=1}^m \sum_{j=1}^k I\{y^{(i)}=p\}x^{(i)}-I\{y^{(i)}=j\}s_p x^{(i)} \end{aligned} $$
I'm not sure where to go from here. From what I've seen online, I shouldn't have the second summation at all, and the second term should just be $s_p$.

I'm sure I'm just missing a step or two, but I'd love if someone could help me move ahead.

$\endgroup$
1
  • $\begingroup$ I don't think this is off-topic. On the other hand, it should be probably marked as [self-study], and it has probably an answer somewhere at CrossValidated already. $\endgroup$ Commented Jun 27, 2018 at 6:54

1 Answer 1

3
$\begingroup$

You are correct up to the second line of your working in the last part, and then you make an error by dropping the requirement that $j=p$ (which means you retain an additional sum that shouldn't be there). Continuing from your last correct step, you should have:

$$\begin{aligned} \frac{\partial \ell}{\partial \Theta_p}(\Theta) &= \sum_{i=1}^m \sum_{j=1}^k \mathbb{I}(y^{(i)}=j) (\delta_{pj}-s_p) x^{(i)} \\[6pt] &= \sum_{i=1}^m x^{(i)} \sum_{j=1}^k \mathbb{I}(y^{(i)}=j) (\delta_{pj}-s_p) \\[6pt] &= \sum_{i=1}^m x^{(i)} \bigg[ \sum_{j=1}^k \mathbb{I}(y^{(i)}=j) \mathbb{I}(p=j) - s_p \sum_{j=1}^k \mathbb{I}(y^{(i)}=j) \Bigg] \\[6pt] &= \sum_{i=1}^m x^{(i)} \bigg[ \mathbb{I}(y^{(i)}=p) - s_p \Bigg] \\[6pt] &= \sum_{i=1}^m x^{(i)} \mathbb{I}(y^{(i)}=p) - s_p \sum_{i=1}^m x^{(i)}. \\[6pt] \end{aligned} $$

(The penultimate step follows from the fact that $\sum_{j=1}^k \mathbb{I}(y^{(i)}=j) = 1$ for all $i = 1, ..., m$.)

$\endgroup$
2
  • $\begingroup$ Thanks! I totally missed the insight that the summation of the indicators would be 1. $\endgroup$
    – Rahul
    Commented Jun 27, 2018 at 6:46
  • $\begingroup$ Note that $s_p$ is effectively a function of $x^{(i)}$, and so it can not be moved before the summation in the last term on the last line of the multiline formula. $\endgroup$ Commented Mar 14, 2023 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.