I'm trying to understand the derivatives w.r.t. the softmax arguments when used in conjunction with a squared loss (for example as the last layer of a neural network).
I am using the following notation:
Softmax argument for class $i$:
$a_i=b_i + W_i \mathbf{h}$
Vector of softmax arguments:
$\mathbf{a} = (a_1,...,a_K)^T$
Softmax probability for class $i$:
$p_i(\mathbf{a}) = \frac{e^{a_i}}{\sum_k{e^{a_k}}}$
$\mathbf{p(a)} = (p_1(\mathbf{a}),...,p_K(\mathbf{a}))^T$
Vector with true labels:
$\mathbf{y} = (0, ... 0,1, 0,...0)^T$
Squared loss:
$L(\mathbf{p(a)},\mathbf{y}) = \sum_i(p_i(\mathbf{a}) - y_i)^2$
Now, I want to get the derivative of the loss w.r.t. a particular softmax argument $a_j$. This is what I arrived at:
$\frac{d}{da_j} L(\mathbf{p(a)},\mathbf{y})$
$= \sum_i \frac{d}{da_j} [p_i^2(\mathbf{a}) -2y_ip_i(\mathbf{a}) + y_i^2] $
$= \sum_i \frac{d}{dp_i} [p_i^2(\mathbf{a}) -2y_ip_i(\mathbf{a})] \frac{dp_i}{da_j} $
$=2 \sum_i (p_i(\mathbf{a}) - y_i) p_i(\mathbf{a}) (1_{i=j} - p_j(\mathbf{a})) $
Looking at the draft of the Bengio et al. Neural Networks book (Chapter 6, http://www.iro.umontreal.ca/~bengioy/DLbook/), they state this equation:
$\frac{d}{da_j} L(\mathbf{p(a)},\mathbf{y}) = 2 (\mathbf{p(a)} - \mathbf{y}) \odot \mathbf{p} \odot (1 - \mathbf{p}) $
Where $\odot$ is element-wise multiplication.
How should I interprete this? Is there a mistake in my derivation?