Derivative of softmax with random loss function I am having trouble understanding the derivative of the softmax with respect to loss function.I will try to explain my understanding with one example. Suppose we have vector of 3 values and we want to compute softmax [a1 = 5, a2 = 3, a3 = 2]. The output is vector again with size 3 and all the values should sum up to 1.
Output = [ o1 = 0.84, o2 = 0.11, o3 = 0.05]. This is because of the formula of softmax which is normalized by the sum of all the exp of the input. Softmax formula: 

This gives the output for the first value of the output vector o1 = 0.84. Now suppose we have a random loss function lets call it L , which again outputs vector with values [ y1, y2, y3 ]. The output from the softmax will be supplied as input to the L function.
So what is the derivative of the softmax with respect to L. Lets take for simplicity a1 we want to find the derivative of a1 with respect to the L function :

This is the equation using the chain rule, and now substitute the derivative of o1 with respect with a1 we end up with :

Are my calculations correct or i am missing something.
 A: 
So what is the derivative of the softmax with respect to L.

Normally, the loss function used in neural networks is a scalar, which is why we work with partial derivatives ($L:\mathbb R^n \mapsto  \mathbb R^1$).
If you have a vector valued input to a loss function and a vector valued output ($L:\mathbb R^n \mapsto  \mathbb R^m$), first-order derivatives are summarized in a Jacobian matrix.
Let's assume you have a loss function that produces a vector. In this case, your derivative calculation would make sense only if $a_1$ is not used in the calculation of $y_2$ and $y_3$. Otherwise, for each parameter (e.g. $a_1$), you end up with a Jacobian as opposed to a partial derivative. How to update parameters using a vector-valued derivative is unclear, as you are now trying to optimize for multiple different outcomes. This is not to say that we don't do this, in fact, multi-task learning does exactly this. As it turns out, there is a whole lot of research on this topic alone. Check out this paper for example. Other friendly and more accessible introduction to this setting are here and here.
