1
$\begingroup$

Softmax function $a^L_j = z^L_j / sum_k(z^L_k)$. When we think about the monotonicity of the Softmax function, $∂a^L_j/∂z^L_k$ is positive if j=k, and negative if j≠k. As a consequence, increasing $z^L_j$ is guaranteed to increase the corresponding output activation $a^L_j$, and will decrease all the other output activations.

My question is, decreasing $z^L_j$ will decrease the corresponding output and will increase all the other output activations. But, $\frac{∂a^L_j}{∂z^L_k}$ is always positive if j=k, and negative if j≠k. So, I'm curious how to prove that decreasing $z^L_j$ will actually decrease the corresponding output when we always have $\frac{∂a^L_j}{∂z^L_k}$ is positive when j=k?

Improve this question
$\endgroup$
5
  • $\begingroup$ Isn’t this immediately true from the definition of a partial derivative? $\endgroup$
    – Arya McCarthy
    Jul 19, 2021 at 10:56
  • $\begingroup$ Uh.. can you elaborate on it? da/dz = a - a^2, where 0 <= a <= 1, j=k. I was thinking the partial derivative of a with respect to z's always positive so a was always increasing, regardless if z's decreasing or not. $\endgroup$
    – Kay
    Jul 19, 2021 at 11:36
  • $\begingroup$ A partial derivative is the (instantaneous) rate of change of one variable with respect to another. With a positive partial derivative, you increase $z$ by some amount, you increase $a$ by a corresponding amount. Similarly, if you decrease $z$, then you decrease $a$ by a corresponding amount. $\endgroup$
    – Arya McCarthy
    Jul 19, 2021 at 15:53
  • $\begingroup$ I see now. Thank you very much. By the way, is there any way to vote up your answer on comment? $\endgroup$
    – Kay
    Jul 19, 2021 at 23:45
  • $\begingroup$ I’ve turned this into an answer now, so yes. $\endgroup$
    – Arya McCarthy
    Jul 20, 2021 at 19:41

1 Answer 1

Reset to default
2
$\begingroup$

(Consolidating my comments into an answer.)

This comes directly from the definition of a partial derivative.

A partial derivative is the (instantaneous) rate of change of one variable with respect to another. With a positive partial derivative, you increase $z$ by some amount, you increase $a$ by a corresponding amount. Similarly, if you decrease $z$, then you decrease $a$ by a corresponding amount.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.