This is a simple question but I'll give some background. The softmax function $S: \mathbb R^K \to \mathbb R^K$ is defined by $$ S(u) = \begin{bmatrix} \frac{e^{u_1}}{\sum_j e^{u_j}} \\ \frac{e^{u_2}}{\sum_j e^{u_j}} \\ \vdots \\ \frac{e^{u_K}}{\sum_j e^{u_j}} \\ \end{bmatrix}. $$ The cross-entropy loss function $\ell$ takes as input probability vectors $p$ and $q$ (vectors whose components are nonnegative and sum to $1$) and returns as output the number $$ \ell(p,q) = -\sum_{k=1}^K p_k \log(q_k). $$ These two functions are fundamental ingredients of machine learning algorithms. They go together so nicely that it feels as if they are meant to be united into a single entity. Given a probability vector $p \in \mathbb R^K$, define $h: \mathbb R^K \to \mathbb R$ by $$ h(u) = \ell(p,S(u)). $$ Question: Is there a standard name for this function $h$? I've done some Googling but haven't found a name for it.

Here are some details about why $h$ is such a nice function. Notice that \begin{align} h(u) &= - \sum_{k = 1}^K p_k \log\left(\frac{e^{u_k}}{\sum_j e^{u_j}}\right) \\ &= -\sum_{k=1}^K p_k u_k - p_k \log\left(\sum_j e^{u_j} \right) \\ &= - \langle p, u \rangle + \log\left(\sum_j e^{u_j} \right). \end{align} The formula for $h$ has simplified nicely, and the logSumExp function has appeared. The logSumExp function is a natural companion of $S$, and in fact the gradient of logSumExp is equal to $S$. It follows that $$ \nabla h(u) = S(u) - p, $$ which is a beautiful formula that can be interpreted as follows: If the probability vector $S(u)$ agrees perfectly with $p$, then the gradient is $0$, meaning that no change is needed.

  • 1
    $\begingroup$ many machine learning frameworks seem to have converged on calling this "cross-entropy with logits" -- quite unimaginative if nothing else $\endgroup$ – shimao Feb 26 at 21:57

In terms of statistical physics, softmax on any vector just returns the associated Boltzmann distribution with temperature equal to the inverse of the Boltzmann constant.

Letting $q = S(u)$, this distribution is directly related to entropy. In particular, the Boltzmann distribution is the solution to the constrained optimisation problem

$$ \max_{q_1, q_2, \ldots,q_K} -\sum_k q_k \log q_k \\ \text{s.t.}\quad \langle q \rangle := \sum_l q_l u_l = E, $$ where $E$ is some mean energy. The expression $\langle q \rangle$ is just the expected energy of a system with possible states given by $u$.

Your observation follows directly from this - since $q$ is the maximizer, then if you have that $q = p$ the gradient, of course, has to be 0.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.