Is there a name for the composition of the cross-entropy and softmax functions?

Question

This is a simple question but I'll give some background. The softmax function $S: \mathbb R^K \to \mathbb R^K$ is defined by $$ S(u) = \begin{bmatrix} \frac{e^{u_1}}{\sum_j e^{u_j}} \\ \frac{e^{u_2}}{\sum_j e^{u_j}} \\ \vdots \\ \frac{e^{u_K}}{\sum_j e^{u_j}} \\ \end{bmatrix}. $$ The cross-entropy loss function $\ell$ takes as input probability vectors $p$ and $q$ (vectors whose components are nonnegative and sum to $1$) and returns as output the number $$ \ell(p,q) = -\sum_{k=1}^K p_k \log(q_k). $$ These two functions are fundamental ingredients of machine learning algorithms. They go together so nicely that it feels as if they are meant to be united into a single entity. Given a probability vector $p \in \mathbb R^K$, define $h: \mathbb R^K \to \mathbb R$ by $$ h(u) = \ell(p,S(u)). $$ Question: Is there a standard name for this function $h$? I've done some Googling but haven't found a name for it.

---

Here are some details about why $h$ is such a nice function. Notice that \begin{align} h(u) &= - \sum_{k = 1}^K p_k \log\left(\frac{e^{u_k}}{\sum_j e^{u_j}}\right) \\ &= -\sum_{k=1}^K p_k u_k - p_k \log\left(\sum_j e^{u_j} \right) \\ &= - \langle p, u \rangle + \log\left(\sum_j e^{u_j} \right). \end{align} The formula for $h$ has simplified nicely, and the logSumExp function has appeared. The logSumExp function is a natural companion of $S$, and in fact the gradient of logSumExp is equal to $S$. It follows that $$ \nabla h(u) = S(u) - p, $$ which is a beautiful formula that can be interpreted as follows: If the probability vector $S(u)$ agrees perfectly with $p$, then the gradient is $0$, meaning that no change is needed.

Answer 1

In terms of statistical physics, softmax on any vector just returns the associated Boltzmann distribution with temperature equal to the inverse of the Boltzmann constant.

Letting $q = S(u)$, this distribution is directly related to entropy. In particular, the Boltzmann distribution is the solution to the constrained optimisation problem

$$ \max_{q_1, q_2, \ldots,q_K} -\sum_k q_k \log q_k \\ \text{s.t.}\quad \langle q \rangle := \sum_l q_l u_l = E, $$ where $E$ is some mean energy. The expression $\langle q \rangle$ is just the expected energy of a system with possible states given by $u$.

Your observation follows directly from this - since $q$ is the maximizer, then if you have that $q = p$ the gradient, of course, has to be 0.