# Is there a name for the composition of the cross-entropy and softmax functions?

This is a simple question but I'll give some background. The softmax function $$S: \mathbb R^K \to \mathbb R^K$$ is defined by $$S(u) = \begin{bmatrix} \frac{e^{u_1}}{\sum_j e^{u_j}} \\ \frac{e^{u_2}}{\sum_j e^{u_j}} \\ \vdots \\ \frac{e^{u_K}}{\sum_j e^{u_j}} \\ \end{bmatrix}.$$ The cross-entropy loss function $$\ell$$ takes as input probability vectors $$p$$ and $$q$$ (vectors whose components are nonnegative and sum to $$1$$) and returns as output the number $$\ell(p,q) = -\sum_{k=1}^K p_k \log(q_k).$$ These two functions are fundamental ingredients of machine learning algorithms. They go together so nicely that it feels as if they are meant to be united into a single entity. Given a probability vector $$p \in \mathbb R^K$$, define $$h: \mathbb R^K \to \mathbb R$$ by $$h(u) = \ell(p,S(u)).$$ Question: Is there a standard name for this function $$h$$? I've done some Googling but haven't found a name for it.

Here are some details about why $$h$$ is such a nice function. Notice that \begin{align} h(u) &= - \sum_{k = 1}^K p_k \log\left(\frac{e^{u_k}}{\sum_j e^{u_j}}\right) \\ &= -\sum_{k=1}^K p_k u_k - p_k \log\left(\sum_j e^{u_j} \right) \\ &= - \langle p, u \rangle + \log\left(\sum_j e^{u_j} \right). \end{align} The formula for $$h$$ has simplified nicely, and the logSumExp function has appeared. The logSumExp function is a natural companion of $$S$$, and in fact the gradient of logSumExp is equal to $$S$$. It follows that $$\nabla h(u) = S(u) - p,$$ which is a beautiful formula that can be interpreted as follows: If the probability vector $$S(u)$$ agrees perfectly with $$p$$, then the gradient is $$0$$, meaning that no change is needed.

• many machine learning frameworks seem to have converged on calling this "cross-entropy with logits" -- quite unimaginative if nothing else Feb 26, 2020 at 21:57

Letting $$q = S(u)$$, this distribution is directly related to entropy. In particular, the Boltzmann distribution is the solution to the constrained optimisation problem
$$\max_{q_1, q_2, \ldots,q_K} -\sum_k q_k \log q_k \\ \text{s.t.}\quad \langle q \rangle := \sum_l q_l u_l = E,$$ where $$E$$ is some mean energy. The expression $$\langle q \rangle$$ is just the expected energy of a system with possible states given by $$u$$.
Your observation follows directly from this - since $$q$$ is the maximizer, then if you have that $$q = p$$ the gradient, of course, has to be 0.