In the wiki, the softmax function is defined as the gradient-log-normalizer of the categorical probability distribution. A partial explanation to the log-normalizer is found here, but what does gradient-log-normalizer stand for?


Using notation from the wikipedia page (https://en.wikipedia.org/wiki/Exponential_family), an exponential family is a family of probability distributions that have pmfs/pdfs that can be written as (noting that $\theta$, $x$ can be vector valued): $$f_{\theta}(x)=h(x)\exp[\eta(\theta)^Tt(x)-A(\theta)]$$ where $\eta(\theta)=\eta$ are the natural parameters, $t(x)$ are the sufficient statistics, and $A(\theta)$ is the log normalizer (sometimes called the log partition function). The reason $A(\theta)$ is called the log normalizer, as it can be verified that, in the continuous case, for this to be a valid pdf, we must have $$A(\theta)=\log\left[\int h(x)\exp[\eta(\theta)^Tt(x)]dx\right],$$ and in the discrete case, for this to be a valid pmf, we must have $$A(\theta)=\log\left[\sum_x h(x)\exp[\eta(\theta)^Tt(x)]\right].$$ In each case we notice that $\int h(x)\exp[\eta(\theta)^Tt(x)]dx$ and $\sum_x h(x)\exp[\eta(\theta)^Tt(x)]$ are the normalization constants of the distributions, hence the name log normalizer.

Now to see the specific relationship between the softmax function and the $k$ dimensional categorical distribution, we'll have to use a specific parameterization of the distribution. Namely, let $\theta_1,\cdots,\theta_{k-1}$ be such that $0<\theta_1,\cdots,\theta_{k-1}$ and $\sum_{i=1}^{k-1}\theta_i<1$, and define $\theta_k=1-\sum_{i=1}^{k-1}\theta_i$ (letting $\theta=(\theta_1,\cdots,\theta_{k})$). The pmf for this distribution is (letting $x=(x_1,\cdots,x_{k})$ be a one hot vector, i.e. $x_i=1$ and $x_j=0$ for $i\neq j$): $$f_{\theta}(x)=\prod_{i=1}^k\theta_i^{x_i}.$$ To write this as an exponential family, note that $h(x)=1$, $\eta(\theta)=(\log[\theta_1/\theta_k],\cdots, \log[\theta_{k-1}/\theta_k],0)$, $t(x)=(x_1,\cdots,x_{k})$, and $A(\theta)=-\log[\theta_k]$, so: $$f_{\theta}(x)=\exp[(\log[\theta_1/\theta_k],\cdots, \log[\theta_{k-1}/\theta_k],0)^T(x_1,\cdots,x_{k})-(-\log[\theta_k])].$$

Now let's suggestively write $\eta(\theta_i)=\log[\theta_i/\theta_k]=\eta_i$, so that we can write $\theta_i=\frac{e^{\eta_i}}{\sum_{j=1}^ke^{\eta_j}}$. Then the log normalizer becomes $$A(\eta)=-\log\left[\frac{e^{\eta_k}}{\sum_{j=1}^ke^{\eta_j}}\right]= -\log\left[\frac{1}{\sum_{j=1}^ke^{\eta_j}}\right]=\log\left[\sum_{j=1}^ke^{\eta_j}\right].$$ Taking the partial derivative with respect to $\eta_i$, we find $$\frac{\partial}{\partial \eta_i}A(\eta)=\frac{e^{\eta_i}}{\sum_{j=1}^ke^{\eta_j}},$$ revealing that the gradient of the log normalizer is indeed the softmax function: $$\nabla A(\eta)=\left[\frac{e^{\eta_1}}{\sum_{j=1}^ke^{\eta_j}},\cdots,\frac{e^{\eta_k}}{\sum_{j=1}^ke^{\eta_j}}\right].$$

  • $\begingroup$ Wow!! That was a great explanation and has totally sense. Thank you :) $\endgroup$
    – tashuhka
    Jan 2 '18 at 13:11
  • 1
    $\begingroup$ I've been looking for this derivation for a long time! I'm wondering, in what context did you have to develop this knowledge? Did you see this as part of a course or textbook? I kept finding references to this relationship on the internet but no one actually gave the details. $\endgroup$ Sep 16 '18 at 16:52
  • 2
    $\begingroup$ @zipzapboing I actually didn't know this property of the softmax until I saw OP's question! However I did have a casella and berger level stats course (where exponential families and some of their other properties are introduced) under my belt, which allowed me to know that proving the property wouldn't be that hard with the right parameterization. $\endgroup$
    – aleshing
    Sep 16 '18 at 17:36
  • 1
    $\begingroup$ @zipzapboing: in addition to aleshing's comment, see also exercise 3.32(a) of casella and berger's text "stastical inference" [2nd ed]. chapter 3 contains helpful context (imho) $\endgroup$ Jul 26 '21 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.