# How is softmax unit derived and what is the implication?

I'm trying to understand why the softmax function is defined as such:

$\frac{e^{z_{j}}} {\Sigma^{K}_{k=1}{e^{z_{k}}}} = \sigma(z)$

I understand how this normalizes the data and properly maps to some range (0, 1) but the different between weight probabilities varies exponentially rather than linearly. Is there a reason why we want this behaviour?

Also this equation seems rather arbitrary and I feel that it a large family of equations could satisfy our requirements. I have not seen any derivations online so I'm assuming it is merely a definition. Why not choose any other definition that satisfies the same requirements?

• You might want to Google logistic regression and multinomial regression – seanv507 Apr 8 '15 at 7:28
• Also, search this site! – kjetil b halvorsen May 4 '15 at 14:19

The expectation parameters are the expected values of $x_k$ where $x_k$ are the number of times you observed outcome $k$. This is the right parametrization for converting a set of observations to a maximum likelihood distribution. You simply average in this space. This is what you want when you are modeling observations.