Softmax regression: Intuition about why distribution of $y$ is in terms of $e^{\theta^Tx}$ as opposed to just $\theta^Tx$

Question

I'm going through Andrew Ng's lecture notes on Machine Learning and I just learnt about softmax regression there.

We see that, for softmax regression, the conditional distribution of $y$ given $x$ is given as:

This formula contains terms of form $e^{\theta^Tx}$. I was just wondering if there is an intuitive explanation for this? Or, why isn't the derived formula for probability simpler like:

$$\frac{\theta^Tx}{\sum_j\theta_{j}^Tx}$$

And is there an intuitive explanation for what that would mean?

Answer 1

You need power to get rid of negative values. When you raise positive number to the power - you will always get positive value. For negative power - the result is just small and for positive - it's big and grows exponentially.

By using softmax you will never get negative probability nor probability higher then 1 and will never divide by zero when calculating it

$$\frac{ e^{\Theta_{i}^T x }}{ \sum_{j} e^{\Theta_{j}^T x } }$$

Answer 2

There is an intuitive definition. I tried to explain the softmax in this answer. To put it simply, you are interpreting the unbounded $\theta_i^Tx$ as log-odds, and the softmax converts them to probabilities in $[0,1]$. Your formula has no such interpretation.