4
$\begingroup$

I'm going through Andrew Ng's lecture notes on Machine Learning and I just learnt about softmax regression there.

We see that, for softmax regression, the conditional distribution of $y$ given $x$ is given as:

This formula contains terms of form $e^{\theta^Tx}$. I was just wondering if there is an intuitive explanation for this? Or, why isn't the derived formula for probability simpler like:

$$\frac{\theta^Tx}{\sum_j\theta_{j}^Tx}$$

And is there an intuitive explanation for what that would mean?

$\endgroup$
2
$\begingroup$

You need power to get rid of negative values. When you raise positive number to the power - you will always get positive value. For negative power - the result is just small and for positive - it's big and grows exponentially.

By using softmax you will never get negative probability nor probability higher then 1 and will never divide by zero when calculating it

$$\frac{ e^{\Theta_{i}^T x }}{ \sum_{j} e^{\Theta_{j}^T x } }$$

$\endgroup$
  • $\begingroup$ And why using e instead of arbitary number > 1? $\endgroup$ – mrgloom Sep 1 '17 at 15:42
  • 1
    $\begingroup$ It makes taking derivative simpler. $\endgroup$ – Serhiy Sep 3 '17 at 17:57
1
$\begingroup$

There is an intuitive definition. I tried to explain the softmax in this answer. To put it simply, you are interpreting the unbounded $\theta_i^Tx$ as log-odds, and the softmax converts them to probabilities in $[0,1]$. Your formula has no such interpretation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.