6
$\begingroup$

I am doing an introduction to ML with tensorflow and I came across softmax activation function. Why is in the softmax formula e? Why not 2? 3? 7?

$$ \text{softmax}(x)_i = \frac{\exp(x_i)}{\sum_j \exp(x_j)} $$

$$ \begin{eqnarray} \sum_j a^L_j & = & \frac{\sum_j e^{z^L_j}}{\sum_k e^{z^L_k}} = 1. \tag{79}\end{eqnarray} $$

Tensorflow tutorial

NN book

$\endgroup$
  • $\begingroup$ @Tim I think the answer to that question really doesn't get at the heart of the issue here. Usually you're not trying to interpret the softmax variables in the same way as you would with functions in econometrics. I thought it was more that it was easier to calculate the derivatives of softmax. $\endgroup$ – John Aug 6 '17 at 13:16
  • $\begingroup$ @Tim I understand why compounded interest limit yields 'e' but I am unable to transpose the reason for this connection into softmax. $\endgroup$ – Gillian Aug 6 '17 at 13:36
  • $\begingroup$ It seems it could be any arbitrary base and it would give us approximatelly (maybe even precisely?) correct image of what the distribution looks like. And working with irrational numbers such as 'e' surely slows down computation. $\endgroup$ – Gillian Aug 6 '17 at 13:44
  • 1
    $\begingroup$ How does this slow the computation? In any case you'd be dealing with floating-point numbers... $\endgroup$ – Tim Aug 6 '17 at 13:48
7
$\begingroup$

Using a different base is equivalent to scaling your data

Let $\mathbf{z} = \left(\ln a\right) \mathbf{y}$

Now observe that $e^{z_i} = a^{y_i}$ hence:

$$ \frac{e^{z_i}}{\sum_j e^{z_j}} = \frac{a^{y_i}}{\sum_j a^{y_j}}$$

You often have a linear model inside the softmax function (eg. $z_i = \mathbf{x}' \mathbf{w}_i$). The $\mathbf{w}$ in $\mathbf{x}' \mathbf{w}$ can scale the data so allowing a different base wouldn't add any explanatory power. If the scaling can change, there's a sense in which different base $a$ are all equivalent models.

So why base $e$?

In exponential settings, $e$ is typically the most aesthetically beautiful, natural base to use: $\frac{d}{dx} e^x = e^x$. A lot of math can look prettier on the page when you use base $e$.

$\endgroup$
  • $\begingroup$ Would the function be the same if e to the z was replaced with ln(z) ? $\endgroup$ – Jack Vial Nov 9 '17 at 17:05
  • $\begingroup$ @Jack I'm not sure I follow what you specifically had in mind? $\endgroup$ – Matthew Gunn Nov 9 '17 at 17:24
  • $\begingroup$ If e^x = ln(x). Can the softmax function be written using ln(x) instead of e^x? $\endgroup$ – Jack Vial Nov 9 '17 at 18:01
  • $\begingroup$ You could write the softmax as $\operatorname{Softmax}(\mathbf{x})_i = \frac{e^{x_i}}{\sum_{j=1}^k e^{x_j}} = \exp\left(\ln\left( \frac{e^{x_i}}{\sum_{j=1}^k e^{x_j}} \right) \right) = \exp\left( x_i - \ln\left( \sum_{j=1}^k e^{x_j} \right)\right)$. But other than that, I really don't know what you're looking for? $\endgroup$ – Matthew Gunn Nov 9 '17 at 18:42
  • 1
    $\begingroup$ @SebastianNielsen My point was that $f(x) = \frac{a^x}{a^x + 1} $ and $f(x) = \frac{e^{bx}}{e^{bx} + 1}$ are literally the same function for $b = \ln a$. Basic math: $e^{x \ln a} = e^{\ln a^x} = a^x$. $\endgroup$ – Matthew Gunn Oct 22 '18 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.