I am doing an introduction to ML with tensorflow and I came across softmax activation function. Why is in the softmax formula e? Why not 2? 3? 7?
$$ \text{softmax}(x)_i = \frac{\exp(x_i)}{\sum_j \exp(x_j)} $$
$$ \begin{eqnarray} \sum_j a^L_j & = & \frac{\sum_j e^{z^L_j}}{\sum_k e^{z^L_k}} = 1. \tag{79}\end{eqnarray} $$
Let $\mathbf{z} = \left(\ln a\right) \mathbf{y}$
Now observe that $e^{z_i} = a^{y_i}$ hence:
$$ \frac{e^{z_i}}{\sum_j e^{z_j}} = \frac{a^{y_i}}{\sum_j a^{y_j}}$$
Multiplying vector $\mathbf{y}$ by the natural logarithm of $a$ is equivalent to switching the softmax function to base $a$ instead of base $e$.
You often have a linear model inside the softmax function (eg. $z_i = \mathbf{x}' \mathbf{w}_i$). The $\mathbf{w}$ in $\mathbf{x}' \mathbf{w}$ can scale the data so allowing a different base wouldn't add any explanatory power. If the scaling can change, there's a sense in which different base $a$ are all equivalent models.
In exponential settings, $e$ is typically the most aesthetically beautiful, natural base to use: $\frac{d}{dx} e^x = e^x$. A lot of math can look prettier on the page when you use base $e$.
Some math becomes easier with $e$ as a base, that's why. Otherwise, consider this form of softmax: $\frac{e^{ax_i}}{\sum_j e^{ax_j}}$, which is equivalent to $\frac{b^{x_i}}{\sum_j b^{x_j}}$, where $b=e^a$.
Now, consider this function: $\sum_i\frac{e^{ax_i}}{\sum_j e^{ax_j}} x_i$. You can play with coefficient $a$ making the function less or more soft max.
When $a\to\infty$, it is $\max(x)$ because $\lim_{a\to\infty}\frac{e^{ax_i}}{\sum_j e^{ax_j}}=\mathrm{argmax}(x)$.
When $a=1$ it is $\mathrm{softmax}(x)\cdot x$ - a smoother version of max.
When $a=0$ it is as soft as it gets: a simple average $\frac 1 n \sum_i x_i$
This is indeed a somewhat arbitrary choice:
The choice of the softmax function seems somehow arbitrary as there are many other possible normalizing functions. It is thus unclear why the log-softmax loss would perform better than other loss alternatives.
Some potential reasons why this may be preferred over other normalizing functions:
$\DeclareMathOperator*{\argmax}{\arg\!\max}$
In the context of classification, we want to predict the most likely class $c^*$ that a feature vector $\mathbf x$ belongs to as $$c^* = \argmax_{i} P(C = c_i \mid \mathbf x)$$ where $c_1,\dots,c_N$ are the $N$ different classes. It is often convenient to re-write $P(C = c_i \mid \mathbf x)$ using Bayes' rule. Note that
\begin{align}
P(C = c_i \mid \mathbf x) &= \frac{p(\mathbf x \mid C = c_i)P(C = c_i)}{p(\mathbf x)} \\
&= \frac{p(\mathbf x \mid C = c_i)P(C = c_i)}{\sum_{j=1}^N p(\mathbf x \mid C = c_j) P(C = c_j)}
\end{align}
For any $a,b \in \mathbb R$, $$a^{\log_a(b)} = b$$ So,
\begin{align}
P(C = c_i \mid \mathbf x) &= \frac{p(\mathbf x \mid C = c_i)P(C = c_i)}{\sum_{j=1}^N p(\mathbf x \mid C = c_j) P(C = c_j)} \\
&= \frac{a^{\log_a\left(p(\mathbf x \mid C = c_i)P(C = c_i)\right)}}{\sum_{j=1}^N a^{\log_a\left(p(\mathbf x \mid C = c_j) P(C = c_j)\right)}}
\end{align}
Letting $a = e$ such that
\begin{align}
z_i &= \ln\left(p(\mathbf x \mid C = c_i)P(C = c_i)\right) \\
z_j &= \ln\left(p(\mathbf x \mid C = c_j)P(C = c_j)\right)
\end{align}
and $$\mathbf z = \begin{bmatrix} z_1 \\ \vdots \\ z_N\end{bmatrix}$$ we get
\begin{align}
P(C = c_i \mid \mathbf x) &= \frac{\exp\left(\ln\left(p(\mathbf x \mid C = c_i)P(C = c_i)\right)\right)}{\sum_{j=1}^N \exp\left(\ln\left(p(\mathbf x \mid C = c_j) P(C = c_j)\right)\right)} \\
&= \frac{\exp\left(z_i\right)}{\sum_{j=1}^N \exp\left(z_j\right)} \\
&= \text{softmax}(\mathbf z)_i
\end{align}
However, we could have chosen $a$ to be any other real number other than $e$. So, there isn't really anything special about $e$ except its nice properties.
