What's the relation between the output of a neural network and a Multinomial distribution?

Question

I am reading this paper, which has the following paragraph -

"The gold standard for deep neural nets is to use the softmax operator to convert the continuous activations of the output layer to class probabilities. The eventual model can be interpreted as a multinomial distribution whose parameters, hence discrete class probabilities, are determined by neural net outputs. For a K−class classification problem, the likelihood function for an observed tuple (x, y) is"

$P r(y|x, \theta) = \text{Mult}(y | \sigma(f_1(x, \theta)), \ldots, \sigma(f_K(x, \theta)))$

"where $\text{Mult}(\ldots)$ is a multinomial mass function, $f_j (x, θ)$ is the $j^{th}$ output channel of an arbitrary neural net $f(·)$ parametrized by $θ$, and $\sigma(u_j) = \frac{e^{u_j}}{\sum_{i=1}^{K} e^{u_K}}$ is the softmax function."

I don't why the above expression holds. In addition, aren't the parameters of a Multinomial distribution, $n$ and $k$ as per Wikipedia? Here $n$ is the number of trials and $k$ is the number of mutually exclusive events (integer).

Edit 1 (After an answer was accepted):

The reason why I do not understand this is that the lhs should ideally result in -

$P(y | x, \theta) = \arg\max_i (\sigma(f_i(x, \theta)))$

Whereas, I don't even know how to represent the rhs as a multinomial expression. The pmf of a multinomial expression with parameters $n$ and $k$ is given by -

$Pr(Y_1 = y_1 \text{ and } \dots Y_k = y_k|p_1 \text{ and } \dots p_k) = \frac{n!}{y_1!\cdots y_k!} p_1^{y_1} \cdots p_k^{y_k}$

All, I understand here is that $n = 1$. No idea, what to substitute in $p$, because $\sum_{j=1}^{k} x_j = n$.

Edit 2:

Corrected the typo in the Multinomial distribution above.

As suggested in the comments, let me start with a simple Bernoulli distribution. In that case, I will suppose my neural network only had a single output -

$p = \sigma(f(x, \theta))$

$q = 1 - p$

$P(y = 1|p, q) = \sigma(f(x, \theta))$

The binomial distribution case will be trivial, as the number of trials will always be one.

Now, let's do the Multinomial distribution. According to Wikipedia, the Multinomial distribution is defined as, "For n independent trials each of which leads to a success for exactly one of k categories, with each category having a given fixed success probability, the multinomial distribution gives the probability of any particular combination of numbers of successes for the various categories." Of course, here $n=1$ and $k=K$. Let's assume the first neuron of the softmax contains the maximum probability. Therefore,

$p = \sigma(f(x, \theta))$

$q = 1 - p$

$P(y_1 = 1, \cdots ,y_k = 0|p, q) = \sigma(f(x_1, \theta))$

Answer 1

If we take $n=1$ and $k=K$ and use the probabilities given by the softmax function, then the two definitions match.

As an example, consider $n=1$ and $K=2$. Then the outcome is $y=0$ or $y=1$. Let $p_0$ denote $\mathbb{P}(y=0)$ and $p_1$ denote $\mathbb{P}(y=1)$. By the Kolmogorov axioms, we know $p_0 + p_1 = 1$ and $p_0 \ge 0$ and $p_1 \ge 0$. When the $p_i$ are fixed and the trials are independent, this is a Bernoulli distribution.

In terms of your notation, we know that the $p_i$ are not fixed, but instead vary with the features $x$ and the parameters $\theta$. So we can write $$ p_i = \frac{\exp(f_i(x,\theta))}{\exp(f_0(x,\theta))+\exp(f_1(x,\theta))} $$ where the $f_i$ are the outputs of the neural network.

It's somewhat cumbersome, but we can write this in the form of a multinomial distribution. Most of the values that appear in the multinomial form can be simplified.

$$\begin{align} \mathbb P(y=0) &=\frac{1!}{(1-0)!0!} \left(\frac{\exp(f_0(x,\theta))}{\exp(f_0(x,\theta))+\exp(f_1(x,\theta))}\right)^1 \left(\frac{\exp(f_1(x,\theta))}{\exp(f_0(x,\theta))+\exp(f_1(x,\theta))}\right)^0 \\ &= \frac{\exp(f_0(x,\theta))}{\exp(f_0(x,\theta))+\exp(f_1(x,\theta))} \end{align}$$

and similarly for the case $\mathbb P(y=1)$. This example can be generalized to the multinomial case.

In all of these circumstances, the $p_i$ are the only thing that you are estimating (as functions of features and parameters). This model assumes that you know the number of draws $n$ and the number of classes $K$.