5
$\begingroup$

I am reading this paper, which has the following paragraph -

"The gold standard for deep neural nets is to use the softmax operator to convert the continuous activations of the output layer to class probabilities. The eventual model can be interpreted as a multinomial distribution whose parameters, hence discrete class probabilities, are determined by neural net outputs. For a K−class classification problem, the likelihood function for an observed tuple (x, y) is"

$P r(y|x, \theta) = \text{Mult}(y | \sigma(f_1(x, \theta)), \ldots, \sigma(f_K(x, \theta)))$

"where $\text{Mult}(\ldots)$ is a multinomial mass function, $f_j (x, θ)$ is the $j^{th}$ output channel of an arbitrary neural net $f(·)$ parametrized by $θ$, and $\sigma(u_j) = \frac{e^{u_j}}{\sum_{i=1}^{K} e^{u_K}}$ is the softmax function."

I don't why the above expression holds. In addition, aren't the parameters of a Multinomial distribution, $n$ and $k$ as per Wikipedia? Here $n$ is the number of trials and $k$ is the number of mutually exclusive events (integer).

Edit 1 (After an answer was accepted):

The reason why I do not understand this is that the lhs should ideally result in -

$P(y | x, \theta) = \arg\max_i (\sigma(f_i(x, \theta)))$

Whereas, I don't even know how to represent the rhs as a multinomial expression. The pmf of a multinomial expression with parameters $n$ and $k$ is given by -

$Pr(Y_1 = y_1 \text{ and } \dots Y_k = y_k|p_1 \text{ and } \dots p_k) = \frac{n!}{y_1!\cdots y_k!} p_1^{y_1} \cdots p_k^{y_k}$

All, I understand here is that $n = 1$. No idea, what to substitute in $p$, because $\sum_{j=1}^{k} x_j = n$.

Edit 2:

Corrected the typo in the Multinomial distribution above.

As suggested in the comments, let me start with a simple Bernoulli distribution. In that case, I will suppose my neural network only had a single output -

$p = \sigma(f(x, \theta))$

$q = 1 - p$

$P(y = 1|p, q) = \sigma(f(x, \theta))$

The binomial distribution case will be trivial, as the number of trials will always be one.

Now, let's do the Multinomial distribution. According to Wikipedia, the Multinomial distribution is defined as, "For n independent trials each of which leads to a success for exactly one of k categories, with each category having a given fixed success probability, the multinomial distribution gives the probability of any particular combination of numbers of successes for the various categories." Of course, here $n=1$ and $k=K$. Let's assume the first neuron of the softmax contains the maximum probability. Therefore,

$p = \sigma(f(x, \theta))$

$q = 1 - p$

$P(y_1 = 1, \cdots ,y_k = 0|p, q) = \sigma(f(x_1, \theta))$

$\endgroup$
10
  • 1
    $\begingroup$ Because this question is about the meaning of symbols, You’ll need to explain what your symbols mean to get a useful answer $\endgroup$
    – Sycorax
    Nov 7, 2023 at 21:45
  • $\begingroup$ @Sycorax The changes have been made. Please let me know if further clarification is required. $\endgroup$ Nov 7, 2023 at 22:13
  • $\begingroup$ It’s not clear what you mean when you say the parameters of a multinomial distribution are n and k. This seems to be the core of the question, so we need to understand what n and k mean to you. $\endgroup$
    – Sycorax
    Nov 7, 2023 at 22:25
  • 1
    $\begingroup$ One thing to watch out for is that you're using $x$ to denote features in the first expression, and then using $x$ to denote the event $y$ in the addendum at the end. So a first step is to use consistent notation. Finally, the $p_i$ are also parameters of a multinomial distribution. $\endgroup$
    – Sycorax
    Nov 8, 2023 at 14:54
  • 1
    $\begingroup$ I've worked the Bernoulli example that I alluded to. If you have additional questions about neural networks or probability distributions, it's best to ask in a new thread. I also recommend reading a high-quality textbook on the topic, such as Goodfellow's Deep Learning. $\endgroup$
    – Sycorax
    Nov 10, 2023 at 21:31

1 Answer 1

6
$\begingroup$

If we take $n=1$ and $k=K$ and use the probabilities given by the softmax function, then the two definitions match.

As an example, consider $n=1$ and $K=2$. Then the outcome is $y=0$ or $y=1$. Let $p_0$ denote $\mathbb{P}(y=0)$ and $p_1$ denote $\mathbb{P}(y=1)$. By the Kolmogorov axioms, we know $p_0 + p_1 = 1$ and $p_0 \ge 0$ and $p_1 \ge 0$. When the $p_i$ are fixed and the trials are independent, this is a Bernoulli distribution.

In terms of your notation, we know that the $p_i$ are not fixed, but instead vary with the features $x$ and the parameters $\theta$. So we can write $$ p_i = \frac{\exp(f_i(x,\theta))}{\exp(f_0(x,\theta))+\exp(f_1(x,\theta))} $$ where the $f_i$ are the outputs of the neural network.

It's somewhat cumbersome, but we can write this in the form of a multinomial distribution. Most of the values that appear in the multinomial form can be simplified.

$$\begin{align} \mathbb P(y=0) &=\frac{1!}{(1-0)!0!} \left(\frac{\exp(f_0(x,\theta))}{\exp(f_0(x,\theta))+\exp(f_1(x,\theta))}\right)^1 \left(\frac{\exp(f_1(x,\theta))}{\exp(f_0(x,\theta))+\exp(f_1(x,\theta))}\right)^0 \\ &= \frac{\exp(f_0(x,\theta))}{\exp(f_0(x,\theta))+\exp(f_1(x,\theta))} \end{align}$$

and similarly for the case $\mathbb P(y=1)$. This example can be generalized to the multinomial case.

In all of these circumstances, the $p_i$ are the only thing that you are estimating (as functions of features and parameters). This model assumes that you know the number of draws $n$ and the number of classes $K$.

$\endgroup$
5
  • $\begingroup$ Isn't the lhs a distribution whereas the rhs a number? $\endgroup$ Nov 8, 2023 at 0:02
  • 1
    $\begingroup$ A probability distribution assigns numbers (probabilities) to outcomes. $\endgroup$
    – Sycorax
    Nov 8, 2023 at 0:09
  • $\begingroup$ Got it. Also, isn't $y|x \sim Mult(n, K)$, a modeling assumption rather a fact? I think the author is assuming that the data labels given the features is drawn from a multinomial distribution. $\endgroup$ Nov 8, 2023 at 0:53
  • 1
    $\begingroup$ Yes, this is an assumption. There are numerous alternatives ways to write down losses for categorical outcomes, but this one is the "gold standard," according to the authors. $\endgroup$
    – Sycorax
    Nov 8, 2023 at 1:40
  • $\begingroup$ Sorry, I thought over the question more during the night and found that I had some more doubts. I added an edit but I can make a new question if you'd like. $\endgroup$ Nov 8, 2023 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.