0
$\begingroup$

Im reading this paper: Uncertainty in Deep Learning and in it (page4), the softmax loss is defined as

\begin{align*} E(X,Y) = -\frac{1}{N} \sum^N_{n=1} \log(\hat{p}_{n,c_n}), \end{align*}

where $c_n \in \{1,...,D\}$ is the class label for input $n$, and

$\hat{p}_{nd} = \exp(\hat{y}_{nd})/(\sum_{d'} \exp(\hat{y}_{nd'})$ is the element-wise softmax-function applied to model prediction vector $\hat{y}$.

Where does the class label $d$ actually show up in the softmax calculation? I.e. what does $\hat{y}_{nd}$ or $\hat{p}_{n,c_n}$ respectively shall mean?

Thanks

$\endgroup$
9
  • $\begingroup$ The $y_i$ is either 0 or 1. $\endgroup$ May 24, 2020 at 10:44
  • $\begingroup$ yes Sir, thats correct for (binary) classification $\endgroup$
    – MJimitater
    May 24, 2020 at 10:52
  • $\begingroup$ There are $D$ classes. $\hat{y}_{nd}$ for $d \in \{1,\dots,D\}$ are the probabilities for the $n$th sampling unit, and $\hat{y}_{n,c_n}$ is the largest probability (of the $D$ probabilities). $\endgroup$
    – papgeo
    May 24, 2020 at 13:09
  • $\begingroup$ Thanks @papgeo for your answer. Your explaination of $\hat{y}_{nd}$ makes sense, thank you! And you mean $\hat{p}_{n,c_n}$ to be the largest probability? $\endgroup$
    – MJimitater
    May 24, 2020 at 15:21
  • $\begingroup$ If you formulate those insights into an answer, Im willing to accept that. $\endgroup$
    – MJimitater
    May 24, 2020 at 15:22

1 Answer 1

1
$\begingroup$

We are given $N$ input-output pairs: $\{(\textbf{x}_1,y_1),\dots,(\textbf{x}_N,y_N)\}$. The $\textbf{x}$'s are input vectors, also known as covariates or predictors, and the $y$'s are the labels. The problem here is to learn how to predict a class label given the vector of inputs $\textbf{x}$.

$\hat{y}_{nd}$ is what brings the $\textbf{x}$'s into the computations. It is also known as the linear predictor and it is given by $\hat{y}_{nd} = \beta_{0d} + \beta_{1d} x_{1n} + \dots + \beta_{pd} x_{pn}$, where the $\beta$'s are regression coefficients (to be estimated) and $x_{1n}, \dots, x_{pn}$ are the elements of the input vector $\textbf{x}_n$.

$\hat{p}_{n,c_n}$ is the probability of the class that sampling unit $n$ belongs to.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.