Suppose you had a classification problem where you are trying to predict a class label (e.g., $[0 \: 1 \: 0]^T$) with a model. One way to do this is to use log loss:

$\Large \ell_{\log} = -\sum_i[y_i\log \hat{y}_i + (1-y_i)\log (1-\hat{y}_i)]$

This is attractive because it does the right thing: it pushes $\hat{y}$ to $\inf$ when $y_i$ is $1$, and to $- \inf$ when $y_i$ is zero. But another way to do this is with elementwise division:

$\Large \ell_{\text{div}} = \sum_i[\frac{\hat{y}_i}{\max (y_i, \epsilon)} + \frac{y_i}{\max (\hat{y_i}, \epsilon)}]$

Here, the minimum of the function is attained when $y$ matches $x$ on all dimensions. Isn't this a preferable cost function? Why isn't it used?

Note: $\epsilon$ is a small positive constant to prevent division by zero.

EDIT: My question isn't only about $0$ vs $1$ classification. I am also interested in situations where $y$ has real-valued entries, and we are interested in generating a vector $\hat{y}$ that matches it, in the sense that the two are "close together" in some sense. For classification problems, $\log$ is the way to go; but what about real-valued vectors?

  • $\begingroup$ The prediction $\hat{y}_i$ is determined through a function of the covariates $x_i$. For example, in logistic regression, that function ensures that $\hat{y}_i \in [0,1]$ so cannot be ``pushed to $-\infty$ or $\infty$''. This similarly holds in other scenarios. $\endgroup$
    – user257566
    Commented Jun 28, 2021 at 21:09
  • 3
    $\begingroup$ Not sure how you invented this function; what makes it preferable? It's problematic because of numerical stability in that division, for one thing. // Also - the original loss function, the log loss, isn't useful for multi class classification. $\endgroup$ Commented Jun 28, 2021 at 23:46
  • 2
    $\begingroup$ Please don't re-post a question that was closed! It was closed for a reason. Instead, take some time to edit the original question to provide the requested details/clarity. This automatically nominates it for reopening. $\endgroup$ Commented Jun 29, 2021 at 0:02
  • $\begingroup$ Your example class label has 3 components, but the expression you've written for cross-entropy is used for binary targets. $\endgroup$
    – Sycorax
    Commented Jun 29, 2021 at 1:12
  • $\begingroup$ @AryaMcCarthy: the log loss is indeed useful for multi-class classification, it's the log score. If you have classes $1, \dots, n$ and predicted class membership probabilities of $\hat{p}_1, \dots, \hat{p}_n$ (summing to $1$), and if the actual class of the instance turns out to be $i$, then the score is $\pm\log\hat{p}_i$ (with $\pm$ depending on whether you want a positively oriented score or not). Note how this just turns into the formula above for the 2-class case. Compare the tag wiki and references therein. $\endgroup$ Commented Jun 29, 2021 at 5:52


Browse other questions tagged or ask your own question.