In decision theory, a loss function signature is supposed to be

output space * output space -> error

There seems to be many different definition of 'the logistic loss' on the web

  • Some define it as 'the negative of the log likelihood' boyd : this is clearly not a loss function in the decision theory point of view

  • Some define it as a function of R * [K ] -> R upen : same pb wih the domain

So would the correct decision theory loss be expressed in term of distribution ?

That is:

  • given an x, I get back a distribution over the K cases, given by the softmax equation in the case of the logistic regression (output space = distributions)
  • my loss function is the KL distance between this distribution and the observed one (signature = distributions * distribution -> R)

PS: R is for the real number and [K] is the discrete set of integer from 1 to K

  • $\begingroup$ An explanation of your notation would help people understand what you're asking. What are "R" and "[K]"? Note, too, that there is no such thing as a "correct" loss: that is something adopted beforehand by the decision maker. $\endgroup$
    – whuber
    Commented Feb 3, 2013 at 23:01
  • $\begingroup$ indeed but there is a 'logistic loss' which you will find in the literature. (and it so happens that it in the legendre conjugate of the softmax function which yields to simple updates I think) $\endgroup$
    – nicolas
    Commented Feb 4, 2013 at 16:30
  • $\begingroup$ What, then, is your question? Is it about what kind of mathematical object any loss function should be, or is it about what specifically a "logistic loss" is? $\endgroup$
    – whuber
    Commented Feb 4, 2013 at 16:31
  • $\begingroup$ the reported loss functions, for which I provide a link, can not be losses in the decision theory sense. hence the question with an interrogation point. $\endgroup$
    – nicolas
    Commented Feb 4, 2013 at 17:51


Browse other questions tagged or ask your own question.