# Logistic regression, loss function and KL divergence [closed]

In decision theory, a loss function signature is supposed to be

output space * output space -> error


There seems to be many different definition of 'the logistic loss' on the web

• Some define it as 'the negative of the log likelihood' boyd : this is clearly not a loss function in the decision theory point of view

• Some define it as a function of R * [K ] -> R upen : same pb wih the domain

So would the correct decision theory loss be expressed in term of distribution ?

That is:

• given an x, I get back a distribution over the K cases, given by the softmax equation in the case of the logistic regression (output space = distributions)
• my loss function is the KL distance between this distribution and the observed one (signature = distributions * distribution -> R)

PS: R is for the real number and [K] is the discrete set of integer from 1 to K

• An explanation of your notation would help people understand what you're asking. What are "R" and "[K]"? Note, too, that there is no such thing as a "correct" loss: that is something adopted beforehand by the decision maker.
– whuber
Commented Feb 3, 2013 at 23:01
• indeed but there is a 'logistic loss' which you will find in the literature. (and it so happens that it in the legendre conjugate of the softmax function which yields to simple updates I think) Commented Feb 4, 2013 at 16:30
• What, then, is your question? Is it about what kind of mathematical object any loss function should be, or is it about what specifically a "logistic loss" is?
– whuber
Commented Feb 4, 2013 at 16:31
• the reported loss functions, for which I provide a link, can not be losses in the decision theory sense. hence the question with an interrogation point. Commented Feb 4, 2013 at 17:51