The definition of the logistic regression loss function I use is this:

enter image description here

We draw the data i.i.d. according to some distribution $D$, realised by some $\langle X, Y \rangle$. Now if $h_w$ was the real data generating distribution, that is $\mathbb{P}(Y = 1 \ | \ X = x) = h_w(x)$, I'd expect the loss function to be zero, but I do not think this is the case. Is that a calculation mistake or is it unreasonable to ask the loss function to be zero for a perfect hypothesis? If so, why?


$$\mathbb{E}[\log(1 + exp(-Y\langle w, X \rangle))] = \mathbb{E}[\mathbb{E}[\log(1 + exp(-Y\langle w, X \rangle))] \ | \ X = x]$$

$$= \int \log(1 + exp(-\langle w, x \rangle)) \cdot \frac{1}{1 + e^{-\langle w, x \rangle}} + \log(1 + exp(\langle w, x \rangle)) \cdot \frac{1}{1 + e^{\langle w, x \rangle}} d\mathbb{P}^X(x)$$

and this is not zero because the function we integrate is not zero almost everywhere.

  • $\begingroup$ It will be zero with perfect foresight. Eg if throwing a fair die, model will predict 0.5 but actual outcomes will be (0,1,0,0,1) etc $\endgroup$ – seanv507 Feb 23 at 11:38
  • $\begingroup$ @seanv507 So is there a mistake in my calculation (and if so, where?), or am I interpreting what it means to generate the data according to $h_w$ incorrectly? $\endgroup$ – Pel de Pinda Feb 23 at 15:15
  • $\begingroup$ Just work through the case of a coin toss.x is 1, w is ?.there is a difference between accurately predicting the distribution of an event ( eg coin toss .5) and accurately predicting the event: on the next throw I get a head. $\endgroup$ – seanv507 Feb 23 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.