# Logistic regression loss function not zero for perfect model

The definition of the logistic regression loss function I use is this:

We draw the data i.i.d. according to some distribution $$D$$, realised by some $$\langle X, Y \rangle$$. Now if $$h_w$$ was the real data generating distribution, that is $$\mathbb{P}(Y = 1 \ | \ X = x) = h_w(x)$$, I'd expect the loss function to be zero, but I do not think this is the case. Is that a calculation mistake or is it unreasonable to ask the loss function to be zero for a perfect hypothesis? If so, why?

Calculation

$$\mathbb{E}[\log(1 + exp(-Y\langle w, X \rangle))] = \mathbb{E}[\mathbb{E}[\log(1 + exp(-Y\langle w, X \rangle))] \ | \ X = x]$$

$$= \int \log(1 + exp(-\langle w, x \rangle)) \cdot \frac{1}{1 + e^{-\langle w, x \rangle}} + \log(1 + exp(\langle w, x \rangle)) \cdot \frac{1}{1 + e^{\langle w, x \rangle}} d\mathbb{P}^X(x)$$

and this is not zero because the function we integrate is not zero almost everywhere.

• It will be zero with perfect foresight. Eg if throwing a fair die, model will predict 0.5 but actual outcomes will be (0,1,0,0,1) etc – seanv507 Feb 23 at 11:38
• @seanv507 So is there a mistake in my calculation (and if so, where?), or am I interpreting what it means to generate the data according to $h_w$ incorrectly? – Pel de Pinda Feb 23 at 15:15
• Just work through the case of a coin toss.x is 1, w is ?.there is a difference between accurately predicting the distribution of an event ( eg coin toss .5) and accurately predicting the event: on the next throw I get a head. – seanv507 Feb 23 at 20:49