# When is logistic regression Bayes-optimal?

In logistic regression, we model the posterior probability $P(y=1 | x)$ with the help of a sigmoidal function: $$P(y=1 | x) = \frac{1}{1+\exp(-x)} = h(x)$$

If we classify a datapoint to the class y = 1 if $h(x) > 0.5$, is then our classification Bayes-optimal, since it chooses to classify according to the higher posterior class probability? Is this connected to the fact that the minimizer of log loss is $\ln\frac{h(x)}{1-h(x)}?$

• I do not understand the question. Let us consider $x \in \mathbb{R}^2$. Take any function $g(y,x)$ such that $g(0,x) + g(1,x) = 1$ and any arbitrary density $f_X$ for $x$ then $(x,y) \mapsto g(y,x) f_X(x)$ defines a common density for which the conditional density $f_{Y|X}(y|x)$ is precisely $g(y,x)$. Taking $h(x) = g(1,x)$ and a predictor $p(x) = 1$ iff. $h(x) > 0.5$ always yields a Bayes optimal classifier (see win.tue.nl/~rmcastro/2DI70/files/2DI70_Lecture_Notes.pdf p. 16). So: what restrictions do you expect? The answer is just: if $f_{Y|X}$ really is the sigmoidal function... – Fabian Werner Feb 13 '18 at 15:10
• then the logistic regression is Bayes optimal. – Fabian Werner Feb 13 '18 at 15:10

Assume your data obbeys an unknown perfect law $P(y=1|x)=f(x)$. Then the Bayes optimal classifier is "classify y=1 when $f(x)>0.5$". This is true for any law and not related to anything algebraic. In practice you don't know $f$ and you can't, so that the Bayes-optimal classifier is only a theoretical object.
Now, imagine you don't know $f$ but you know that $f(x)=logit^{-1}(\beta x)$ and only ignore $\beta$. This happens only in simulations where you control the underlying true law and hide $\beta$. You estimate it as $\hat\beta$ and you say "classify y=1 when $logit^{-1}(\hat\beta x)>0.5$". This is not Bayes optimal since don't have the exact $\beta$. It is asymptotically Bayes optimal since with infinite training data $\hat\beta=\beta$.
There is a case where you can measure this: simulate data with an $f$ that is not logistic and see how good the logistic approximation is. This is not a real situation though.
• What would happen if the simulated $f$ was Gaussian ? Are the two related ? – Xavier Bourret Sicotte Jul 5 '18 at 11:58