# Loss function space for a linear binary classification model

How would the loss function look like for a linear binary classification model? Will it always be convex with a single extreme point?

I know that for nonlinear classification models there can be multiple local optimas, but can you assume that a linear classification model will only have 1 local optima (which is also the global optima) ? Or is this dependent of the loss function?

• What is a linear classification model, logit/probit-style GLM? I can see that, but those models are nonlinear (in some sense), so the terminology could be confusing.
– Dave
May 14, 2021 at 12:24
• @Dave I see, what I mean with linear is that the model does not handle correlation between input features. So even if the output function is nonlinear (like sigmoid or any other logistic function) it will still not handle the correlation. May 14, 2021 at 12:31
• What is an example of a mode that does not handle correlation between features? Logistic regression can be pretty good, even when there is much correlation.
– Dave
May 14, 2021 at 12:33
• I think classifying XOR can't be done by a linear model. XOR is probably the easiest example where there is correlation between input features. May 14, 2021 at 12:34
• So that’s nonlinear by your definition…what would be linear by your definition? (Your definition is totally nonstandard, but you mean something by it, and you can edit the question and title to express what you mean once we pin down what exactly that is.)
– Dave
May 14, 2021 at 12:36

This would depend on the loss function that is used. For example, in the case of logistic regression, where the loss function is the binary cross entropy function, if the conditional probability distribution is modeled by any distribution from the exponential family, then the binary cross entropy function will be convex in the parameters. More precisely, if the conditional probability distribution is modeled as $$q(c|\mathbf{x}) = \sigma(g(\mathbf{x};\mathbf{\theta}))$$ where $$c$$ is the class, $$\mathbf{x}$$ is the input feature vector, $$\sigma$$ is the logistic function, and $$g$$ is a chosen function parameterized by the parameters $$\theta$$, then the binary cross entropy function is convex in $$\theta$$ if $$g$$ is a linear function of $$\theta$$. Otherwise, if $$g$$ is a non-linear function of $$\theta$$, then the binary cross entropy function will be non-convex in $$\theta$$.