0
$\begingroup$

How would the loss function look like for a linear binary classification model? Will it always be convex with a single extreme point?

I know that for nonlinear classification models there can be multiple local optimas, but can you assume that a linear classification model will only have 1 local optima (which is also the global optima) ? Or is this dependent of the loss function?

$\endgroup$
6
  • $\begingroup$ What is a linear classification model, logit/probit-style GLM? I can see that, but those models are nonlinear (in some sense), so the terminology could be confusing. $\endgroup$
    – Dave
    May 14, 2021 at 12:24
  • $\begingroup$ @Dave I see, what I mean with linear is that the model does not handle correlation between input features. So even if the output function is nonlinear (like sigmoid or any other logistic function) it will still not handle the correlation. $\endgroup$
    – Kevin
    May 14, 2021 at 12:31
  • $\begingroup$ What is an example of a mode that does not handle correlation between features? Logistic regression can be pretty good, even when there is much correlation. $\endgroup$
    – Dave
    May 14, 2021 at 12:33
  • $\begingroup$ I think classifying XOR can't be done by a linear model. XOR is probably the easiest example where there is correlation between input features. $\endgroup$
    – Kevin
    May 14, 2021 at 12:34
  • $\begingroup$ So that’s nonlinear by your definition…what would be linear by your definition? (Your definition is totally nonstandard, but you mean something by it, and you can edit the question and title to express what you mean once we pin down what exactly that is.) $\endgroup$
    – Dave
    May 14, 2021 at 12:36

1 Answer 1

1
$\begingroup$

This would depend on the loss function that is used. For example, in the case of logistic regression, where the loss function is the binary cross entropy function, if the conditional probability distribution is modeled by any distribution from the exponential family, then the binary cross entropy function will be convex in the parameters. More precisely, if the conditional probability distribution is modeled as $$ q(c|\mathbf{x}) = \sigma(g(\mathbf{x};\mathbf{\theta})) $$ where $c$ is the class, $\mathbf{x}$ is the input feature vector, $\sigma$ is the logistic function, and $g$ is a chosen function parameterized by the parameters $\theta$, then the binary cross entropy function is convex in $\theta$ if $g$ is a linear function of $\theta$. Otherwise, if $g$ is a non-linear function of $\theta$, then the binary cross entropy function will be non-convex in $\theta$.

$\endgroup$
3
  • $\begingroup$ Thanks, but is not safe to assume this convex characteristic for any loss functions for a linear model? The motivation behind my question is from reading an article where it states "FGSM is exactly the optimal attack against a linear binary classification model" and I'm split on how the loss function must look like for this assumption to hold. $\endgroup$
    – Kevin
    May 14, 2021 at 12:14
  • $\begingroup$ Surely, the loss function must have 1 extreme point for a projected gradient ascent step (FGSM) to be optimal against a linear model. $\endgroup$
    – Kevin
    May 14, 2021 at 12:15
  • $\begingroup$ Loss functions are derived from some parameter estimation technique. In the case of the binary cross entropy function, the parameters that minimize it are the same parameters that maximize the likelihood function of the logistic model. Similarly, the parameters that minimize the mean squared-error also maximize the likelihood function of a gaussian model. In the case of these two loss functions, my answer still applies. $\endgroup$
    – mhdadk
    May 14, 2021 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.