The origin of logistic regression is actually logistic curve which varies from the value 0 to the value 1. It looks like the letter S, and it specifies the growth of species.

If our data distribution is in the form S, we can use this logistic curve to represent our data (regression).

However, logistic curve can also be used in classification. Since it takes the values between zero and one, it is usable in also classification.

My confusion starts in this point. Why do we use different loss(cost) functions in logistic regression and logistic regression classification ?

$Loss = L(w) = \sum_{n=1}^N (\sigma(x_n^\top w + b) - y_n)^2$

$Loss = L(w) = \sum_{n=1}^N -y_n\log(\sigma(x_n^\top w + b))- (1-y_n)\log(1 - \sigma(x_n^\top w + b))$

The first loss function specified above is used for logistic regression. The second one is for logistic regression classification.

Actually, In both problems, we tune logistic curve perfectly to minimize loss function. Hence, we can use first loss function, in which least square is used, for also classification problem.

I tried to justify this idea, and I took a reply from someone. In the reply, the following explanation is written:

First loss function is not used for classification, since first loss function depends on least square technique. Least square technique is used to minimize distance between ground truth and predicted labels. Minimizing this distance enables us to represent our data perfectly, which refers to regression. However, we do not want to perform regression. We want to perform classification. For classification, loss function must be more sharp, which means that it must make the mistake more noticeable. Hence, logarithm is used in loss function of logistic regression classification.

What do you think about this explanation. Is it true ? Is there any other perspective which explains why we do not use first loss function in logistic regression classification ?

  • $\begingroup$ See also stats.stackexchange.com/questions/127042/… $\endgroup$ – kjetil b halvorsen Jun 14 '19 at 12:27
  • $\begingroup$ We don't minimize squared error in logistic regression, in both cases (they are the same case) we use same loss function. $\endgroup$ – Tim Jun 14 '19 at 13:13
  • $\begingroup$ The logarithm is employed in the maximum likelihood function because it's a multiplicative model, essentially, $y=\exp(\boldsymbol{\beta}^\top \mathbf{x})$, of course without the logit terminology used. $\endgroup$ – wrktsj Jun 14 '19 at 14:04