What is the "standard" function one minimizes to estimate the parameters for logicistic regression? What is implemented in R?

I thought it was the squared error but a machine learning course I am following suggests a loss function of the type:

$ -\log h_\theta (x)$ for $y =1$

$ -\log (1- h_\theta (x))$ for $y = 0$

$h_\theta(x) = (1+ e^{-\theta^Tx})^{-1} $

This would make the cost function convex.

Is this this estimator same as MLE?

What about for other non-linear regressions such as probit model?

  • $\begingroup$ Logistic regression minimizes cross-entropy $\endgroup$ Commented Feb 25, 2017 at 8:47
  • $\begingroup$ @ŁukaszGrad is this the same as the MLE? $\endgroup$
    – Lost1
    Commented Feb 25, 2017 at 9:21
  • $\begingroup$ Yes, minimizing the cross entropy between the empirical distribution and the model (i.e. cross entropy loss) is equivalent to minimizing the negative log likelihood (i.e. performing maximum likelihood estimation). Other generalized linear models (e.g. probit) can be fit similarly to logistic regression, by maximizing the likelihood. $\endgroup$
    – user20160
    Commented Feb 25, 2017 at 12:19

1 Answer 1


Logistic Regression does not use the squared error as loss function, since the following error function is non-convex:

$J(\theta) = \sum \left(y^{(i)}-(1+ e^{-\theta^Tx^{(i)}})^{-1}\right)^2$

where, $(x^{(i)},y^{(i)})$ represents the $i$th training sample. (As you know, Logistic Regression uses $h_\theta(x)=(1+e^{-\theta^Tx})^{-1}$ as the hypothesis function, which gives the probability of $y=1$.)

Instead of squared error, it uses the negative log-likelihood ($-\log p(D|\theta)$) as the loss function, which is convex. Now, since

$-\log p(D|\theta)=\sum -\log p(y^{(i)} | x^{(i)},\theta)$


$p(y|x,\theta)=h_\theta(x)\space\space if \space y=1$

$p(y|x,\theta)=1-h_\theta(x) \space \space if \space y=0$,

it is easy to see the loss function mentioned in the course you are following.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.