1
$\begingroup$

What is the "standard" function one minimizes to estimate the parameters for logicistic regression? What is implemented in R?

I thought it was the squared error but a machine learning course I am following suggests a loss function of the type:

$ -\log h_\theta (x)$ for $y =1$

$ -\log (1- h_\theta (x))$ for $y = 0$

$h_\theta(x) = (1+ e^{-\theta^Tx})^{-1} $

This would make the cost function convex.

Is this this estimator same as MLE?

What about for other non-linear regressions such as probit model?

$\endgroup$
  • $\begingroup$ Logistic regression minimizes cross-entropy $\endgroup$ – Łukasz Grad Feb 25 '17 at 8:47
  • $\begingroup$ @ŁukaszGrad is this the same as the MLE? $\endgroup$ – Lost1 Feb 25 '17 at 9:21
  • $\begingroup$ Yes, minimizing the cross entropy between the empirical distribution and the model (i.e. cross entropy loss) is equivalent to minimizing the negative log likelihood (i.e. performing maximum likelihood estimation). Other generalized linear models (e.g. probit) can be fit similarly to logistic regression, by maximizing the likelihood. $\endgroup$ – user20160 Feb 25 '17 at 12:19
1
$\begingroup$

Logistic Regression does not use the squared error as loss function, since the following error function is non-convex:

$J(\theta) = \sum \left(y^{(i)}-(1+ e^{-\theta^Tx^{(i)}})^{-1}\right)^2$

where, $(x^{(i)},y^{(i)})$ represents the $i$th training sample. (As you know, Logistic Regression uses $h_\theta(x)=(1+e^{-\theta^Tx})^{-1}$ as the hypothesis function, which gives the probability of $y=1$.)

Instead of squared error, it uses the negative log-likelihood ($-\log p(D|\theta)$) as the loss function, which is convex. Now, since

$-\log p(D|\theta)=\sum -\log p(y^{(i)} | x^{(i)},\theta)$

and

$p(y|x,\theta)=h_\theta(x)\space\space if \space y=1$

$p(y|x,\theta)=1-h_\theta(x) \space \space if \space y=0$,

it is easy to see the loss function mentioned in the course you are following.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.