Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This question already has an answer here:

I want to find the parameters of a logistic function. I read the guide here. It has a very clear explanation, but it did not have the final solution that I need.

Now, we will consider a basis logistic function: $$h_\theta(x^{i})=\frac{1}{1+e^{-\theta_0-\theta_1x}}$$ We want to find $\theta_0$ and $\theta_1$ subject to minimum cost function: $$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+(1-y^{i})\log(1-h_\theta(x^{i}))$$

We use the notation:

$$\theta x^i:=\theta_0+\theta_1 x^i_1 $$


$$\log h_\theta(x^i)=\log\frac{1}{1+e^{-\theta x^i} }=-\log ( 1+e^{-\theta x^i} ),$$ $$\log(1- h_\theta(x^i))=\log(1-\frac{1}{1+e^{-\theta x^i} })=\log (e^{-\theta x^i} )-\log ( 1+e^{-\theta x^i} )=-\theta x^i-\log ( 1+e^{-\theta x^i} ),$$


$$J(\theta)=-\frac{1}{m}\sum_{i=1}^m \left[y_i\theta x^i-\theta x^i-\log(1+e^{-\theta x^i})\right]=-\frac{1}{m}\sum_{i=1}^m \left[y_i\theta x^i-\log(1+e^{\theta x^i})\right],~~(*)$$

where the second equality follows from:

$$-\theta x^i-\log(1+e^{-\theta x^i})= -\left[ \log e^{\theta x^i}+ \log(1+e^{-\theta x^i} ) \right]=-\log(1+e^{\theta x^i}). $$

All I need now is to compute the partial derivatives of $(*)$ w.r.t. $\theta_j$. As

$$\frac{\partial}{\partial \theta_J}y_i\theta x^i=y_ix^i_j$$ $$\frac{\partial}{\partial \theta_j}\log(1+e^{\theta x^i})=\frac{x^i_je^{\theta x^i}}{1+e^{\theta x^i}}=x^i_jh_\theta(x^i), $$

The above steps are correct, but it did not have the solution for $\theta_0$ and $\theta_1$. We can take the derivative and put it equal $0$. Namely,

$$\frac{\partial J(\theta)}{\partial \theta_0}=y-\frac{e^{\theta_0+\theta_1 x}}{1+e^{\theta_0+\theta_1 x}}=0~~(*)$$ $$\frac{\partial J(\theta)}{\partial \theta_1}=yx-\frac{xe^{\theta_0+\theta_1 x}}{1+e^{\theta_0+\theta_1 x}}=0 ~~(**) $$

  • Are $(*)$ and $(**)$ correct?
  • Assume that $y$ and $x$ are known. How do you represent $\theta_0$ and $\theta_1$ by $y$ and $x$?
share|improve this question

marked as duplicate by whuber Jul 21 '14 at 16:49

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

They didn't supply the final solution because there is no closed form solution for estimating the parameters of a logistic regression. Instead, an iterative search algorithm is used. The most common choice is the Newton-Raphson algorithm, but there are many possibilities.

share|improve this answer
This is one difference comparison with linear regression (Gradient descent), right? Thanks – user8264 Jul 21 '14 at 15:48
That's right, @user8264. The non-linear transformations in the GLiM mess it up, but in a linear model there is no problem. – gung Jul 21 '14 at 15:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.