According to Verbeek, we can obtain the logit model by simplifying the first order condition of the log-likelihood function. Where, 

$$logL(\beta) = \Sigma^N_{i=1} y_i logF(x^{'}_i\beta)+ \Sigma^N_{i=1}(1-y_i)log(1-F(x^{'}_i\beta))$$

and the first-order condition is:

$$\frac{{\partial L(\beta)}}{{\partial \beta}} = \Sigma^{N}_{i=1} [\frac{{y_i-F(x^{'}_i\beta)}}{{F(x^{'}_i\beta)(1-F(x^{'}_i\beta))}} f(x^{'}_i\beta)]x_i=0$$ 

where F is some distribution function and f = F' (the derivative of the distribution function)

And we obtain, $$\frac{{\partial L(\beta)}}{{\partial \beta}} = \Sigma^{N}_{i=1} [y_i -\frac{{exp(x^{'}_i\beta)}}{{1+exp(x^{'}_i\beta))}}]x_i=0$$ 

However, I don't understand how this is simplified and I'm not certain how the first order condition is solved, is it an application of the chain rule?

Thank you.


The general first order condition is an application of the chain rule and the definition that

$$f(\eta) := \frac{\partial F(\eta)}{\partial \eta}$$

A good place to start is the following expression $$l_i(\eta) := y_i \log F(\eta) + (1-y_i) \log (1-F(\eta)),$$ which is more simple than the log-likelihood because we ignore the sum and because we ignore $x_i'\beta$. Then simply differentiate with respect to $\eta$ to get

$$y_i \frac{f(\eta)}{F(\eta)} - (1-y_i) \frac{f(\eta)}{1-F(\eta)},$$ isolate factor $f(\eta)$ and multiply the term $f(\eta)/F(\eta)$ with $1-F(\eta)$ in numerator and denominator and multiply fraction $f(\eta)/(1-F(\eta))$ with $F(\eta)$ in numerator and denominator. This gets you

$$f(\eta) \left[ \frac{y_i(1-F)}{F(1-F)} - \frac{(1-y_i)F}{F(1-F)}\right]$$ multiply through in numerators and get the expression

$$f(\eta) \left[ \frac{y_i-F(\eta)}{F(\eta)(1-F(\eta))}\right] = \frac{\partial l_i(\eta)}{\partial \eta} \ \ (1),$$

you have now succesfully differentiated the individual $i$'th contribution of the log-likelihood with respect to $\eta$.

When $\eta = x'\beta$ - as it is in the current case - and you want to differentiate with respect to $\beta$ it follows by chain rule that

$$\frac{\partial l_i (x_i'\beta)}{\partial \beta} = \frac{\partial l_i(\eta)}{\partial \eta} \frac{\partial \eta}{\partial \beta} = \frac{ \partial l_i(\eta)}{\partial \eta} x_i,$$ so simply combine this with (1) and insert $\eta = x_i'\beta$ to get the general first order condition

$$(2)\ \ \ x_i f(x_i'\beta) \left[ \frac{y_i-F(x_i'\beta)}{F(x_i'\beta)(1-F(x_i'\beta))}\right] $$

Now under the specific assumption that

$$F(\eta) = \frac{\exp(\eta)}{1+\exp(\eta)},$$

it follows that

$$f(\eta) = \frac{\partial F(\eta)}{\partial \eta} = \frac{\exp(\eta) (1+\exp(\eta)) - \exp(\eta) \exp(\eta)}{(1+\exp(\eta))^2},$$ when reading this term you should look for the probabilities $Pr(y_i = 1) = F(\eta) = \exp(\eta)/(1+\exp(\eta))$ to notice that this simplifies to $$f(\eta) = \frac{\exp(\eta) (1+\exp(\eta)) - \exp(\eta) \exp(\eta)}{(1+\exp(\eta))^2} = F - F^2 = F(1-F).$$

When you see that $f = F(1-F)$ it is easy to see that (2) reduces to

$$x_i (y_i-F(x_i'\beta)),$$ which is what you wanted given that $F(x_i'\beta) = exp(x_i'\beta)/(1+\exp(x_i'\beta))$.

  • 1
    $\begingroup$ Thank you! Can't tell you how much this has helped $\endgroup$ Oct 24 '20 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.