Deriving Logit Maximum Likelihood Estimator

Question

According to Verbeek, we can obtain the logit model by simplifying the first order condition of the log-likelihood function. Where, 

$$logL(\beta) = \Sigma^N_{i=1} y_i logF(x^{'}_i\beta)+ \Sigma^N_{i=1}(1-y_i)log(1-F(x^{'}_i\beta))$$

and the first-order condition is:

$$\frac{{\partial L(\beta)}}{{\partial \beta}} = \Sigma^{N}_{i=1} [\frac{{y_i-F(x^{'}_i\beta)}}{{F(x^{'}_i\beta)(1-F(x^{'}_i\beta))}} f(x^{'}_i\beta)]x_i=0$$ 

where F is some distribution function and f = F' (the derivative of the distribution function)

And we obtain, $$\frac{{\partial L(\beta)}}{{\partial \beta}} = \Sigma^{N}_{i=1} [y_i -\frac{{exp(x^{'}_i\beta)}}{{1+exp(x^{'}_i\beta))}}]x_i=0$$ 

However, I don't understand how this is simplified and I'm not certain how the first order condition is solved, is it an application of the chain rule?

Thank you.

Answer 1

The general first order condition is an application of the chain rule and the definition that

$$f(\eta) := \frac{\partial F(\eta)}{\partial \eta}$$

A good place to start is the following expression $$l_i(\eta) := y_i \log F(\eta) + (1-y_i) \log (1-F(\eta)),$$ which is more simple than the log-likelihood because we ignore the sum and because we ignore $x_i'\beta$. Then simply differentiate with respect to $\eta$ to get

$$y_i \frac{f(\eta)}{F(\eta)} - (1-y_i) \frac{f(\eta)}{1-F(\eta)},$$ isolate factor $f(\eta)$ and multiply the term $f(\eta)/F(\eta)$ with $1-F(\eta)$ in numerator and denominator and multiply fraction $f(\eta)/(1-F(\eta))$ with $F(\eta)$ in numerator and denominator. This gets you

$$f(\eta) \left[ \frac{y_i(1-F)}{F(1-F)} - \frac{(1-y_i)F}{F(1-F)}\right]$$ multiply through in numerators and get the expression

$$f(\eta) \left[ \frac{y_i-F(\eta)}{F(\eta)(1-F(\eta))}\right] = \frac{\partial l_i(\eta)}{\partial \eta} \ \ (1),$$

you have now succesfully differentiated the individual $i$'th contribution of the log-likelihood with respect to $\eta$.

When $\eta = x'\beta$ - as it is in the current case - and you want to differentiate with respect to $\beta$ it follows by chain rule that

$$\frac{\partial l_i (x_i'\beta)}{\partial \beta} = \frac{\partial l_i(\eta)}{\partial \eta} \frac{\partial \eta}{\partial \beta} = \frac{ \partial l_i(\eta)}{\partial \eta} x_i,$$ so simply combine this with (1) and insert $\eta = x_i'\beta$ to get the general first order condition

$$(2)\ \ \ x_i f(x_i'\beta) \left[ \frac{y_i-F(x_i'\beta)}{F(x_i'\beta)(1-F(x_i'\beta))}\right] $$

Now under the specific assumption that

$$F(\eta) = \frac{\exp(\eta)}{1+\exp(\eta)},$$

it follows that

$$f(\eta) = \frac{\partial F(\eta)}{\partial \eta} = \frac{\exp(\eta) (1+\exp(\eta)) - \exp(\eta) \exp(\eta)}{(1+\exp(\eta))^2},$$ when reading this term you should look for the probabilities $Pr(y_i = 1) = F(\eta) = \exp(\eta)/(1+\exp(\eta))$ to notice that this simplifies to $$f(\eta) = \frac{\exp(\eta) (1+\exp(\eta)) - \exp(\eta) \exp(\eta)}{(1+\exp(\eta))^2} = F - F^2 = F(1-F).$$

When you see that $f = F(1-F)$ it is easy to see that (2) reduces to

$$x_i (y_i-F(x_i'\beta)),$$ which is what you wanted given that $F(x_i'\beta) = exp(x_i'\beta)/(1+\exp(x_i'\beta))$.