Suppose I have data $\{y_i,x_i\}_{i=1}^N$, where $x_i\in\{s_1,...,s_K\}$ and follows a discrete uniform distribution. For each realized $x_i$, $y_i$ is generated by the logit model, i.e., $Pr(Y_i=1|x_i)=\frac{exp(\beta_0+x_i\beta_1)}{1+exp(\beta_0+x_i\beta_1)}$. We want to estimate $(\beta_0,\beta_1)$ using data $\{y_i,x_i\}_{i=1}^N$. There are two natural ways of doing it: one method is the maximum likelihood approach:

$(\widehat{\beta}_0,\widehat{\beta}_1)=\underset{\beta_0,\beta_1}{argmax} \sum_{i=1}^N y_ilog[\frac{exp(\beta_0+x_i\beta_1)}{1+exp(\beta_0+x_i\beta_1)}]+(1-y_i)log[\frac{1}{1+exp(\beta_0+x_i\beta_1)}]$

The other way of doing it is the minimum distance, that is I could first estimate $Pr(Y_i=1|x_i=s_k)$ using $\frac{\sum_{i=1}^N\mathbf{1}(y_i=1,x_i=s_k)}{\sum_{i=1}^N\mathbf{1}(x_i=s_k)}$ for $s\in \{s_1,...,s_K\}$ and denote this estimator as $\widehat{p}_k$. We choose the parameter to minimize the discrepancy between model-implied probability and relative frequency:

$(\widetilde{\beta}_1,\widetilde{\beta}_2)=\underset{\beta_0,\beta_1}{argmin}||[\frac{exp(\beta_0+s_1\beta_1)}{1+exp(\beta_0+s_1\beta_1)},...,\frac{exp(\beta_0+s_K\beta_1)}{1+exp(\beta_0+s_K\beta_1)}]-[\widehat{p}_1,...,\widehat{p}_K] ||^2$.

Intuitively, both estimators should be consistent (converging to the true value that generates our data). But on the other hand, they seem to be doing completely different things, one is trying to make the likelihood as large as possible, and the other is trying to make the model-implied probabilities as close to the relative frequencies as possible. Why the parameter value that maximizes the likelihood is also able to set the distance to zero in the limit? Intuition or formal proof are both welcome.


1 Answer 1


Let's do some math. Some notation to compact things. I will write $\Lambda_i$ to denote the Logistic distribution function evaluated at observation $i$. I will write $\Lambda(s_j)$ to denote its evaluation at the value $s_j$ of the support of $x$, $j=1,...,K$. I will write $\sum_{i=1}^N\mathbf{1}(x_i=s_j)=n_j$ to denote the frequency of $x_i = s_j$. Finally note that $$\frac{\partial \Lambda(z)}{\partial z}=\Lambda(z)[1-\Lambda(z)].$$


The objective function here is

$$\sum_{j=1}^K \big[(1-\Lambda(s_j))-\hat p_j\big]^2$$

The f.o.c with respect to the betas (in abstract) is

$$-2\sum_{j=1}^K \big[1-\Lambda(s_j)-\hat p_j\big]\cdot \Lambda(s_j)[1-\Lambda(s_j)]s_j=0. \tag{1}$$


The objective function here is $$\sum_{i=1}^N \big[y_i\ln(1-\Lambda_i) + (1-y_i)\ln\Lambda_i\big]$$

Taking the derivative with respect to $\beta$ and simpifying, we arrive at the first order condition $$\sum_{i=1}^N(1-\Lambda_i-y_i)\cdot x_i=0$$ $$\implies \sum_{i=1}^N(1-\Lambda_i)\cdot x_i=\sum_{i=1}^Ny_ix_i. \tag{2}$$

We can decompose $(2)$ per the values of $x$, in which case all $\Lambda_i$ for which $x_j = s_j$ become equal to $\Lambda(s_j)$. Namely, $(2)$ can be written

$$(2):\sum_{x_i=s_1}(1-\Lambda_i)\cdot s_1+ \cdots + \sum_{x_i=s_K}(1-\Lambda_i)\cdot s_k = \sum_{x_i=s_1}y_is_1+\cdots + \sum_{x_i=s_K}y_is_K$$

Each sum in the left-hand side has identical elements of number $n_j$. On the right hand side, in each sum, some $y_i$ will be zero and some will equal to unity, and we can see that, in each sum we have $y_i = n_j\hat p_j$.

Using these remarks we can write $(2)$ as

$$n_1\cdot [1-\Lambda (s_1)]\cdot s_1 + \cdots + n_K[1-\Lambda(s_K)]\cdot s_K = n_1\cdot \hat p_1s_1+\cdots + n_K\cdot \hat p_Ks_K,$$

and compacting,

$$ \sum_{j=1}^K n_js_j\big[1-\Lambda(s_j)\big]=\sum_{j=1}^Kn_j\hat p_js_j\tag{3}.$$

It will make no difference if we divide by sample size both sides. Writing $\hat q_j$ for the relative frequency of $x=s_j$, and the f.o.c in maximum likelihood is finally

$$ \sum_{j=1}^K \big[1-\Lambda(s_j)-\hat p_j\big]\hat q_js_j= 0. \tag{4}$$

Comparing $(1)$ and $(4)$, I guess you can take it from here.

  • $\begingroup$ Thank you very much, Alecos! This is very helpful. (1) and (4) indeed look similar. However, comparing (1) and (4), it seems that neither one implies the other. For example, if (4) is true, it seems that (1) is not necessarily true. So it seems that I cannot claim they are asymptotically the same by looking at these two first order conditions. $\endgroup$ Sep 2, 2022 at 3:04
  • 1
    $\begingroup$ @ExcitedSnail You have to think separately about consistency and about the asymptotic distribution. They are both consistent, but the variance of the asymptotic distribution should be different, by the looks of it. $\endgroup$ Sep 2, 2022 at 6:07
  • $\begingroup$ Yes, I'm only thinking about consistency (instead of asymptotic distribution). I guess now I see it: the fact that they are both consistent doesn't necessarily require their consistency conditions to be equivalent (or necessary and sufficient for each other). It's perfectly fine for the two f.o.c. to have different probability limits and not related to each other. Thank you very much! $\endgroup$ Sep 4, 2022 at 7:55
  • 1
    $\begingroup$ @ExcitedSnail Just to be on the safe side: both "f.o.c." are equations set to zero. What happens here is that both lead the $\hat \beta$s to have the same probability limit, and the desired one, i.e. the true $\beta$s, ${\rm plim} \hat \beta_{MD} = {\rm plim} \hat \beta_{ML} = \beta$, but the asymptotic distributions $\sqrt{n}(\hat \beta_{MD} - \beta)$ and $\sqrt{n}(\hat \beta_{ML} - \beta)$ will have different variance. $\endgroup$ Sep 4, 2022 at 10:50
  • $\begingroup$ @ AlecosPapadopoulos Right! What I meant was that the LHS of the two f.o.c. will have different probability limits. In the limit, the two f.o.cs will be $f_1(\beta)=0$ and $f_2(\beta)=0$, $f_1(\cdot)$ is not the same as $f_2(\cdot)$, also $f_1(\beta)=0$ doesn't imply $f_2(\beta)=0$, and $f_2(\beta)=0$ doesn't imply $f_1(\beta)=0$ either. Nevertheless, the solution of the two limiting f.o.c.s happen to be the same. $\endgroup$ Sep 6, 2022 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.