The problem is from John A. Rice "Mathematical Statistics and Data Analysis" proof that in determining the estimate value for a given cell in a test of homogeneity the MLE is what our intuition would dictate.

The set up is a fascinating example of trying to determine (I'm using the first part of the exercise only) if Jane Austen's word counts in "Sense and Sensibility" are the same as in "Emma" or the first part of "Sanditon". Here's my own rendition of part of the data with the Agresti $\pi$-based nomenclature with $\cdot$ and $\cdot \cdot$ indicating summary across columns and rows:

enter image description here

The idea is that:

$H_o: \pi_{i1} = \pi_{i2}=\cdots=\pi_{iJ}$. Or in English, the proportions of each word (rows) in different novels (columns) is identical. There are $J$ columns, each one corresponding to a multinomial distribution with $I$ categories (rows). The he occurrence of a word, $i$-th row, is the same under the null hypothesis across all multinomials (columns), and its MLE is, $\Large \pi_i=n_{i\,\cdot}/n_{\cdot\,\cdot}$ (period)

The proof proceeds as follows:

$\Large \text{lik}(\pi_1,\pi_2,\cdots,\pi_I) = \displaystyle\prod_{j=1}^J {n_{\,\cdot \,j}\choose n_{1\,j}\,n_{2\,j}\,\cdots\, n_{I\,j}}\,\pi_1^{n_1\,j}\,\pi_2^{n_2\,j}\,\cdots\,\,\pi_I^{n_I\,j}$

$\Large=\,\pi_1^{n_1\,\cdot}\,\pi_2^{n_2\,\cdot}\,\cdots\,\,\pi_I^{n_I\,\cdot} \, \displaystyle \prod_{j=1}^J {n_{\,\cdot \,j}\choose n_{1\,j}\,n_{2\,j}\,\cdots\, n_{I\,j}}$

Maximizing the log likelihood with the constraint $\displaystyle \sum_{i=1}^I\pi_i=1$, using a Lagrange multiplier:

$l(\pi,\lambda)=\displaystyle\sum_{j=1}^J log {n_{\,\cdot \,j}\choose n_{1\,j}\,n_{2\,j}\cdots n_{I\,j}} + \displaystyle\sum_{i=1}^I n_{i\,\cdot}\,log\,\pi_i\,+\,\lambda\left(\displaystyle\sum_{i=1}^I\pi_i-1\right) \tag 1$.


$\Large \frac{\partial l}{\partial \pi_i}=\frac{n_{i\,\cdot}}{\pi_i}+\lambda =0 \tag 2$

and $\hat\pi_i=-{n_{i\,\cdot}}/{\lambda}$.

I follow to this point, but I don't know what the closing sentence means (perhaps because I am not familiar with Lagrange): "summing over both sides and applying the constraint, we find $\large\lambda= -n_{\cdot\cdot}$, and $\large\hat\pi_i=n_{i\,\cdot}/n_{\cdot\cdot}$, as was to be proved."

Can I get some help working out the last sentence into a couple of steps?

  • $\begingroup$ I used to like typing latex before this post... the dots are maddening... Incidentally, because they are so tricky I made the equations large, avoiding problems of lack of conspicuity. $\endgroup$ Dec 13 '15 at 19:22
  • $\begingroup$ Jane Austen wrote "Sense and Sensibility" and "Sanditon". $\endgroup$
    – Glen_b
    Dec 14 '15 at 2:50
  • $\begingroup$ Thank you. The example is great, but I have no familiarity with her works, and never studied English literature. I do have, over the years, spent quite a bit of energy looking for ways to deflect any requests to see movie adaptations of her novels. $\endgroup$ Dec 14 '15 at 3:00
  • 1
    $\begingroup$ @Glen_b I just checked John A. Rice's book Mathematical Statistics and Data Analysis and it reads, "Sandit-i-on". It's the third edition, page 517. $\endgroup$ Dec 14 '15 at 3:08

The Lagrange formulation requires solving a set of equations simultaneously and the usual course of action is first to determine the value of the multiplier and eliminate it from the equations before proceeding to solve for the variables of interest.

For your problem, which actually admits a quite elegant solution, remember that we are maximizing this function under the sum constraint, namely $\displaystyle \sum_{i=1}^I\pi_i=1$. What your proof says is that for all $\pi_i$ we have the first order conditions $\widehat{\pi} = -n_i /\lambda$, right? But now we can simplify that by using the constraint, that is

$$ \sum_{i} \widehat{\pi} = -\sum_{i} n_i/\lambda \implies 1 = -n/\lambda$$

whence we get that the Lagrange multiplier, $\lambda$, equals $-n$. Now, using this knowledge we can substitute back and solve for each $\widehat{\pi}_i$ which quite intuitively turns out to be equal to the relative frequency, $\displaystyle{ \frac{n_i}{n}} $

  • $\begingroup$ At what step do you substitute back? $\endgroup$ Dec 13 '15 at 21:47
  • $\begingroup$ @AntoniParellada After you express the multiplier in terms of the variables of your problem. $\endgroup$
    – JohnK
    Dec 13 '15 at 21:48
  • $\begingroup$ I made some changes in the OP. Do you substitute back into equation (1)? $\endgroup$ Dec 13 '15 at 21:54
  • $\begingroup$ @AntoniParellada No, you substitute back in the first order conditions which you have to solve simultaneously.By first order, I mean the partial derivatives of the Lagrangian with respect to each variable. $\endgroup$
    – JohnK
    Dec 13 '15 at 21:55
  • $\begingroup$ On equation (2) getting thus the frequency as a function of the totals for each row? $\endgroup$ Dec 13 '15 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.