Suppose there is a set of $n$ independent observations $y_i$ from the exponential family of distributions. How can we prove that in a saturated GLM model, the fitted MLE values $\hat\mu_i = y_i$ for all $1 \leq i \leq n$?

Edit: Using the new definition proposed I am still not sure I get the proof correct. Let $\dot f$ denotes the derivative of $f$ with respect to the canonical parameter $\theta$, $g$ the link function and $V$ the variance function, we know that if $D = \mathrm{diag}\left( [\dot g(\mu_i)V(\mu_i) ]^{-1} \right)$ is a $p\times p$-matrix then it is easy to see that $D$ must have full rank since it is diagonal with non-zero columns. Maximizing the likelihood means $X^TD(y-\mu) = 0$. If this arises from a saturated model, then by the answer below, $\rm{rank}\ X = p$, We know that the product of two matrices of full rank has full rank, and therefore if we can somehow see that $X$ is square, the invertible matrix theorem tells us there exist an inverse, C, for $X^T$ such that $$ D^{-1}CX^TD(y - \mu) = D^{-1}C 0 = 0 \qquad \text{only if} \qquad y - \mu = 0. $$ So is there something that forbids linearly dependent columns of $X$? Can $X$ be singular? As far as I can tell, it would nail the proof. What do you think?

  • 2
    $\begingroup$ Is this a question from a course or textbook? If so, please add the [self-study] tag & read its wiki. $\endgroup$ Nov 29 '16 at 17:31
  • 3
    $\begingroup$ It's from an exam. I added the self study tag $\endgroup$
    – Mikkel Rev
    Nov 29 '16 at 17:37
  • $\begingroup$ Can't you write the design matrix $X$ for the saturated model explicitly? See stats.stackexchange.com/questions/283. $\endgroup$
    – whuber
    Nov 30 '16 at 5:26
  • $\begingroup$ What do you mean? $\endgroup$
    – Mikkel Rev
    Nov 30 '16 at 6:26

Do you have a working definition of a saturated model? It might help to develop one and work from there. If $Y$ is continuously valued with no ties, then you could claim that the rank of $\mathbf{X}$, e.g. the model matrix, is equal to the sample size $n$. If there are ties in $Y$, you could simplify things by using the weighted likelihood to create frequency weights for non-distinct $Y$ observations and maintain that definition.

  • $\begingroup$ We know that $X^T D(y-\mu) = 0$, where $D$ is the diagonal matrix of $[g'(\theta) V(\theta)]^{-1}(\mu_i)$ for link function $g$. By invertible matrix theorem, $X^T D (y-\mu) = 0$ has only the trivial solution $(y-\mu) = 0$ if $X^T D$ has full rank. This is the case if and only if $X^T D$ has full rank and is a square matrix, this would complete the proof. Is there something that forbids $X^T D$ to be non-square in a saturated model? Is it impossible to have more predictors than observations? Can we somehow guarantee that $D$ has full rank? It is diagonal, but is this enough? $\endgroup$
    – Mikkel Rev
    Nov 30 '16 at 3:09
  • $\begingroup$ Actually, $D$ has full rank. Therefore it is sufficient to see that $X$ is square. Is there something in the definition of GLM that forbids X be $m \times n$ for $m>n$, intuitively this is a problem because it would mean there are two linearly depedent rows in X. $\endgroup$
    – Mikkel Rev
    Nov 30 '16 at 3:27
  • $\begingroup$ @MariusJonsson $\mathbf{X}^T\mathbf{D} \left( y- \mathbf{X}\hat{\beta} \right) = 0$ is the likelihood equation for a GLM. You should focus on the structure of $\mathbf{X}$ as it relates to $\beta$, especially in terms of being a projection. $\endgroup$
    – AdamO
    Dec 2 '16 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.