Suppose $\boldsymbol \beta \in \mathbb{R}^k$ is a vector of coefficients for a generalized linear model with $g \left[ E(Y|X) \right] = X\beta$ for a link function $g$ and I wish to test the composite hypothesis $H_0: \beta_k =0$ versus $H_1: \beta_k \neq 0$ where the other parameters $\beta_{-k}$ are nuisance parameters.

I can do that with a likelihood ratio test. Formally, the likelihood ratio test of such a composite hypothesis is 2 times the difference of the maximized unrestricted and restricted log-likelihoods: $$ \lambda = 2 \left[ L^* - L_{\beta_k = 0}^* \right] $$ where $$L^* = \sup_{\beta \in \mathbb{R}^k} L(y, \boldsymbol{\beta}) \quad (a) $$ $$L_{\beta_k = 0}^*= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta}) \quad (b) $$ and $L(y, \boldsymbol \beta)$ is the log-likelihood of data $y$ evaluated at parameter $\boldsymbol \beta$.

But in practice, instead everyone replaces the constrained maximization $L_{\beta_k = 0}^*$ with

$$ L^*_{-k} = \sup_{\beta_{-k} \in R^{k-1}} L_{k-1} (y, \boldsymbol{\beta}) \quad (c) $$

where $L_{-k}$ is the log-likelihood of the model $g \left[ E(Y|X) \right] = X_{-k}\beta_{-k}$ that deletes the last covariate, as in doing an analysis of deviance.

Is this replacement justified through a slutsky-theorem like argument about the rate of convergence that $L^*_{k-1}$ has to $L_{\beta_k = 0}^*$ (eg at least as fast as $O(n)$?) Or am I missing something that actually implies the two quantities are the same in finite samples? Assuming they are different, does anyone have a reference that discusses the finite sampling differences the two estimates have?

Edit -- Well this is embarrassing

With a little bit more thinking about the maximizations in (b) and (c) it's clear that they solve the same program, hence this entire exercise was ill-posed.

Write \begin{eqnarray*} \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta}) &= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, [\beta_{-k}, \beta_k]) \\ & = \sup_{\beta \in \mathbb{R}^k} \begin{cases} L(y, [\beta_{-k}, \beta_k]) & \beta_k=0 \\ -\infty & \beta_k \neq 0 \end{cases} \\ & = \sup_{\beta \in \mathbb{R}^k} \begin{cases} L_{-k}(y, \beta_{-k}) & \beta_k=0 \\ -\infty & \beta_k \neq 0 \end{cases} \\ & = \sup_{\beta_{-k} \in \mathbb{R}^{k-1}} L_{-k}(y, \beta_{-k}) \end{eqnarray*}


1 Answer 1


Equivalence of Likelihoods

The two quantities

$$L_{\beta_k = 0}^*= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta})$$


$$ L^*_{-k} = \sup_{\beta_{-k} \in R^{k-1}} L_{k-1} (y, \boldsymbol{\beta}) $$

are equivalent for finite samples. If you assume a generalized linear model where $g \left[ E(Y|X) \right] = X\beta$, and no other aspect of the model depends on $X$, for instance the dispersion parameter, then

$$ g[E(Y|X_{-k})]=X_{-k}\beta_{-k}=X\beta|_{\beta_k=0}=g[E(Y|X)]|_{\beta_k=0} $$

and the two likelihoods must be the same, since $X$ appears in both only as a linear function of $\beta$.

Matrix Equations

Why is the estimate $\hat{\beta}$ a solution to both
$$ X^T (Y - g^{-1}(X \beta)) = 0, \quad (1) $$

subject to $\beta_k=0$, and

$$ X_{-k}^T (Y - g^{-1}(X_{-k} \beta_{-k})) = 0, \quad (2) $$

Since $X\beta|_{\beta_k=0}=X_{-k} \beta_{-k}$, the residuals $\hat{e} = Y - g^{-1}(X \beta)$ and $\hat{e_{-k}} = Y - g^{-1}(X_{-k} \beta_{-k})$ are the same in both equations.

Equations (1) and (2) are a statement of orthogonality with respect to the inner product between the residuals $\hat{e}$ and a tangent plane,

$$ T = g^{-1}(X \beta) + \left\{ {\frac{\partial}{\partial \beta}}g^{-1}(X \beta)\cdot \lambda \Big\| \lambda \in \mathbb{R}^{k-1} \right\} $$

This tangent plane again only depends on $X$ through the quantity $X\beta$ and the two orthogonality constraints are equivalent result in the same solution in the restricted space $\left\{\beta \Big\| \beta \in \mathbb{R}^k, \beta_k=0\right\}$.

See this link for more on the geometry of GLMs.

Edit As the OP's numerical example shows, a solution to (2) is not necessarily a solution to (1). However, the solutions agree on the subspace $\left\{\beta \Big\| \beta \in \mathbb{R}^k, \beta_k=0\right\}$ which is all that is necessary.

  • 1
    $\begingroup$ Thanks for the attempt. Note the edit I made above--I goofed including the weight matrix $V$ in the normal equations for the logistic regression--it's not needed since the logit is the canonical link. I agree that if the $\beta_{-k}$ are the same in equations 1 and 2, then (2) implies (1). However, it's not true that the solution to 2 gives the solution to 1, in general. See the edit for a numerical counter example. $\endgroup$
    – Andrew M
    Aug 5, 2014 at 9:57
  • $\begingroup$ @Andrew M You are correct, a solution to (2) is not necessarily a solution to (1), although you can see that they agree on the restricted subspace. I edited to clarify that point. $\endgroup$
    – caburke
    Aug 5, 2014 at 14:59
  • $\begingroup$ can you clarify what you mean that the solutions agree on the subspace $\left\{ \beta \in \mathbb R^k | \beta_k=0 \right\}$? Do you mean that the solution locus (points $\left\{ Y \in \mathbf R^n | Y = g^{-1} (X\beta) \right\}$?) is the same? If so, I agree. But the point of equation (1) or (2) is define which particular point in $\mathbf R^n$ on the solution locus we're going to use. Equation (1) yields a different point than equation (2). $\endgroup$
    – Andrew M
    Aug 5, 2014 at 19:32
  • $\begingroup$ As I understand it, you wanted to determine why 2 different ways of viewing a restricted model, where $\beta_k=0$, result in the same estimate. The restricted subspace where the solution is to be found is $\big\{\beta \in \mathbb{R}^k | \beta_k=0 \big\}$, but if we look for the solution to $(2)$ in $\mathbb{R}^k$, the solution will not in general agree with $(1)$. The only element of $\hat{\beta}_k$ that disagrees with $(1)$ is the estimate of $\beta_k$, which is assumed to be $0$. Thus the requirement that the residuals be orthogonal to $X_k$ is contradictory to the assumption of $\beta_k=0$. $\endgroup$
    – caburke
    Aug 5, 2014 at 19:54
  • $\begingroup$ So the solution of $(2)$ is only a maximum of the restricted likelihood if it is projected onto the restricted parameter space, that is, setting $\beta_k=0$. The more I think about it, $(1)$ is a much more intuitive, especially geometrically. Why project the data onto the enlarged space when you know the solution lies in a subset? More simply, why add $\beta_k$ to the estimation procedure if you assume its value is known? $\endgroup$
    – caburke
    Aug 5, 2014 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.