Constrained Maximization and Likelihood Ratio Tests for Nested Linear Models

Question

Suppose $\boldsymbol \beta \in \mathbb{R}^k$ is a vector of coefficients for a generalized linear model with $g \left[ E(Y|X) \right] = X\beta$ for a link function $g$ and I wish to test the composite hypothesis $H_0: \beta_k =0$ versus $H_1: \beta_k \neq 0$ where the other parameters $\beta_{-k}$ are nuisance parameters.

I can do that with a likelihood ratio test. Formally, the likelihood ratio test of such a composite hypothesis is 2 times the difference of the maximized unrestricted and restricted log-likelihoods: $$ \lambda = 2 \left[ L^* - L_{\beta_k = 0}^* \right] $$ where $$L^* = \sup_{\beta \in \mathbb{R}^k} L(y, \boldsymbol{\beta}) \quad (a) $$ $$L_{\beta_k = 0}^*= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta}) \quad (b) $$ and $L(y, \boldsymbol \beta)$ is the log-likelihood of data $y$ evaluated at parameter $\boldsymbol \beta$.

But in practice, instead everyone replaces the constrained maximization $L_{\beta_k = 0}^*$ with

$$ L^*_{-k} = \sup_{\beta_{-k} \in R^{k-1}} L_{k-1} (y, \boldsymbol{\beta}) \quad (c) $$

where $L_{-k}$ is the log-likelihood of the model $g \left[ E(Y|X) \right] = X_{-k}\beta_{-k}$ that deletes the last covariate, as in doing an analysis of deviance.

Is this replacement justified through a slutsky-theorem like argument about the rate of convergence that $L^*_{k-1}$ has to $L_{\beta_k = 0}^*$ (eg at least as fast as $O(n)$?) Or am I missing something that actually implies the two quantities are the same in finite samples? Assuming they are different, does anyone have a reference that discusses the finite sampling differences the two estimates have?

Edit -- Well this is embarrassing

With a little bit more thinking about the maximizations in (b) and (c) it's clear that they solve the same program, hence this entire exercise was ill-posed.

Write \begin{eqnarray*} \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta}) &= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, [\beta_{-k}, \beta_k]) \\ & = \sup_{\beta \in \mathbb{R}^k} \begin{cases} L(y, [\beta_{-k}, \beta_k]) & \beta_k=0 \\ -\infty & \beta_k \neq 0 \end{cases} \\ & = \sup_{\beta \in \mathbb{R}^k} \begin{cases} L_{-k}(y, \beta_{-k}) & \beta_k=0 \\ -\infty & \beta_k \neq 0 \end{cases} \\ & = \sup_{\beta_{-k} \in \mathbb{R}^{k-1}} L_{-k}(y, \beta_{-k}) \end{eqnarray*}

Answer 1

Equivalence of Likelihoods

The two quantities

$$L_{\beta_k = 0}^*= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta})$$

and

$$ L^*_{-k} = \sup_{\beta_{-k} \in R^{k-1}} L_{k-1} (y, \boldsymbol{\beta}) $$

are equivalent for finite samples. If you assume a generalized linear model where $g \left[ E(Y|X) \right] = X\beta$, and no other aspect of the model depends on $X$, for instance the dispersion parameter, then

$$ g[E(Y|X_{-k})]=X_{-k}\beta_{-k}=X\beta|_{\beta_k=0}=g[E(Y|X)]|_{\beta_k=0} $$

and the two likelihoods must be the same, since $X$ appears in both only as a linear function of $\beta$.

Matrix Equations

Why is the estimate $\hat{\beta}$ a solution to both
$$ X^T (Y - g^{-1}(X \beta)) = 0, \quad (1) $$

subject to $\beta_k=0$, and

$$ X_{-k}^T (Y - g^{-1}(X_{-k} \beta_{-k})) = 0, \quad (2) $$

Since $X\beta|_{\beta_k=0}=X_{-k} \beta_{-k}$, the residuals $\hat{e} = Y - g^{-1}(X \beta)$ and $\hat{e_{-k}} = Y - g^{-1}(X_{-k} \beta_{-k})$ are the same in both equations.

Equations (1) and (2) are a statement of orthogonality with respect to the inner product between the residuals $\hat{e}$ and a tangent plane,

$$ T = g^{-1}(X \beta) + \left\{ {\frac{\partial}{\partial \beta}}g^{-1}(X \beta)\cdot \lambda \Big\| \lambda \in \mathbb{R}^{k-1} \right\} $$

This tangent plane again only depends on $X$ through the quantity $X\beta$ and the two orthogonality constraints are equivalent result in the same solution in the restricted space $\left\{\beta \Big\| \beta \in \mathbb{R}^k, \beta_k=0\right\}$.

See this link for more on the geometry of GLMs.

Edit As the OP's numerical example shows, a solution to (2) is not necessarily a solution to (1). However, the solutions agree on the subspace $\left\{\beta \Big\| \beta \in \mathbb{R}^k, \beta_k=0\right\}$ which is all that is necessary.