# Constrained Maximization and Likelihood Ratio Tests for Nested Linear Models

Suppose $\boldsymbol \beta \in \mathbb{R}^k$ is a vector of coefficients for a generalized linear model with $g \left[ E(Y|X) \right] = X\beta$ for a link function $g$ and I wish to test the composite hypothesis $H_0: \beta_k =0$ versus $H_1: \beta_k \neq 0$ where the other parameters $\beta_{-k}$ are nuisance parameters.

I can do that with a likelihood ratio test. Formally, the likelihood ratio test of such a composite hypothesis is 2 times the difference of the maximized unrestricted and restricted log-likelihoods: $$\lambda = 2 \left[ L^* - L_{\beta_k = 0}^* \right]$$ where $$L^* = \sup_{\beta \in \mathbb{R}^k} L(y, \boldsymbol{\beta}) \quad (a)$$ $$L_{\beta_k = 0}^*= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta}) \quad (b)$$ and $L(y, \boldsymbol \beta)$ is the log-likelihood of data $y$ evaluated at parameter $\boldsymbol \beta$.

But in practice, instead everyone replaces the constrained maximization $L_{\beta_k = 0}^*$ with

$$L^*_{-k} = \sup_{\beta_{-k} \in R^{k-1}} L_{k-1} (y, \boldsymbol{\beta}) \quad (c)$$

where $L_{-k}$ is the log-likelihood of the model $g \left[ E(Y|X) \right] = X_{-k}\beta_{-k}$ that deletes the last covariate, as in doing an analysis of deviance.

Is this replacement justified through a slutsky-theorem like argument about the rate of convergence that $L^*_{k-1}$ has to $L_{\beta_k = 0}^*$ (eg at least as fast as $O(n)$?) Or am I missing something that actually implies the two quantities are the same in finite samples? Assuming they are different, does anyone have a reference that discusses the finite sampling differences the two estimates have?

## Edit -- Well this is embarrassing

With a little bit more thinking about the maximizations in (b) and (c) it's clear that they solve the same program, hence this entire exercise was ill-posed.

Write \begin{eqnarray*} \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta}) &= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, [\beta_{-k}, \beta_k]) \\ & = \sup_{\beta \in \mathbb{R}^k} \begin{cases} L(y, [\beta_{-k}, \beta_k]) & \beta_k=0 \\ -\infty & \beta_k \neq 0 \end{cases} \\ & = \sup_{\beta \in \mathbb{R}^k} \begin{cases} L_{-k}(y, \beta_{-k}) & \beta_k=0 \\ -\infty & \beta_k \neq 0 \end{cases} \\ & = \sup_{\beta_{-k} \in \mathbb{R}^{k-1}} L_{-k}(y, \beta_{-k}) \end{eqnarray*}

## Equivalence of Likelihoods

The two quantities

$$L_{\beta_k = 0}^*= \sup_{\beta_{-k} \in \mathbb{R}^{k-1}, \beta_k=0} L(y, \boldsymbol{\beta})$$

and

$$L^*_{-k} = \sup_{\beta_{-k} \in R^{k-1}} L_{k-1} (y, \boldsymbol{\beta})$$

are equivalent for finite samples. If you assume a generalized linear model where $g \left[ E(Y|X) \right] = X\beta$, and no other aspect of the model depends on $X$, for instance the dispersion parameter, then

$$g[E(Y|X_{-k})]=X_{-k}\beta_{-k}=X\beta|_{\beta_k=0}=g[E(Y|X)]|_{\beta_k=0}$$

and the two likelihoods must be the same, since $X$ appears in both only as a linear function of $\beta$.

## Matrix Equations

Why is the estimate $\hat{\beta}$ a solution to both
$$X^T (Y - g^{-1}(X \beta)) = 0, \quad (1)$$

subject to $\beta_k=0$, and

$$X_{-k}^T (Y - g^{-1}(X_{-k} \beta_{-k})) = 0, \quad (2)$$

Since $X\beta|_{\beta_k=0}=X_{-k} \beta_{-k}$, the residuals $\hat{e} = Y - g^{-1}(X \beta)$ and $\hat{e_{-k}} = Y - g^{-1}(X_{-k} \beta_{-k})$ are the same in both equations.

Equations (1) and (2) are a statement of orthogonality with respect to the inner product between the residuals $\hat{e}$ and a tangent plane,

$$T = g^{-1}(X \beta) + \left\{ {\frac{\partial}{\partial \beta}}g^{-1}(X \beta)\cdot \lambda \Big\| \lambda \in \mathbb{R}^{k-1} \right\}$$

This tangent plane again only depends on $X$ through the quantity $X\beta$ and the two orthogonality constraints are equivalent result in the same solution in the restricted space $\left\{\beta \Big\| \beta \in \mathbb{R}^k, \beta_k=0\right\}$.

See this link for more on the geometry of GLMs.

Edit As the OP's numerical example shows, a solution to (2) is not necessarily a solution to (1). However, the solutions agree on the subspace $\left\{\beta \Big\| \beta \in \mathbb{R}^k, \beta_k=0\right\}$ which is all that is necessary.

• Thanks for the attempt. Note the edit I made above--I goofed including the weight matrix $V$ in the normal equations for the logistic regression--it's not needed since the logit is the canonical link. I agree that if the $\beta_{-k}$ are the same in equations 1 and 2, then (2) implies (1). However, it's not true that the solution to 2 gives the solution to 1, in general. See the edit for a numerical counter example. – Andrew M Aug 5 '14 at 9:57
• @Andrew M You are correct, a solution to (2) is not necessarily a solution to (1), although you can see that they agree on the restricted subspace. I edited to clarify that point. – caburke Aug 5 '14 at 14:59
• can you clarify what you mean that the solutions agree on the subspace $\left\{ \beta \in \mathbb R^k | \beta_k=0 \right\}$? Do you mean that the solution locus (points $\left\{ Y \in \mathbf R^n | Y = g^{-1} (X\beta) \right\}$?) is the same? If so, I agree. But the point of equation (1) or (2) is define which particular point in $\mathbf R^n$ on the solution locus we're going to use. Equation (1) yields a different point than equation (2). – Andrew M Aug 5 '14 at 19:32
• As I understand it, you wanted to determine why 2 different ways of viewing a restricted model, where $\beta_k=0$, result in the same estimate. The restricted subspace where the solution is to be found is $\big\{\beta \in \mathbb{R}^k | \beta_k=0 \big\}$, but if we look for the solution to $(2)$ in $\mathbb{R}^k$, the solution will not in general agree with $(1)$. The only element of $\hat{\beta}_k$ that disagrees with $(1)$ is the estimate of $\beta_k$, which is assumed to be $0$. Thus the requirement that the residuals be orthogonal to $X_k$ is contradictory to the assumption of $\beta_k=0$. – caburke Aug 5 '14 at 19:54
• So the solution of $(2)$ is only a maximum of the restricted likelihood if it is projected onto the restricted parameter space, that is, setting $\beta_k=0$. The more I think about it, $(1)$ is a much more intuitive, especially geometrically. Why project the data onto the enlarged space when you know the solution lies in a subset? More simply, why add $\beta_k$ to the estimation procedure if you assume its value is known? – caburke Aug 5 '14 at 20:01