Suppose $X_{1}, X_{2},\ldots,X_{n}$ are i.i.d. observations from a multivariate normal distribution $N(\mu,\Sigma)$ where $\Sigma$ is known.

Use the likelihood ratio procedure to produce a test statistic for $H_{0}: R\mu=r$ versus $H_{1}: R\mu\neq r$. Assume $R$ is a given matrix and $r$ is a given vector.

This is the question that I cannot find the direct answer to anywhere. How does one go about answering this? I tried to use the Lagrangian for the constraint case under $H_{0}$ but I am getting nowhere. Any tip would be greatly appreciated.

  • $\begingroup$ Is $R$ invertible? $\endgroup$
    – user20160
    Feb 22 '20 at 0:34
  • $\begingroup$ It does not say anything about it being invertible unfortunately. $\endgroup$
    – user9827
    Feb 22 '20 at 9:28

Likelihood ratio test statistic

Let $\ell(\mu)$ denote the log likelihood of mean $\mu$ (assuming known covariance matrix $\Sigma$):

$$\ell(\mu) = \sum_{i=1}^n \log \mathcal{N}(x_i \mid \mu, \Sigma)$$

The problem involves a nested model, and the likelihood ratio test statistic has the standard form:

$$S = -2 \Big( \ell(\mu_0) - \ell(\hat{\mu}) \Big)$$

$\mu_0$ is the mean that maximizes the likelihood, subject to the constraints imposed under the null hypothesis. $\hat{\mu}$ is the maximum likelihood estimate for the mean (without any constraints), which is just the mean of the data: $\hat{\mu} = \frac{1}{n} \sum_{i=1}^n x_i$. Plugging these in, the test statistic can be simplified to:

$$S = n (\hat{\mu} - \mu_0)^T \Sigma^{-1} (\hat{\mu} - \mu_0)$$

The main challenge is how to find $\mu_0$, which is the solution to a constrained optimization problem:

$$\mu_0 = \arg \max_\mu \ell(\mu) \quad \text{s.t. } R \mu = r$$

Finding $\mu_0$

First, let's assume that the problem is feasible (i.e. there exists a $\mu$ such that $R \mu = r$). If $R$ is invertible, then there's a unique choice $\mu_0 = R^{-1} r$, and we're done. Otherwise, there's a continuum of possible choices that satisfy the constraints, and we must find one that maximizes the likelihood.

Maximizing the likelihood is equivalent to minimizing the negative log likelihood, which is proportional to the following:

$$-\ell(\mu) \propto \frac{1}{n} \sum_{i=1}^n (x-\mu)^T \Sigma^{-1} (x-\mu)$$

Expanding things out, discarding constant terms (which don't affect the solution), and substituting in $\hat{\mu} = \frac{1}{n} \sum_{i=1}^n x_i$, we can reformulate the optimization problem as:

$$\mu_0 = \arg \min_\mu \ \mu^T \Sigma^{-1} \mu - 2 (\Sigma^{-1} \hat{\mu})^T \mu \quad \text{s.t. } R \mu = r$$

This is a quadratic program with a linear equality constraint, so there's a unique solution. The Lagrangian is:

$$\mathcal{L}(\mu) = \mu^T \Sigma^{-1} \mu - 2 (\Sigma^{-1} \hat{\mu})^T \mu + (R \mu - r)^T \lambda$$

where $\lambda$ is a vector of Lagrange multipliers. Differentiating the Lagrangian w.r.t. $\mu$ and $\lambda$ and setting the gradients to zero yields the following system of linear equations:

$$\begin{bmatrix} 2 \Sigma^{-1} & R^T \\ R & \mathbf{0} \end{bmatrix} \begin{bmatrix} \mu \\ \lambda \end{bmatrix} = \begin{bmatrix} 2 \Sigma^{-1} \hat{\mu} \\ r \end{bmatrix}$$

The simplest approach is to solve this linear system directly. Let:

$$A = \begin{bmatrix} 2 \Sigma^{-1} & R^T \\ R & \mathbf{0} \end{bmatrix} \quad \quad z= \begin{bmatrix} \mu_0 \\ \lambda \end{bmatrix} \quad \quad y = \begin{bmatrix} 2 \Sigma^{-1} \hat{\mu} \\ r \end{bmatrix}$$

Solve $A z = y$ for $z$. For example, $z = A^+ y$ (using the Moore-Penrose pseudoinverse of $A$; but using something like the LU decomposition would probably be more efficient). Then $\mu_0 = [z_1, \dots, z_d]^T$ (where $d$ is the dimensionality of the data).


I assume $X_1,X_2,\ldots,X_n$ are i.i.d $p$-variate normal $N_p(\mu,\Sigma)$.

We know that unrestricted MLE of $\mu$ is the sample mean vector:

$$\hat\mu=\frac1n\sum_{i=1}^n X_i \sim N_p \left(\mu,\frac{\Sigma}n\right)$$

Now MLE of $R\mu$ is $R\hat\mu$. And if $R$ is a $p\times p$ matrix, then $R\hat\mu$ is also $p$-variate normal:

$$R\hat\mu \sim N_p \left(R\mu,R\left(\frac{\Sigma}n\right) R^T\right)$$

This implies

$$(R\hat\mu -R\mu)^T\left(R\left(\frac{\Sigma}n\right) R^T\right)^{-1}(R\hat\mu-R\mu) \sim \chi^2_p$$


$$T=n(R\hat\mu -r)^T(R\Sigma R^T)^{-1}(R\hat\mu-r) \stackrel{H_0}\sim \chi^2_p$$

I would try to express the likelihood ratio test criterion $\Lambda$ in terms of $T$, so that small values of $\Lambda$ correspond to large values of $T$. Then $T$ is a suitable test statistic for testing $H_0$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.