# Maximization of quotient of quadratic forms in linear regression

I would like to find maximum of the following function:

$$I = \max_{a\in \mathbb{R}^p} \frac{(a'\hat{\beta})^2}{S^2a'(X'X)^{-1}a},$$

where $$X$$ is a design matrix and of course $$Y$$ is normally distributed $$N(X\beta, \sigma^2I).$$

Trying to do it the standard way, i.e. differentiating the quotient doesn't work well, I can't solve the resulting equation for $$a$$. Is it manageable to do it through differentiation?

I tried to do it using Cauchy-Schwarz inequality. I obtained:

$$I = \frac{1}{S^2} \max_{a\in \mathbb{R}^p} \frac{a'(X'X)^{-1}X'YY'X(X'X)^{-1}a}{a'(X'X)^{-1}X'X(X'X)^{-1}a}.$$

And now by Cauchy-Schwarz inequality what I get is

$$\frac{1}{S^2} \frac{(c'Y)^2}{c'c} \leq \frac{1}{S^2} \frac{c'cY'Y}{c'c} = \frac{Y'Y}{S^2},$$

where $$c = X(X'X)^{-1}a$$

But the equality in Cauchy-Schwarz inequality holds, when there exists $$d\in \mathbb{R}$$ such that $$c = dY$$ but it requires invertibility of $$(XX').$$

I would appreciate any input.

• $(XX’)$ can’t be invertible because it is not full rank. Nov 17, 2019 at 23:01

Using that $$\hat\beta= (X^TX)^{-1} X^T Y$$ so $$(a^T \hat\beta )^2=a^T (X^TX)^{-1} X^T Y Y^T X (X^TX)^{-1} a$$, you can write $$I$$ as $$I= max_{a\in\mathbb{R}^p}\frac{a^T C a}{a^T B a}$$ where $$C=(X^TX)^{-1} X^T Y Y^T X (X^TX)^{-1}$$ and $$B=(X^TX)^{-1}$$, where we left out $$S^2$$ to simplify, since it is just a constant so does not influence the solution. Then you can apply the answer given to your other question.

Consider the following result, discussed at https://math.stackexchange.com/q/1226524/321264 :

If $$B$$ is a symmetric positive definite matrix and $$x,d$$ are real vectors, then $$\max_{x\ne 0}\frac{(x^Td)^2}{x^TBx}=d^T B^{-1}d \tag{1}$$

The proof is based on the following generalization of Cauchy-Schwarz inequality:

$$(x^T d)^2\le (x^TBx)(d^TB^{-1}d) \tag{2}$$

And to prove $$(2)$$, one can apply the usual Cauchy-Schwarz inequality to $$B^{1/2}x$$ and $$B^{-1/2}d$$.

I assume $$X$$ is of full column rank, whence $$X^TX$$ is a positive definite matrix.

Using $$(1)$$, your $$I$$ reduces to

$$I=\frac{\hat\beta^T(X^TX)\hat\beta}{S^2}$$