2
$\begingroup$

Question: Suppose that $\boldsymbol{Y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$, and the errors have zero mean, and are uncorrelated with constant variance. Let $\hat{\boldsymbol{\beta}}$ be the least squares estimator of $\boldsymbol{\beta}$. Let $\boldsymbol{H} = \boldsymbol{X}(\boldsymbol{X}^T \boldsymbol{X})^{-1}\boldsymbol{X}^T$ be the hat matrix, and $h_{ii}$ be the $i$-th element on its diagonal. Show that $$ \frac{e_i^2}{\| (\boldsymbol{I} - \boldsymbol{H})\boldsymbol{Y} \|^2} \le 1 - h_{ii}$$ where $e_i = Y_i - x_i^T \hat{\boldsymbol{\beta}}$ is the $i$-th residual and $\boldsymbol{I}$ is the $n \times n$ identity matrix, where $n$ is the size of vector $\boldsymbol{Y}$.

Attempt: I got as far as writing the $e_i$ in terms of the matrices above.

$$ e_i = \sum_j (\delta_{ij} - h_{ij})Y_j = Y_i - \sum_j h_{ij}Y_j,$$

where I use $\boldsymbol{e} = (\boldsymbol{I}-\boldsymbol{H})\boldsymbol{Y}$. I square to obtain

$$ e_i^2 = Y_i^2 - 2 Y_i \sum_j h_{ij}Y_j + (\sum_j h_{ij}Y_j)^2,$$

which is then simplified to

$$ e_i^2 = Y_i^2 - 2 Y_i \sum_j h_{ij}Y_j + \sum_j h_{ij}^2 Y_j^2 + \sum_j \sum_{k \neq j} h_{ij} h_{ik} Y_i Y_j.$$

This did not lead me anywhere further. Could anyone please offer me a hint?

$\endgroup$
0

2 Answers 2

3
$\begingroup$

Note that, from $\mathbf e=(\mathbf I-\mathbf H) \mathbf Y, $ \begin{align}e_i&=-h_{i1}Y_1-h_{i2}Y_2-\cdots+(1-h_{ii})Y_i-\cdots-h_{in}Y_n\\&=\mathbf c^\top\mathbf Y, \tag 1\label 1\end{align} where $\mathbf c:=(-h_{i1}, h_{i2}, \ldots, (1-h_{ii}), \ldots, -h_{in})^\top.$

Using the fact that $\textrm{SS}(\mathbf c^\top\mathbf Y) =\left(\mathbf c^\top\mathbf Y\right)^2/\mathbf c^\top\mathbf c,$ and $\mathbf c^\top\mathbf c$ in $\eqref 1$ being equal to $(1-h_{ii}),$ the sum of squares due to the $i$th residual error component is $$\textrm{SSE}_i=\frac{e_i^2}{(1-h_{ii})}.\tag 2$$

Therefore, the sum of squares for error with $i$th case deleted can be given as $$\textrm{SSE}_{[i]}=\textrm{SSE}-\textrm{SSE}_i; \tag 3\label 3$$ (alternatively, you can write \begin{align}\textrm{SSE}_{[i]}&=\mathbf Y_{[i]}^\top\mathbf Y_{[i]}-\mathbf Y_{[i]}^\top\mathbf X_{[i]}\left(\mathbf X_{[i]}^\top\mathbf X_{[i]}\right) ^{-1}\mathbf X_{[i]}^\top\mathbf Y_{[i]}\\&=\left(\mathbf Y^\top\mathbf Y-y_i^2\right) -\left(\mathbf Y^\top\mathbf X-y_i\mathbf x_i^\top\right) \left\{\hat{\boldsymbol{\beta}}-\left[(\mathbf X^\top \mathbf X)^{-1}\mathbf x_ie_i\right]/(1-h_{ii})\right\};\end{align} simplify it and this would yield $\eqref 3.$)

$\textrm{SSE}= \mathbf e^\top\mathbf e=\|\mathbf e\|^2=\|(\mathbf I-\mathbf H) \mathbf Y\|^2$ and that $\textrm{SSE}_{[i]}\geq 0;$ from $\eqref 3,$ you can conclude your concerned inequality relation.


References:

$\rm [I]$ Applied Regression Analysis, Norman R. Draper, Harry Smith, John Wiley & Sons, $1998, $ pp. $163, ~207-208.$

$\rm [II]$ Plane Answers to Complex Questions: The Theory of Linear Models, Ronald Christensen, Springer Science$+$ Business, $2011,$ sec. $ 13.6,$ pp. $372-373.$

$\endgroup$
2
$\begingroup$

To save some typing, I will not make symbols boldfaced in the following answer.

In essence, this is just a disguise of Cauchy-Schwarz inequality. To wit, let $u_i$ denote the $n$-long column vector whose $i$-th entry is $1$ and all the remaining entries $0$. By definition, the inequality of your interest is then (the key observation is that $1 - h_{ii} = u_i^\top(I - H)u_i$): \begin{align*} \left(u_i^\top(I - H)Y\right)^2 \leq u_i^\top(I - H)u_i \cdot Y^\top(I - H)Y. \tag{1}\label{zx1} \end{align*} Now use the property that $I - H$ is symmetric and idempotent, the expression in the parentheses of the left hand side of $\eqref{zx1}$ is $((I - H)u_i)^\top (I - H)Y$. Hence $\eqref{zx1}$ is equivalent to \begin{align} ((I - H)u_i)^\top(I - H)Y)^2 \leq u_i^\top(I - H)u_i \cdot Y^\top(I - H)Y. \tag{2}\label{zx2} \end{align}

It should be clear now that $\eqref{zx2}$ is the Cauchy-Schwarz inequality (the inner-product space form) $$(a^\top b)^2 \leq \|a\|^2 \|b\|^2$$ with $a = (I - H)u_i$ and $b = (I - H)Y$.

$\endgroup$
1
  • 1
    $\begingroup$ It should be fine now. I think the culprit is that I used the same "label" as you to reference it. It is indeed a somewhat strange bug to be fixed by stackexchange (before making the label differ from yours, even the equation cannot be rendered) $\endgroup$
    – Zhanxiong
    Oct 31, 2023 at 4:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.