Consider OLS regression with the true model $y = {\theta^{*}}^{\text{T}} x + \varepsilon$, where $x$ denotes the (deterministic) independent variables, $y$ denotes the dependent (random) variable, and $\varepsilon$ is the random error.

Let $\hat{\theta}_n$ denote the OLS estimate of $\theta^*$ from data $\{(x_j,y_j)\}_{j=1}^{n}$, and $\hat{\theta}_{[i]}$ denote the OLS estimate of $\theta^*$ with the $i^{\text{th}}$ data point removed.

How can I

  1. bound the squared difference between the estimates of $y$ at the data point $x_i$ obtained from these two estimates of $\theta^*$, i.e., $\lvert{\hat{\theta}}^{\text{T}}_n x_i - {\hat{\theta}}^{\text{T}}_{[i]} x_i\rvert^2$?
  2. bound the squared difference between the predictions of $y$ at a new observation of $x = \bar{x}$ obtained from these two estimates, i.e.,$\lvert{\hat{\theta}}^{\text{T}}_n \bar{x} - {\hat{\theta}}^{\text{T}}_{[i]} \bar{x}\rvert^2$?

In particular, I would like to derive expressions for the above differences that show that they vanish as $n \to \infty$ (say in expectation), which is what I expect under reasonable assumptions.

What I know: $\hat{\theta}_n - \hat{\theta}_{[i]}$ can be expressed in terms of leverages using some linear algebra tricks. I am not sure how to simplify the resulting expressions to show that the difference between the two predictions vanishes in the limit.

Edit: Partial progress for part 1.

Let $X$ denote the design matrix for x (with the $i^{\text{th}}$ row of $X$ equal to $x^{\text{T}}_i$), and $H := X (X^{\text{T}} X)^{-1} X^{\text{T}}$ denote the hat/projection matrix. Then from the above link, we know that $$\lvert{\hat{\theta}}^{\text{T}}_n x_i - {\hat{\theta}}^{\text{T}}_{[i]} x_i\rvert^2 = \frac{(H_{ii})^2 (y_i - \hat{\theta}^{\text{T}}_n x_i)^2}{(1 - H_{ii})^2},$$ where $H_{ii}$, which denotes the $(i,i)^{\text{th}}$ element of $H$, is the leverage score for the $i^{\text{th}}$ observation. As pointed out by Michael in the comments, we know that $0 \leq H_{ii} \leq 1$ and $\sum_{i=1}^{n} H_{ii} = p$, where $p$ is the dimension of $x$, under mild assumptions on the random variable $x$. We can show that $\mathbb{E}[(y_i - \hat{\theta}^{\text{T}}_n x_i)^2]$ converges to $\mathbb{E}[\varepsilon^2]$ as $n \to \infty$ under mild assumptions, so it suffices to establish $H_{ii} \to 0$ as $n \to \infty$. I am not sure how to conclude this is true irrespective of the design for every $i \in \{1,\cdots,n\}$. I guess what we can readily say is that, on average, $H_{ii} \to 0$ as $n \to \infty$.

  • $\begingroup$ Hint: The sum of all leverages equals the number of regressors, no matter the sample size. $\endgroup$ – Michael M Oct 14 '19 at 17:53
  • $\begingroup$ @MichaelM I added some progress, but am not sure how to finish the proof with your hint. Can you please expand on your hint? $\endgroup$ – madnessweasley Oct 15 '19 at 16:05
  • 1
    $\begingroup$ I think you have added the hint in a neat way. For instance, if you have a single categorical covariable with m levels and one level appears only twice, then its leverage will stay 0.5, no matter how large n gets. And the difference between the insample residual and its leave-one-out version can be very high. $\endgroup$ – Michael M Oct 15 '19 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.