# Difference between predictions of the OLS model and a leave-one-out model

Consider OLS regression with the true model $$y = {\theta^{*}}^{\text{T}} x + \varepsilon$$, where $$x$$ denotes the (deterministic) independent variables, $$y$$ denotes the dependent (random) variable, and $$\varepsilon$$ is the random error.

Let $$\hat{\theta}_n$$ denote the OLS estimate of $$\theta^*$$ from data $$\{(x_j,y_j)\}_{j=1}^{n}$$, and $$\hat{\theta}_{[i]}$$ denote the OLS estimate of $$\theta^*$$ with the $$i^{\text{th}}$$ data point removed.

How can I

1. bound the squared difference between the estimates of $$y$$ at the data point $$x_i$$ obtained from these two estimates of $$\theta^*$$, i.e., $$\lvert{\hat{\theta}}^{\text{T}}_n x_i - {\hat{\theta}}^{\text{T}}_{[i]} x_i\rvert^2$$?
2. bound the squared difference between the predictions of $$y$$ at a new observation of $$x = \bar{x}$$ obtained from these two estimates, i.e.,$$\lvert{\hat{\theta}}^{\text{T}}_n \bar{x} - {\hat{\theta}}^{\text{T}}_{[i]} \bar{x}\rvert^2$$?

In particular, I would like to derive expressions for the above differences that show that they vanish as $$n \to \infty$$ (say in expectation), which is what I expect under reasonable assumptions.

What I know: $$\hat{\theta}_n - \hat{\theta}_{[i]}$$ can be expressed in terms of leverages using some linear algebra tricks. I am not sure how to simplify the resulting expressions to show that the difference between the two predictions vanishes in the limit.

Edit: Partial progress for part 1.

Let $$X$$ denote the design matrix for x (with the $$i^{\text{th}}$$ row of $$X$$ equal to $$x^{\text{T}}_i$$), and $$H := X (X^{\text{T}} X)^{-1} X^{\text{T}}$$ denote the hat/projection matrix. Then from the above link, we know that $$\lvert{\hat{\theta}}^{\text{T}}_n x_i - {\hat{\theta}}^{\text{T}}_{[i]} x_i\rvert^2 = \frac{(H_{ii})^2 (y_i - \hat{\theta}^{\text{T}}_n x_i)^2}{(1 - H_{ii})^2},$$ where $$H_{ii}$$, which denotes the $$(i,i)^{\text{th}}$$ element of $$H$$, is the leverage score for the $$i^{\text{th}}$$ observation. As pointed out by Michael in the comments, we know that $$0 \leq H_{ii} \leq 1$$ and $$\sum_{i=1}^{n} H_{ii} = p$$, where $$p$$ is the dimension of $$x$$, under mild assumptions on the random variable $$x$$. We can show that $$\mathbb{E}[(y_i - \hat{\theta}^{\text{T}}_n x_i)^2]$$ converges to $$\mathbb{E}[\varepsilon^2]$$ as $$n \to \infty$$ under mild assumptions, so it suffices to establish $$H_{ii} \to 0$$ as $$n \to \infty$$. I am not sure how to conclude this is true irrespective of the design for every $$i \in \{1,\cdots,n\}$$. I guess what we can readily say is that, on average, $$H_{ii} \to 0$$ as $$n \to \infty$$.

• Hint: The sum of all leverages equals the number of regressors, no matter the sample size. – Michael M Oct 14 '19 at 17:53
• @MichaelM I added some progress, but am not sure how to finish the proof with your hint. Can you please expand on your hint? – madnessweasley Oct 15 '19 at 16:05
• I think you have added the hint in a neat way. For instance, if you have a single categorical covariable with m levels and one level appears only twice, then its leverage will stay 0.5, no matter how large n gets. And the difference between the insample residual and its leave-one-out version can be very high. – Michael M Oct 15 '19 at 18:06