1
$\begingroup$

Suppose I have an independent vector $X$ and a dependent scalar random variable $Y$ and I wish to construct a regression model to predict $Y$ using $X$ given data $\{(x_i,y_i)\}_{i=1}^{n}$. For concreteness, suppose the true relationship is linear with $Y = {\theta^*}^{\text{T}} X + \varepsilon$, where $\varepsilon$ are errors independent of $X$ with $\mathbb{E}[\varepsilon] = 0$, and my least squares multiple linear regression yields an estimate $\hat{\theta}_n$ of $\theta^*$.

Background: As I understand it from these lecture notes, there are typically two different ways of analyzing the performance of the regression estimates depending on whether we are in the fixed design ($X$ is deterministic and $\{x_i\}_{i=1}^{n}$ are chosen by us) or random design ($X$ is a random variable and $\{x_i\}_{i=1}^{n}$ are random draws from the distribution of $X$) setting. In the fixed design setting, we care about the mean-squared estimation error at the training data points $\dfrac{1}{n} \displaystyle\sum_{i=1}^{n} \left\lvert {\theta^*}^{\text{T}} x_i - \hat{\theta}^{\text{T}}_n x_i \right\rvert^2$, whereas in the random design setting, we probably care about the average mean-squared prediction error averaged over the distribution of $X$, i.e. $\mathbb{E}_X\left[ \left({\theta^* - \hat{\theta}_n}^{\text{T}}\right) X + \varepsilon \right]^2$.

Main question: Suppose we have a result for the fixed design setting, but the result does not place any restrictions on the choice of the design points $\{x_i\}_{i=1}^{n}$. To be concrete, consider Theorem 2.2 in these lecture notes, which concludes that for fixed design least squares regression, we have without any restriction on the design points $\{x_i\}_{i=1}^{n}$: $$\mathbb{E}\left[\dfrac{1}{n} \displaystyle\sum_{i=1}^{n} \left\lvert {\theta^*}^{\text{T}} x_i - \hat{\theta}^{\text{T}}_n x_i \right\rvert^2\right] \leq C \dfrac{\sigma^2 r}{n},$$

$$ \mathbb{P}\left( \dfrac{1}{n} \displaystyle\sum_{i=1}^{n} \left\lvert {\theta^*}^{\text{T}} x_i - \hat{\theta}^{\text{T}}_n x_i \right\rvert^2 > C\sigma^2\dfrac{r + \log\left( \frac{1}{\delta} \right)}{n} \right) \leq \delta ,$$

when the errors $\varepsilon$ are subGaussian with variance proxy $\sigma^2$, where $r$ is the rank of the design matrix $[x_1 \: x_2 \: \cdots \: x_n]$, $C > 0$ is a constant, and $\delta \in (0,1)$ is a reliability level.

Can I simply use the second probabilistic inequality above for the random design case since it holds conditionally for every random draw of the points $\{x_i\}_{i=1}^{n}$? Does the first inequality on the expected error hold as well by the law of iterated expectation? Do I need to assume that the draws $\{x_i\}_{i=1}^{n}$ are i.i.d., or do these hold for non-i.i.d. samples as well?

PS: My question is complementary to this and this.

$\endgroup$