Consider the standard regression model with i.i.d. observations $(X_i,Y_i)$ for $i=1,2,\dots,n$:

$$ Y_i = \beta_0 + \beta_1 X_{i} + \varepsilon_i, \quad \quad i = 1,2,\dots,n, $$ where the regressors $X_i$ are considered to be random variables as opposed to fixed observations, and the errors are normally distributed conditional on the regressors and have fixed variance.

Suppose we solve this model using ordinary least squares and obtain estimated coefficients $\hat \beta_0, \hat \beta_1$.

Now define $n$ new random variables $$ Z_i = \hat \beta_0 + \hat \beta_1 X_{i}, \quad \quad i=1,2,\dots,n. $$

How do we calculate $\text{Var}[\sum_{i=1}^n Z_i]$? I'm not sure if $Z_i$ are independent because they are constructed using $\hat \beta_0$ and $\hat \beta_1$ which makes it seem like the $Z_i$ could be dependent on each other?

Note: I want to treat $\hat \beta_0$ and $\hat \beta_1$ as random. Wikipedia says these estimates are normally distributed since the errors are normally distributed.

  • 1
    $\begingroup$ Are these "$X_i$" in the new variables the very same as the random variables used to model your observations, or are they intended to represent new independent variables with the same distributions?? Do you want to compute the full variance of the sum of the $Z_i,$ or the variance conditional on the $X_i$ (a natural thing to do), or the variance conditional on the estimates $\hat\beta_i$ (a little strange, but still mathematically meaningful)? $\endgroup$ – whuber Oct 28 '20 at 17:41
  • $\begingroup$ I don't know if this relates to the application you have in mind, but have a look at the entry about "random effects models" on Wikipedia. $\endgroup$ – StijnDeVuyst Oct 28 '20 at 19:05
  • $\begingroup$ @whuber The $X_i$ are exactly the same as the random variables used to model the observations. I am interested in the full variance of the sum of the $Z_i$'s since then randomness in both the $X_i$'s and the $\beta_i$'s gets fully accounted for. $\endgroup$ – Bertus101 Oct 28 '20 at 19:06
  • $\begingroup$ This looks like you are seeking a way to estimate the variance of $\sum_i \beta_0+\beta_1 X_i.$ If so, there might be much simpler ways to do so, such by estimating the variance of $\sum Y_i$ directly from the data and subtracting a suitable multiple of the estimated variance of the $\varepsilon_i.$ $\endgroup$ – whuber Oct 28 '20 at 19:09
  • $\begingroup$ Although it might appear like that, I really am interested in the variance of $\sum_i \hat \beta_0 + \hat \beta_1 X_i$ and not $\sum_i \beta_0 + \beta_1 X_i$. $\endgroup$ – Bertus101 Oct 28 '20 at 19:13

If the regression coefficients are estimated using OLS and the variance of the errors is $\sigma^2$, then $$ \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \end{bmatrix} |X \sim N\Big( \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix} , \sigma^2\begin{bmatrix} n & \sum_i X_i\\ \sum_i X_i & \sum_i X_i^2 \end{bmatrix}^{-1} \Big). $$ where $X=(X_1...X_n)$. Now, let $$ Z = \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \end{bmatrix}' \begin{bmatrix} n \\ \sum_i X_i \end{bmatrix}. $$ Then using the law of total variance $$ Var(Z) = Var(E(Z|X)) + E(Var(Z|X)) $$ $$ =Var\Big(\begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix}' \begin{bmatrix} n \\ \sum_i X_i \end{bmatrix}\Big) + E\Big( \sigma^2 \begin{bmatrix} n \\ \sum_i X_i \end{bmatrix}'\begin{bmatrix} n & \sum_i X_i\\ \sum_i X_i & \sum_i X_i^2 \end{bmatrix}^{-1}\begin{bmatrix} n \\ \sum_i X_i \end{bmatrix} \Big). $$ If the variance of $X$ is $\sigma_x^2I_n$ then the first term above is $n\beta_1^2\sigma_x^2$, and I believe the second term is $n\sigma_2^2$. So the answer to your question is $$ n(\beta_1^2\sigma_x^2 + \sigma_2^2). $$

  • $\begingroup$ In proving consistency for standard linear regression, the assumption is made that $\text{plim}_{n\to \infty} \frac{X'X}{n} = Q$ where $Q$ is a positive definite matrix. Could this assumption be factored into your derivation somehow to show that the variance is bounded as $n \to \infty$? Because the result you give seems a bit troublesome since it says the variance diverges as $n \to \infty$. $\endgroup$ – Bertus101 Oct 29 '20 at 11:55
  • 1
    $\begingroup$ My answer assumes that X'X is invertible and thus positive definite, which is definitely true for large n. Also, why would the variance converge? The mean of Zs would converge, but I don't see why its sum would. $\endgroup$ – Carlos Llosa Oct 29 '20 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.