5
$\begingroup$

I have a simple linear OLS regression $Y_i = \alpha+ \beta_1 X_{1i} + \beta_2 X_{2i} + e_i$ where $e_i \sim N(0,\sigma)$. I have estimated the regression from the data and obtained estimates for my coefficients as well as the corresponding covariance matrix. Assume my dataset is large (>500)

Now I would like to construct the 95% confidence interval for $Z = a_1 \beta_1 + a_2 \beta_2$. My first thought was that I could assume that Z follows a normal distribution because my regression coefficients are approximately normal and a linear combination of normal variables is also a normal variable. I do find some sources where they treat $Z$ as a normal random variable.

But now I am in doubt because I found that a linear combination of normal random variables is only normal if they are independent. So is it still save to assume $Z$ is random?

$\endgroup$
  • 2
    $\begingroup$ Your coefficients are constants, so I wonder what you mean by saying that they "are approximately normal". At any rate, your coefficients are a function of your data (viz. your $Y$ values), which are random. Thus, they are random, & functions of them ($Z$) are random in turn. I think you meant to ask if it is safe to assume they are independent. This will turn out to be true if $X_1$ & $X_2$ are independent. However, a linear combination of normal random variables is still random if the variables are not independent, you just have to use a more complicated formula. $\endgroup$ – gung Jan 13 '15 at 22:50
  • $\begingroup$ I actually meant the sampling distribution of the estimates. I actually want to construct the 95% CI of $z = \beta_1 + \beta_2$. My approach was to determine the expected value as the sum of the estimates and the variance of z with the formula of the variance for correlated variables ($\operatorname{Var}\left(\sum_{i=1}^n X_i\right) = \sum_{i=1}^n \sum_{j=1}^n \operatorname{Cov}(X_i, X_j) = \sum_{i=1}^n \operatorname{Var}(X_i) + 2\sum_{1\le i}\sum_{<j\le n}\operatorname{Cov}(X_i,X_j)$. In order to construct a CI, I need to know if the distribution of Z is normal. $\endgroup$ – user58571 Jan 13 '15 at 23:03
  • $\begingroup$ Related: stats.stackexchange.com/questions/16724/… $\endgroup$ – Andrew M Jan 13 '15 at 23:13
4
$\begingroup$

It's actually not true that your variables need to be independent so that their sum will be normal.

If X and Y are jointly normally distributed with mean $\mu_{1} $ and $\mu_{2} $ and variance $\sigma_{1}^{2}$ and $\sigma_{2}^{2}$ with correlation $\rho $ then Z is still normally distributed with mean $\mu_{1}+\mu_{2} $ and variance $\sqrt { \sigma_{2}^{2}+ \sigma_{1}^{2}+2 \rho \sigma_{2} \sigma_{1}}$ Hopefully that gives you what you need.

$\endgroup$
  • $\begingroup$ This is not true. Let X be N(0,1) and Y be exactly -X. Clearly Y is also N(0,1), but X+Y = 0 almost surely, which is NOT normal. $\endgroup$ – Christopher Aden Jan 13 '15 at 22:51
  • $\begingroup$ Sorry, I should have included that they should be jointly normal. $\endgroup$ – Nick Thieme Jan 13 '15 at 22:51
  • $\begingroup$ A necessary and sufficient condition is that their Covariance matrix is positive definite. That prevents my counterexample and supports your claim. $\endgroup$ – Christopher Aden Jan 13 '15 at 22:54
  • $\begingroup$ Ok, but then the question remains if one can assume that the regression coefficients of an OLS regression are jointly normally distributed? If so, any pointers to some source which derives/explains this assumption would be great $\endgroup$ – user58571 Jan 13 '15 at 23:09
  • 1
    $\begingroup$ @ChristopherAden A necessary condition of OLS is that the covariance matrix is positive definite, if this were not true you could not even estimate the coefficients. Thus under the assumptions of OLS where the error are i.i.d normal, any linear combinations of estimated coefficients is also normal with mean and variance calculated similar to as suggested above. $\endgroup$ – Zachary Blumenfeld Jan 13 '15 at 23:10
6
$\begingroup$

it would be better to explain this in matrix notation. Suppose the general Gauss-Markov linear model $$\mathbf{y = X \boldsymbol \beta + \boldsymbol \epsilon}$$For your case, $\mathbf{X}$ = ($\mathbf{1}$, $\mathbf{x_1}$, $\mathbf{x_2}$) and $\mathbf{\boldsymbol \beta} = (\alpha, \; \beta_1 \; \beta_2)'.$ The OLS estimator is $$\hat{\boldsymbol \beta} = \left(\mathbf{X}'\mathbf{X} \right)^{-1}\mathbf{X}'\mathbf{y}.$$ Since $\mathbf{y} \ \sim \ N(\mathbf{X \boldsymbol \beta}, \sigma^2)$, then $$\hat{\boldsymbol \beta} \ \sim \ N \left(\mathbf{X \boldsymbol \beta}, \sigma^2(\mathbf{X}'\mathbf{X})^{-1} \right).$$ Based on the above result, you are able to derive the distribution of $Z = \alpha_1 \beta_1 + \alpha_2 \beta_2$ if you write $Z = \boldsymbol \lambda' \boldsymbol \beta$, where $\lambda' = (0, \alpha_1, \alpha_2)$. Denote correspondingly that $$\hat{Z} = \alpha_1 \hat{\beta}_1 + \alpha_2 \hat{\beta}_2 = \boldsymbol \lambda' \hat{\boldsymbol \beta},$$ then you can easily get that $$\frac{\hat{Z} - Z}{\sigma\sqrt{\boldsymbol \lambda' (\mathbf{X}' \mathbf{X})^{-1} \boldsymbol \lambda}} \ \sim \ N(0, 1).$$ Note that $\sigma$ is unknown, so you cannot construct the confidence interval based on the normal distribution. A rigorous approach is to derive the confidence interval based on a $t$-test statistic (I just ignore the derivation but give the result, you should be able to find details in any linear model book or through website). Specifically,$$\frac{\hat{Z} - Z}{\hat{\sigma} \sqrt{\boldsymbol \lambda' (\mathbf{X}' \mathbf{X})^{-1} \boldsymbol \lambda}} \ \sim \ t(n-p),$$ where $\hat{\sigma}^2 = \frac{\mathbf{y}'(\mathbf{I - P_X}) \mathbf{y}}{n-p}$, $\mathbf{P_X}$ is the projection matrix, and $p = rank(\mathbf{X})$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.