# Multiple Linear Regression Coefficient Estimators and Hypothesis Testing

A multiple linear regression model is considered. It is assumed that $$Y_i = \beta_1x_{i1} + \beta_2x_{i2} + \beta_3x_{i3} + \epsilon_i$$ where $$\epsilon$$-s are independent and have the same normal distribution with zero expectation and unknown variance $$\sigma^2$$. 100 measurements are made, i.e $$i = 1,2,..., 100.$$ The explanatory variables take the following values: $$x_{i1} = 2$$ for $$1 \leq i \leq 25$$ and $$0$$ otherwise, $$x_{i2} = \sqrt{2}$$ for $$26 \leq i \leq 75$$ and $$0$$ otherwise, $$x_{i3} = 2$$ for $$76 \leq i \leq 100$$ and $$0$$ otherwise.

a) Let $$\hat{\beta_1},\hat{\beta_2}, \hat{\beta_3}$$ be least squares estimators of $$\beta_1, \beta_2, \beta_3$$. Prove that in the considered case $$\hat{\beta_1},\hat{\beta_2}, \hat{\beta_3}$$ are independent, and $$Var(\hat{\beta_1}) = Var(\hat{\beta_3}) = Var(\hat{\beta_3})$$ Do these properties hold in the general case? If not, give counterexamples.

b) Perform a test for $$H_0: \beta_1 + \beta_3 = 2\beta_2$$vs.$$H_1: \beta_1 + \beta_3 \neq 2\beta_2$$ The significance level is 0.05. The least squares estimates of $$\beta_1, \beta_2$$ and $$\beta_3$$ are $$0.9812, 1.8851$$ and $$3.4406$$, respectively. The unbiased estimate of the variance $$\sigma^2$$ is $$3.27$$.

For a) I know the OLS estimator for $$\hat{\beta} = (X^TX)^{-1}X^Ty$$, and $$Var(\hat{\beta}) = \sigma^2 (X^TX)^{-1}$$. But I don't know how to attain explicit expressions for each of the coefficients from this. Although it seems quite clear that the estimators are independent, for instance $$P(\hat{\beta_3} = \beta_3, \hat{\beta_1} = 0, \hat{\beta_2} = 0) = P(\hat{\beta_3} = \beta_3)$$ but I don't how to write a proper proof. I believe the estimators are generally dependent and have unequal variance, but can't come up with any particular examples.

For b) not sure what test-statistic to use (t or F) and how to set it up. Also don't know the standard errors of the coefficients

• Looks like a homework question. If it is homework, please add the self-study tag and read the tag wiki. – StubbornAtom Apr 12 '20 at 13:55
• For part (b) you can also check out general linear hypothesis. – StubbornAtom Apr 17 '20 at 19:41

You have already given the formula for the variance matrix for the coefficient estimator. The Gramian matrix of the design matrix for the regression ---which appears in that formula--- is:

\begin{aligned} \mathbf{x}^\text{T} \mathbf{x} &= \begin{bmatrix} \mathbf{x}_1 \cdot \mathbf{x}_1 & \mathbf{x}_1 \cdot \mathbf{x}_2 & \mathbf{x}_1 \cdot \mathbf{x}_3 \\ \mathbf{x}_2 \cdot \mathbf{x}_1 & \mathbf{x}_2 \cdot \mathbf{x}_2 & \mathbf{x}_2 \cdot \mathbf{x}_3 \\ \mathbf{x}_3 \cdot \mathbf{x}_1 & \mathbf{x}_3 \cdot \mathbf{x}_2 & \mathbf{x}_3 \cdot \mathbf{x}_3 \\ \end{bmatrix} \\[6pt] &= \begin{bmatrix} \sum_{i=1}^{25} 2 \times 2 & 0 & 0 \\ 0 & \sum_{i=26}^{75} \sqrt{2} \times \sqrt{2} & 0 \\ 0 & 0 & \sum_{i=76}^{100} 2 \times 2 \\ \end{bmatrix} \\[6pt] &= \begin{bmatrix} 25 \times 4 & 0 & 0 \\ 0 & 50 \times 2 & 0 \\ 0 & 0 & 25 \times 4 \\ \end{bmatrix} \\[6pt] &= 100 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}. \\[6pt] \end{aligned}

This is proportionate to the identity matrix, which means that the coefficient estimators are uncorrelated with equal variance. You therefore have:

$$\mathbb{V}(\hat{\boldsymbol{\beta}}) = \sigma^2 (\mathbf{x}^\text{T} \mathbf{x})^{-1} = \frac{\sigma^2}{100} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}. \\[6pt]$$

You should be able to derive the coefficient estimates from the standard regression formula and then use these to formulate the hypothesis tests. Each hypothesis test is testing a linear combination of the coefficients, so you can use the rules for linear combinations of normal random variables to derive the standard errors of the test statistics.

• Is it possible to prove that the coefficient estimators being uncorrelated implies they are independent in this case? Or would you have to use a different method to prove independence? – Pame Apr 17 '20 at 12:38
• @Pame It’s a property of Multivariate Normal Distribution that if the components are pairwise uncorrelated then they are Independent. You can even verify this by putting $\rho = 0$ for a bivariate normal distribution and see that it reduces into a product of 2 univariate normal distribution. – Vishaal Sudarsan Apr 17 '20 at 13:48
• Are the $\beta$ coefficients generally multivariate normal for multiple linear regression, or is it special for this case? I understand they are univariate normal due to the residuals being normal. – Pame Apr 24 '20 at 9:21

As You know that the OLS estimator is a linear function of $$y$$. $$\hat{\beta} \sim N(\beta, \sigma^2(X’X)^{-1})$$ since $$\epsilon \sim N(0,\sigma^2)$$.

1) All you need to show is that the matrix $$(X’X)^{-1}$$ is a scalar matrix. Just compute the inverse using appropriate entries for $$x_1,x_2,x_3$$

2) All you need to do is test the following Hypothesis :-

$$H_0 : l’{\beta} = 0$$ vs $$H_1 : l’{\beta} \neq 0$$

Where $$l’$$ is (1,-2,1).

Now use a $$t$$ test on $$l’\hat{\beta}$$ which follows a univariate normal distribution.

• For 2) with the t-statistic don´t you get $SE(l' \beta$) in the denominator which is $0$? – Pame Apr 12 '20 at 17:02
• @Pame I haven’t done the calculations but the unbiased estimate of variance is given to be 3.27 and $Var(\hat{\beta_1} -2\hat{\beta_2} + \hat{\beta_3}) = 6*Var(\hat{\beta_1})$ since the variance is same for all \hat{\beta_i} and they’re independent as shown in 1). so your Standard error would be 0 only if the $Var(\hat{\beta})$ is 0. Which will happen only if $(X’X)^{-1}$ which is impossible. So to answer your question, the denominator won’t be 0 you must be going wrong somewhere. – Vishaal Sudarsan Apr 12 '20 at 18:39