4
$\begingroup$

A multiple linear regression model is considered. It is assumed that $$ Y_i = \beta_1x_{i1} + \beta_2x_{i2} + \beta_3x_{i3} + \epsilon_i$$ where $\epsilon$-s are independent and have the same normal distribution with zero expectation and unknown variance $\sigma^2$. 100 measurements are made, i.e $i = 1,2,..., 100.$ The explanatory variables take the following values: $x_{i1} = 2$ for $1 \leq i \leq 25$ and $0$ otherwise, $x_{i2} = \sqrt{2}$ for $26 \leq i \leq 75$ and $0$ otherwise, $x_{i3} = 2$ for $76 \leq i \leq 100$ and $0$ otherwise.

a) Let $\hat{\beta_1},\hat{\beta_2}, \hat{\beta_3}$ be least squares estimators of $\beta_1, \beta_2, \beta_3$. Prove that in the considered case $\hat{\beta_1},\hat{\beta_2}, \hat{\beta_3}$ are independent, and $$Var(\hat{\beta_1}) = Var(\hat{\beta_3}) = Var(\hat{\beta_3})$$ Do these properties hold in the general case? If not, give counterexamples.

b) Perform a test for $$H_0: \beta_1 + \beta_3 = 2\beta_2$$vs.$$H_1: \beta_1 + \beta_3 \neq 2\beta_2$$ The significance level is 0.05. The least squares estimates of $\beta_1, \beta_2$ and $\beta_3$ are $0.9812, 1.8851$ and $3.4406$, respectively. The unbiased estimate of the variance $\sigma^2$ is $3.27$.

For a) I know the OLS estimator for $\hat{\beta} = (X^TX)^{-1}X^Ty$, and $Var(\hat{\beta}) = \sigma^2 (X^TX)^{-1}$. But I don't know how to attain explicit expressions for each of the coefficients from this. Although it seems quite clear that the estimators are independent, for instance $P(\hat{\beta_3} = \beta_3, \hat{\beta_1} = 0, \hat{\beta_2} = 0) = P(\hat{\beta_3} = \beta_3)$ but I don't how to write a proper proof. I believe the estimators are generally dependent and have unequal variance, but can't come up with any particular examples.

For b) not sure what test-statistic to use (t or F) and how to set it up. Also don't know the standard errors of the coefficients

$\endgroup$
2
  • 3
    $\begingroup$ Looks like a homework question. If it is homework, please add the self-study tag and read the tag wiki. $\endgroup$ – StubbornAtom Apr 12 '20 at 13:55
  • 1
    $\begingroup$ For part (b) you can also check out general linear hypothesis. $\endgroup$ – StubbornAtom Apr 17 '20 at 19:41
4
+50
$\begingroup$

You have already given the formula for the variance matrix for the coefficient estimator. The Gramian matrix of the design matrix for the regression ---which appears in that formula--- is:

$$\begin{aligned} \mathbf{x}^\text{T} \mathbf{x} &= \begin{bmatrix} \mathbf{x}_1 \cdot \mathbf{x}_1 & \mathbf{x}_1 \cdot \mathbf{x}_2 & \mathbf{x}_1 \cdot \mathbf{x}_3 \\ \mathbf{x}_2 \cdot \mathbf{x}_1 & \mathbf{x}_2 \cdot \mathbf{x}_2 & \mathbf{x}_2 \cdot \mathbf{x}_3 \\ \mathbf{x}_3 \cdot \mathbf{x}_1 & \mathbf{x}_3 \cdot \mathbf{x}_2 & \mathbf{x}_3 \cdot \mathbf{x}_3 \\ \end{bmatrix} \\[6pt] &= \begin{bmatrix} \sum_{i=1}^{25} 2 \times 2 & 0 & 0 \\ 0 & \sum_{i=26}^{75} \sqrt{2} \times \sqrt{2} & 0 \\ 0 & 0 & \sum_{i=76}^{100} 2 \times 2 \\ \end{bmatrix} \\[6pt] &= \begin{bmatrix} 25 \times 4 & 0 & 0 \\ 0 & 50 \times 2 & 0 \\ 0 & 0 & 25 \times 4 \\ \end{bmatrix} \\[6pt] &= 100 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}. \\[6pt] \end{aligned}$$

This is proportionate to the identity matrix, which means that the coefficient estimators are uncorrelated with equal variance. You therefore have:

$$\mathbb{V}(\hat{\boldsymbol{\beta}}) = \sigma^2 (\mathbf{x}^\text{T} \mathbf{x})^{-1} = \frac{\sigma^2}{100} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}. \\[6pt] $$

You should be able to derive the coefficient estimates from the standard regression formula and then use these to formulate the hypothesis tests. Each hypothesis test is testing a linear combination of the coefficients, so you can use the rules for linear combinations of normal random variables to derive the standard errors of the test statistics.

$\endgroup$
3
  • $\begingroup$ Is it possible to prove that the coefficient estimators being uncorrelated implies they are independent in this case? Or would you have to use a different method to prove independence? $\endgroup$ – Pame Apr 17 '20 at 12:38
  • 1
    $\begingroup$ @Pame It’s a property of Multivariate Normal Distribution that if the components are pairwise uncorrelated then they are Independent. You can even verify this by putting $\rho = 0$ for a bivariate normal distribution and see that it reduces into a product of 2 univariate normal distribution. $\endgroup$ – Vishaal Sudarsan Apr 17 '20 at 13:48
  • $\begingroup$ Are the $\beta$ coefficients generally multivariate normal for multiple linear regression, or is it special for this case? I understand they are univariate normal due to the residuals being normal. $\endgroup$ – Pame Apr 24 '20 at 9:21
1
$\begingroup$

As You know that the OLS estimator is a linear function of $y$. $\hat{\beta} \sim N(\beta, \sigma^2(X’X)^{-1})$ since $\epsilon \sim N(0,\sigma^2)$.

1) All you need to show is that the matrix $(X’X)^{-1}$ is a scalar matrix. Just compute the inverse using appropriate entries for $x_1,x_2,x_3$

2) All you need to do is test the following Hypothesis :-

$H_0 : l’{\beta} = 0$ vs $H_1 : l’{\beta} \neq 0$

Where $l’$ is (1,-2,1).

Now use a $t$ test on $l’\hat{\beta}$ which follows a univariate normal distribution.

$\endgroup$
2
  • $\begingroup$ For 2) with the t-statistic don´t you get $SE(l' \beta$) in the denominator which is $0$? $\endgroup$ – Pame Apr 12 '20 at 17:02
  • $\begingroup$ @Pame I haven’t done the calculations but the unbiased estimate of variance is given to be 3.27 and $Var(\hat{\beta_1} -2\hat{\beta_2} + \hat{\beta_3}) = 6*Var(\hat{\beta_1})$ since the variance is same for all \hat{\beta_i} and they’re independent as shown in 1). so your Standard error would be 0 only if the $Var(\hat{\beta})$ is 0. Which will happen only if $(X’X)^{-1}$ which is impossible. So to answer your question, the denominator won’t be 0 you must be going wrong somewhere. $\endgroup$ – Vishaal Sudarsan Apr 12 '20 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.