Finding OLS estimator for $\beta$ where $y_i=\beta+ 2 \beta x_i+\epsilon_i$

Consider the following model with the usual OLS assumptions: $$\epsilon_i$$ are uncorrelated random variables with mean zero and constant variance $$\sigma^2$$.

$$y_i=\beta+ 2 \beta x_i+\epsilon_i$$

$$(a)$$Derive the least squares estimator for $$\beta$$

$$(b)$$Compute $$\mathsf{E}(\hat{\beta})$$

$$(c)$$ Compute $$\mathsf{Var}(\hat{\beta})$$

Attempted Solutions:

$$(a)$$ Let $$S(\beta)=\sum(y_i-(\beta+2\beta x_i))^2$$

Then set

$$\frac{\partial S}{\partial \beta}=2\sum(y_i-\beta-2\beta x_i)(-1-2x_i)=0$$

From here, I get that

\begin{align*} \frac{\partial S}{\partial \beta} &=2\sum(y_i-\beta-2\beta x_i)(-1-2x_i)\\\\ &=2\sum(-y_i+\beta+4\beta x_i-2y_i x_i+4\beta x_i^2)\\\\ &=2\left(\sum-y_i-2y_ix_i+\beta\sum(1+4x_i+4x_i^2)\right) \end{align*}

Thus,

$$\hat{\beta}=\frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2}$$

Checking that this is, in fact, a minimum,

$$\frac{\partial}{\partial\beta}\left(2\sum-y_i-2y_ix_i+2\beta\sum1+4x_i+4x_i^2\right)=2\gt0 \text{ }\checkmark$$

Is that valid?

For $$(b)$$ and $$(c)$$ I am stumped. I think I need to figure out what can be treated as constants within my estimate and what needs to be treated as random variables.

My first thought for $$(b)$$ is to note that

$$\mathsf E\left(\frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2}\right)=\mathsf E\left(\frac{\sum (\beta+2\beta x_i+\epsilon_i)+2x_i(\beta+2\beta x_i+\epsilon_i)}{\sum1+4x_i+4x_i^2}\right)$$

but I don't see how that would help me.

Edit:

I have

\begin{align*} \mathsf{Var}\left(\frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2}\right) &=\mathsf{Var}\left(\frac{\sum (\beta+2\beta x_i+\epsilon_i)+2x_i(\beta+2\beta x_i+\epsilon_i)}{\sum1+4x_i+4x_i^2}\right)\\\\ &=\mathsf{Var}\left(\frac{\sum \epsilon_i+2x_i\epsilon_i+\sum\beta+4\beta x_i +2\beta x_i^2}{\sum1+4x_i+4x_i^2}\right)\\\\ &=\mathsf{Var}\left(\frac{\sum \epsilon_i+2x_i\epsilon_i}{\sum1+4x_i+4x_i^2}+\frac{\sum\beta+4\beta x_i +2\beta x_i^2}{\sum1+4x_i+4x_i^2}\right)\\\\ &=\mathsf{Var}\left(\frac{\sum \epsilon_i+2x_i\epsilon_i}{\sum1+4x_i+4x_i^2}\right)\\\\ &=\mathsf{Var}\left(\frac{\sum\epsilon_i(1+2x_i)}{\sum(1+2x_i)^2}\right) \end{align*}

but I cannot isolate the $$\epsilon_i$$, the only random variable remaining.

• Hint: You can simplify your calculations by defining $t_i:=1+2x_i$. Sep 21 '18 at 4:39
• I believe my result for the variance is very close to Ahmad's now. However, I'm not sure how I can isolate the $\epsilon_i$. I can't manipulate this further by bringing it outside of the sum, can I?
– Remy
Sep 21 '18 at 4:46
• You can't bring $\epsilon_i$ outside of the sum, but you can bring out $\sigma^2$. See my answer below. Sep 21 '18 at 4:57

To finish off, write $$\operatorname{Var}\left(\frac{\sum\epsilon_it_i}{\sum t_i^2}\right)\stackrel{(1)}=\frac{\operatorname{Var}(\sum\epsilon_it_i)}{(\sum t_i^2)^2}\stackrel{(2)}=\frac{\sum\sigma^2t_i^2}{(\sum t_i^2)^2}\stackrel{(3)}=\frac{\sigma^2\sum t_i^2}{(\sum t_i^2)^2}\stackrel{(4)}=\frac{\sigma^2}{\sum t_i^2}.$$ I've written $$t_i:=1+2x_i$$ for brevity. Step (1) is the rule $$\operatorname{Var}(cX)=c^2\operatorname{Var}(X)$$. Step (2) is using the uncorrelatedness of the $$\epsilon$$'s. Step (3) is where you can pull out the constant $$\sigma^2$$ from the summation. Step (4) is cancelling a common factor from top and bottom.

If you use the transformed explanatory variable $$x_{*i} = 1+2x_i$$ then the model easily simplifies to a simple linear regression without an intercept term:

$$y_i = \beta x_{*i} + \epsilon_i \quad \quad \quad \epsilon_i \sim \text{IID N}(0, \sigma^2).$$

From here, all the usual mathematical results for this linear regression hold. In particular, the OLS estimator is unbiased, with variance given by the usual formula. Specific results are below.

Since this is a simple linear regression (without an intercept) you have OLS estimator given by:

$$\hat{\beta} = \frac{\sum x_{*i} Y_i}{\sum x_{*i}^2} = \frac{\sum (1+2x_i) Y_i}{\sum (1+2x_i)^2} = \frac{\sum (1+2x_i) Y_i}{\sum (1+4x_i + 4x_i^2)}.$$

Since $$\mathbb{E}(Y_i) = \beta x_{*i} = \beta (1+2x_i)$$ you have:

$$\mathbb{E}(\hat{\beta}) = \frac{\sum (1+2x_i) \mathbb{E}(Y_i)}{\sum (1+2x_i)^2} = \beta \frac{\sum (1+2x_i)^2}{\sum (1+2x_i)^2} = \beta.$$

Since $$\mathbb{V}(Y_i) = \sigma^2$$ you have:

$$\mathbb{V}(\hat{\beta}) = \frac{\sum (1+2x_i)^2 \mathbb{V}(Y_i)}{(\sum (1+2x_i)^2)^2} = \frac{\sigma^2}{\sum (1+2x_i)^2}.$$

Part $$(a)$$ seems legit.

Computing $$E(\hat{\beta})$$

Taking it from where you stopped $$\begin{equation} \hat{\beta}=\frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2} \end{equation}$$ Then $$\begin{equation} E\hat{\beta}=\frac{\sum Ey_i+2x_iEy_i}{\sum1+4x_i+4x_i^2} \end{equation}$$ We know that $$\begin{equation} E(y_i) = E\beta+ E2 \beta x_i+E\epsilon_i= \beta+ 2 \beta x_i \end{equation}$$ So $$\begin{equation} E\hat{\beta}=\frac{\sum \beta+ 2 \beta x_i+2x_i(\beta+ 2 \beta x_i)}{\sum1+4x_i+4x_i^2} \end{equation}$$ that is $$\begin{equation} E\hat{\beta}=\frac{\sum \beta+ 4 \beta x_i+ 4 \beta x_i^2}{\sum1+4x_i+4x_i^2} = \beta \Bigg( \frac{\sum1+4x_i+4x_i^2}{\sum1+4x_i+4x_i^2} \Bigg) = \beta \end{equation}$$

Computing var $$\hat{\beta}$$

Note that $$\begin{equation} \operatorname{var} \hat{\beta} = E(\hat{\beta}-\beta)^2 = E( \frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2} - \beta)^2 \end{equation}$$ that is $$\begin{equation} \operatorname{var} \hat{\beta} = E\Big( \frac{\sum y_i+2y_ix_i -\beta-4x_i\beta-4x_i^2\beta }{\sum1+4x_i+4x_i^2} \Big)^2 \end{equation}$$ Using $$y_i=\beta+ 2 \beta x_i+\epsilon_i$$, we get $$\begin{equation} \operatorname{var} \hat{\beta} = E\Big( \frac{\sum \beta+ 2 \beta x_i+\epsilon_i+2(\beta+ 2 \beta x_i+\epsilon_i)x_i -\beta-4x_i\beta-4x_i^2\beta }{\sum1+4x_i+4x_i^2} \Big)^2 \end{equation}$$ that is $$\begin{equation} \operatorname{var} \hat{\beta} = E\Big( \frac{\sum \epsilon_i+2\epsilon_ix_i }{\sum1+4x_i+4x_i^2} \Big)^2 \end{equation}$$ that is $$\begin{equation} \operatorname{var} \hat{\beta} = \frac{E(\sum \epsilon_i+2\epsilon_ix_i )^2}{(\sum1+4x_i+4x_i^2)^2} \tag{1} \end{equation}$$ Consider the numerator of the above expression, $$\begin{equation} E(\sum \epsilon_i+2\epsilon_ix_i )^2 = E(\sum_i\epsilon_i+2\epsilon_ix_i )(\sum_j \epsilon_j+2\epsilon_jx_j ) \end{equation}$$ which is $$\begin{equation} E(\sum \epsilon_i+2\epsilon_ix_i )^2 = \sum_{ij} E(\epsilon_i\epsilon_j)(1+2x_i)(1+2x_j) \end{equation}$$ But $$\begin{equation} E(\epsilon_i\epsilon_j)=\sigma^2 \delta(i-j) \end{equation}$$ where $$\delta(x) = 1$$ if $$x = 0$$ and $$0$$ otherwise. We will get $$\begin{equation} E(\sum \epsilon_i+2\epsilon_ix_i )^2 = \sum_{ij} \sigma^2 (1+2x_i)(1+2x_j)\delta(i-j) \end{equation}$$ So the sum boils down to $$\begin{equation} E(\sum \epsilon_i+2\epsilon_ix_i )^2 = \sum_{i=j} \sigma^2 (1+2x_i)(1+2x_j) = \sigma^2\sum_i (1+2x_i)^2 \end{equation}$$ Replace in equation $$(1)$$, we get $$\begin{equation} \operatorname{var} \hat{\beta} = \frac{\sigma^2\sum_i (1+2x_i)^2}{(\sum1+4x_i+4x_i^2)^2} \end{equation}$$ The denominator is also written as $$\begin{equation} \operatorname{var} \hat{\beta} = \frac{\sigma^2\sum_i (1+2x_i)^2}{(\sum_i (1+2x_i)^2)^2} = \sigma^2 \frac{1}{\sum_i (1+2x_i)^2} \end{equation}$$