Finding OLS estimator for $\beta$ where $y_i=\beta+ 2 \beta x_i+\epsilon_i$

Question

Consider the following model with the usual OLS assumptions: $\epsilon_i$ are uncorrelated random variables with mean zero and constant variance $\sigma^2$.

$$y_i=\beta+ 2 \beta x_i+\epsilon_i$$

$(a)$Derive the least squares estimator for $\beta$

$(b)$Compute $\mathsf{E}(\hat{\beta})$

$(c)$ Compute $\mathsf{Var}(\hat{\beta})$

Attempted Solutions:

$(a)$ Let $S(\beta)=\sum(y_i-(\beta+2\beta x_i))^2$

Then set

$$\frac{\partial S}{\partial \beta}=2\sum(y_i-\beta-2\beta x_i)(-1-2x_i)=0$$

From here, I get that

$$\begin{align*} \frac{\partial S}{\partial \beta} &=2\sum(y_i-\beta-2\beta x_i)(-1-2x_i)\\\\ &=2\sum(-y_i+\beta+4\beta x_i-2y_i x_i+4\beta x_i^2)\\\\ &=2\left(\sum-y_i-2y_ix_i+\beta\sum(1+4x_i+4x_i^2)\right) \end{align*}$$

Thus,

$$\hat{\beta}=\frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2}$$

Checking that this is, in fact, a minimum,

$$\frac{\partial}{\partial\beta}\left(2\sum-y_i-2y_ix_i+2\beta\sum1+4x_i+4x_i^2\right)=2\gt0 \text{ }\checkmark$$

Is that valid?

For $(b)$ and $(c)$ I am stumped. I think I need to figure out what can be treated as constants within my estimate and what needs to be treated as random variables.

My first thought for $(b)$ is to note that

$$\mathsf E\left(\frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2}\right)=\mathsf E\left(\frac{\sum (\beta+2\beta x_i+\epsilon_i)+2x_i(\beta+2\beta x_i+\epsilon_i)}{\sum1+4x_i+4x_i^2}\right)$$

but I don't see how that would help me.

Edit:

I have

$$\begin{align*} \mathsf{Var}\left(\frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2}\right) &=\mathsf{Var}\left(\frac{\sum (\beta+2\beta x_i+\epsilon_i)+2x_i(\beta+2\beta x_i+\epsilon_i)}{\sum1+4x_i+4x_i^2}\right)\\\\ &=\mathsf{Var}\left(\frac{\sum \epsilon_i+2x_i\epsilon_i+\sum\beta+4\beta x_i +2\beta x_i^2}{\sum1+4x_i+4x_i^2}\right)\\\\ &=\mathsf{Var}\left(\frac{\sum \epsilon_i+2x_i\epsilon_i}{\sum1+4x_i+4x_i^2}+\frac{\sum\beta+4\beta x_i +2\beta x_i^2}{\sum1+4x_i+4x_i^2}\right)\\\\ &=\mathsf{Var}\left(\frac{\sum \epsilon_i+2x_i\epsilon_i}{\sum1+4x_i+4x_i^2}\right)\\\\ &=\mathsf{Var}\left(\frac{\sum\epsilon_i(1+2x_i)}{\sum(1+2x_i)^2}\right) \end{align*}$$

but I cannot isolate the $\epsilon_i$, the only random variable remaining.

Answer 1

To finish off, write $$\operatorname{Var}\left(\frac{\sum\epsilon_it_i}{\sum t_i^2}\right)\stackrel{(1)}=\frac{\operatorname{Var}(\sum\epsilon_it_i)}{(\sum t_i^2)^2}\stackrel{(2)}=\frac{\sum\sigma^2t_i^2}{(\sum t_i^2)^2}\stackrel{(3)}=\frac{\sigma^2\sum t_i^2}{(\sum t_i^2)^2}\stackrel{(4)}=\frac{\sigma^2}{\sum t_i^2}.$$ I've written $t_i:=1+2x_i$ for brevity. Step (1) is the rule $\operatorname{Var}(cX)=c^2\operatorname{Var}(X)$. Step (2) is using the uncorrelatedness of the $\epsilon$'s. Step (3) is where you can pull out the constant $\sigma^2$ from the summation. Step (4) is cancelling a common factor from top and bottom.

Answer 2

If you use the transformed explanatory variable $x_{*i} = 1+2x_i$ then the model easily simplifies to a simple linear regression without an intercept term:

$$y_i = \beta x_{*i} + \epsilon_i \quad \quad \quad \epsilon_i \sim \text{IID N}(0, \sigma^2).$$

From here, all the usual mathematical results for this linear regression hold. In particular, the OLS estimator is unbiased, with variance given by the usual formula. Specific results are below.

---

Since this is a simple linear regression (without an intercept) you have OLS estimator given by:

$$\hat{\beta} = \frac{\sum x_{*i} Y_i}{\sum x_{*i}^2} = \frac{\sum (1+2x_i) Y_i}{\sum (1+2x_i)^2} = \frac{\sum (1+2x_i) Y_i}{\sum (1+4x_i + 4x_i^2)}.$$

Since $\mathbb{E}(Y_i) = \beta x_{*i} = \beta (1+2x_i)$ you have:

$$\mathbb{E}(\hat{\beta}) = \frac{\sum (1+2x_i) \mathbb{E}(Y_i)}{\sum (1+2x_i)^2} = \beta \frac{\sum (1+2x_i)^2}{\sum (1+2x_i)^2} = \beta.$$

Since $\mathbb{V}(Y_i) = \sigma^2$ you have:

$$\mathbb{V}(\hat{\beta}) = \frac{\sum (1+2x_i)^2 \mathbb{V}(Y_i)}{(\sum (1+2x_i)^2)^2} = \frac{\sigma^2}{\sum (1+2x_i)^2}.$$

Answer 3

Part $(a)$ seems legit.

---

Computing $E(\hat{\beta})$

Taking it from where you stopped \begin{equation} \hat{\beta}=\frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2} \end{equation} Then \begin{equation} E\hat{\beta}=\frac{\sum Ey_i+2x_iEy_i}{\sum1+4x_i+4x_i^2} \end{equation} We know that \begin{equation} E(y_i) = E\beta+ E2 \beta x_i+E\epsilon_i= \beta+ 2 \beta x_i \end{equation} So \begin{equation} E\hat{\beta}=\frac{\sum \beta+ 2 \beta x_i+2x_i(\beta+ 2 \beta x_i)}{\sum1+4x_i+4x_i^2} \end{equation} that is \begin{equation} E\hat{\beta}=\frac{\sum \beta+ 4 \beta x_i+ 4 \beta x_i^2}{\sum1+4x_i+4x_i^2} = \beta \Bigg( \frac{\sum1+4x_i+4x_i^2}{\sum1+4x_i+4x_i^2} \Bigg) = \beta \end{equation}

---

Computing var $\hat{\beta}$

Note that \begin{equation} \operatorname{var} \hat{\beta} = E(\hat{\beta}-\beta)^2 = E( \frac{\sum y_i+2y_ix_i}{\sum1+4x_i+4x_i^2} - \beta)^2 \end{equation} that is \begin{equation} \operatorname{var} \hat{\beta} = E\Big( \frac{\sum y_i+2y_ix_i -\beta-4x_i\beta-4x_i^2\beta }{\sum1+4x_i+4x_i^2} \Big)^2 \end{equation} Using $y_i=\beta+ 2 \beta x_i+\epsilon_i$, we get \begin{equation} \operatorname{var} \hat{\beta} = E\Big( \frac{\sum \beta+ 2 \beta x_i+\epsilon_i+2(\beta+ 2 \beta x_i+\epsilon_i)x_i -\beta-4x_i\beta-4x_i^2\beta }{\sum1+4x_i+4x_i^2} \Big)^2 \end{equation} that is \begin{equation} \operatorname{var} \hat{\beta} = E\Big( \frac{\sum \epsilon_i+2\epsilon_ix_i }{\sum1+4x_i+4x_i^2} \Big)^2 \end{equation} that is \begin{equation} \operatorname{var} \hat{\beta} = \frac{E(\sum \epsilon_i+2\epsilon_ix_i )^2}{(\sum1+4x_i+4x_i^2)^2} \tag{1} \end{equation} Consider the numerator of the above expression, \begin{equation} E(\sum \epsilon_i+2\epsilon_ix_i )^2 = E(\sum_i\epsilon_i+2\epsilon_ix_i )(\sum_j \epsilon_j+2\epsilon_jx_j ) \end{equation} which is \begin{equation} E(\sum \epsilon_i+2\epsilon_ix_i )^2 = \sum_{ij} E(\epsilon_i\epsilon_j)(1+2x_i)(1+2x_j) \end{equation} But \begin{equation} E(\epsilon_i\epsilon_j)=\sigma^2 \delta(i-j) \end{equation} where $\delta(x) = 1$ if $x = 0$ and $0$ otherwise. We will get \begin{equation} E(\sum \epsilon_i+2\epsilon_ix_i )^2 = \sum_{ij} \sigma^2 (1+2x_i)(1+2x_j)\delta(i-j) \end{equation} So the sum boils down to \begin{equation} E(\sum \epsilon_i+2\epsilon_ix_i )^2 = \sum_{i=j} \sigma^2 (1+2x_i)(1+2x_j) = \sigma^2\sum_i (1+2x_i)^2 \end{equation} Replace in equation $(1)$, we get \begin{equation} \operatorname{var} \hat{\beta} = \frac{\sigma^2\sum_i (1+2x_i)^2}{(\sum1+4x_i+4x_i^2)^2} \end{equation} The denominator is also written as \begin{equation} \operatorname{var} \hat{\beta} = \frac{\sigma^2\sum_i (1+2x_i)^2}{(\sum_i (1+2x_i)^2)^2} = \sigma^2 \frac{1}{\sum_i (1+2x_i)^2} \end{equation}