# MLE for $\theta$ in $y_i=\theta x_i+\epsilon_i$

Let $x_i's$ and $\epsilon_i's$ be iid $\sim normal(0,1)$ i$i=1,2... n$ $n>=3$

The objective is to find the MLE for $\theta$ in $y_i=\theta x_i+\epsilon_i$ and check if it is unbiased.

The MLE I got is $\theta^*=\frac{\sum_{i=1}^ny_i}{\sum_{i=1}^nx_i}$

When I try to check for unbiasedness, i run into a problem:

$E(\theta^*)=E(\frac{\sum_{i=1}^ny_i}{\sum_{i=1}^nx_i})=E(\frac{\sum_{i=1}^n\theta x_i+\sum_{i=1}^n\epsilon_i}{\sum_{i=1}^nx_i})=\theta+E(\frac{\sum_{i=1}^n\epsilon_i}{\sum_{i=1}^nx_i})$

How do I evaluate that expectation? I think it should be 1 but I am not sure. Is the MLE for this really biased or did I make a mistake somewhere?

$E[\frac{\sum \epsilon_i}{\sum x_i}]=0$.

To see this, note that $\sum x_i$ is non-random and so the constant $\frac{1}{\sum x_i}$ can be taken outside of the expectation. Mean while, the linearity of expectation tells us that:

$E[\sum \epsilon_i] = \sum E[\epsilon_i] = \sum 0 = 0$ since each $\epsilon_i \sim N(0,1)$ so that $E[\epsilon_i] = 0 \forall i$

Putting this together gives:

$E[\theta^*] = \theta + E[\frac{\sum \epsilon_i}{\sum x_i}]= \theta + \frac{1}{\sum x_i} E[\sum \epsilon_i] = \theta + 0 = \theta$

Edit: Due to the odd wording of the question, suppose that $x_i$ are indeed random and not yet sampled. So that $y_i = \theta x_i + \epsilon_i$ with both $x_i$ and $\epsilon_i$ being standard normal random variables. Then the above solution still works provided we can assume $x_i$ and $\epsilon_i$ are independent.

This, in my opinion, is a completely valid assumption since $\epsilon_i$ are just stated to be random noise and so should not be dependent on the covariates $x_i$.

If this is the case, then we can use the fact that for independent random variables $X$,$Y$, and functions $f(x)$,$g(y)$, we know that:

$E[f(X)g(Y)] = E[f(X)]\times E[g(Y)]$

So, in this context $f(x) = \frac{1}{\sum x_i}$ and $g(\epsilon) = \sum \epsilon_i$

This gives us that:

$E[\frac{\sum \epsilon_i}{\sum x_i}] = E[\frac{1}{\sum x_i}]\times E[\sum \epsilon_i] = E[\frac{1}{\sum x_i}] \times 0 = 0$

So that we don't actually need to evaluate the expectation of $\frac{1}{\sum x_i}$.

If on the other hand we can't assume they are independent, I don't think there's anything you can do.

• But the problem did not say that the $x_i's$ are non-random – user164144 Jul 5 '17 at 0:23
• The point of this model is that you break down $y_i$ into a non-random component $\theta x_i$ and a random 'noise' component $\epsilon_i$. This is why they state $\epsilon_i \sim N(0,1)$, but do not state a distribution for $x_i$, which are simply your measured explanatory variables. – Patty Jul 5 '17 at 0:24
• But the problem did state a distribution. The exact wording of the problem is "...where covariates $x_i's$ and noise $\epsilon_i's$ are iid random variables with standard normal distribution. – user164144 Jul 5 '17 at 0:29
• It's stating that the $x_i's$ are covariates used to explain $y_i's$ and the $\epsilon_i's$ are random noise which are standard normal random variables. It would make no sense if you also assume $x_i$ are random in this context. – Patty Jul 5 '17 at 0:31
• Just saying that that is the exact wording of the problem. It states that both the $x_i's$ and the $\epsilon_i's$ are iid standard normal. Is this really not possible to solve as the problem is stated? – user164144 Jul 5 '17 at 0:36