Let $x_i's$ and $\epsilon_i's$ be iid $\sim normal(0,1)$ i$i=1,2... n$ $n>=3$

The objective is to find the MLE for $\theta$ in $y_i=\theta x_i+\epsilon_i $ and check if it is unbiased.

The MLE I got is $\theta^*=\frac{\sum_{i=1}^ny_i}{\sum_{i=1}^nx_i}$

When I try to check for unbiasedness, i run into a problem:

$E(\theta^*)=E(\frac{\sum_{i=1}^ny_i}{\sum_{i=1}^nx_i})=E(\frac{\sum_{i=1}^n\theta x_i+\sum_{i=1}^n\epsilon_i}{\sum_{i=1}^nx_i})=\theta+E(\frac{\sum_{i=1}^n\epsilon_i}{\sum_{i=1}^nx_i})$

How do I evaluate that expectation? I think it should be 1 but I am not sure. Is the MLE for this really biased or did I make a mistake somewhere?


1 Answer 1


$E[\frac{\sum \epsilon_i}{\sum x_i}]=0$.

To see this, note that $\sum x_i$ is non-random and so the constant $\frac{1}{\sum x_i}$ can be taken outside of the expectation. Mean while, the linearity of expectation tells us that:

$E[\sum \epsilon_i] = \sum E[\epsilon_i] = \sum 0 = 0$ since each $\epsilon_i \sim N(0,1)$ so that $E[\epsilon_i] = 0 \forall i$

Putting this together gives:

$E[\theta^*] = \theta + E[\frac{\sum \epsilon_i}{\sum x_i}]= \theta + \frac{1}{\sum x_i} E[\sum \epsilon_i] = \theta + 0 = \theta$

Edit: Due to the odd wording of the question, suppose that $x_i$ are indeed random and not yet sampled. So that $y_i = \theta x_i + \epsilon_i$ with both $x_i$ and $\epsilon_i$ being standard normal random variables. Then the above solution still works provided we can assume $x_i$ and $\epsilon_i$ are independent.

This, in my opinion, is a completely valid assumption since $\epsilon_i$ are just stated to be random noise and so should not be dependent on the covariates $x_i$.

If this is the case, then we can use the fact that for independent random variables $X$,$Y$, and functions $f(x)$,$g(y)$, we know that:

$E[f(X)g(Y)] = E[f(X)]\times E[g(Y)]$

So, in this context $f(x) = \frac{1}{\sum x_i}$ and $g(\epsilon) = \sum \epsilon_i$

This gives us that:

$E[\frac{\sum \epsilon_i}{\sum x_i}] = E[\frac{1}{\sum x_i}]\times E[\sum \epsilon_i] = E[\frac{1}{\sum x_i}] \times 0 = 0$

So that we don't actually need to evaluate the expectation of $\frac{1}{\sum x_i}$.

If on the other hand we can't assume they are independent, I don't think there's anything you can do.

  • $\begingroup$ But the problem did not say that the $x_i's$ are non-random $\endgroup$
    – user164144
    Jul 5, 2017 at 0:23
  • $\begingroup$ The point of this model is that you break down $y_i$ into a non-random component $\theta x_i$ and a random 'noise' component $\epsilon_i$. This is why they state $\epsilon_i \sim N(0,1)$, but do not state a distribution for $x_i$, which are simply your measured explanatory variables. $\endgroup$
    – Patty
    Jul 5, 2017 at 0:24
  • $\begingroup$ But the problem did state a distribution. The exact wording of the problem is "...where covariates $x_i's$ and noise $\epsilon_i's$ are iid random variables with standard normal distribution. $\endgroup$
    – user164144
    Jul 5, 2017 at 0:29
  • 1
    $\begingroup$ It's stating that the $x_i's$ are covariates used to explain $y_i's$ and the $\epsilon_i's$ are random noise which are standard normal random variables. It would make no sense if you also assume $x_i$ are random in this context. $\endgroup$
    – Patty
    Jul 5, 2017 at 0:31
  • $\begingroup$ Just saying that that is the exact wording of the problem. It states that both the $x_i's$ and the $\epsilon_i's$ are iid standard normal. Is this really not possible to solve as the problem is stated? $\endgroup$
    – user164144
    Jul 5, 2017 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.