0
$\begingroup$

Let $x_i's$ and $\epsilon_i's$ be iid $\sim normal(0,1)$ i$i=1,2... n$ $n>=3$

The objective is to find the MLE for $\theta$ in $y_i=\theta x_i+\epsilon_i $ and check if it is unbiased.

The MLE I got is $\theta^*=\frac{\sum_{i=1}^ny_i}{\sum_{i=1}^nx_i}$

When I try to check for unbiasedness, i run into a problem:

$E(\theta^*)=E(\frac{\sum_{i=1}^ny_i}{\sum_{i=1}^nx_i})=E(\frac{\sum_{i=1}^n\theta x_i+\sum_{i=1}^n\epsilon_i}{\sum_{i=1}^nx_i})=\theta+E(\frac{\sum_{i=1}^n\epsilon_i}{\sum_{i=1}^nx_i})$

How do I evaluate that expectation? I think it should be 1 but I am not sure. Is the MLE for this really biased or did I make a mistake somewhere?

$\endgroup$
2
$\begingroup$

$E[\frac{\sum \epsilon_i}{\sum x_i}]=0$.

To see this, note that $\sum x_i$ is non-random and so the constant $\frac{1}{\sum x_i}$ can be taken outside of the expectation. Mean while, the linearity of expectation tells us that:

$E[\sum \epsilon_i] = \sum E[\epsilon_i] = \sum 0 = 0$ since each $\epsilon_i \sim N(0,1)$ so that $E[\epsilon_i] = 0 \forall i$

Putting this together gives:

$E[\theta^*] = \theta + E[\frac{\sum \epsilon_i}{\sum x_i}]= \theta + \frac{1}{\sum x_i} E[\sum \epsilon_i] = \theta + 0 = \theta$


Edit: Due to the odd wording of the question, suppose that $x_i$ are indeed random and not yet sampled. So that $y_i = \theta x_i + \epsilon_i$ with both $x_i$ and $\epsilon_i$ being standard normal random variables. Then the above solution still works provided we can assume $x_i$ and $\epsilon_i$ are independent.

This, in my opinion, is a completely valid assumption since $\epsilon_i$ are just stated to be random noise and so should not be dependent on the covariates $x_i$.

If this is the case, then we can use the fact that for independent random variables $X$,$Y$, and functions $f(x)$,$g(y)$, we know that:

$E[f(X)g(Y)] = E[f(X)]\times E[g(Y)]$

So, in this context $f(x) = \frac{1}{\sum x_i}$ and $g(\epsilon) = \sum \epsilon_i$

This gives us that:

$E[\frac{\sum \epsilon_i}{\sum x_i}] = E[\frac{1}{\sum x_i}]\times E[\sum \epsilon_i] = E[\frac{1}{\sum x_i}] \times 0 = 0$

So that we don't actually need to evaluate the expectation of $\frac{1}{\sum x_i}$.

If on the other hand we can't assume they are independent, I don't think there's anything you can do.

$\endgroup$
  • $\begingroup$ But the problem did not say that the $x_i's$ are non-random $\endgroup$ – user164144 Jul 5 '17 at 0:23
  • $\begingroup$ The point of this model is that you break down $y_i$ into a non-random component $\theta x_i$ and a random 'noise' component $\epsilon_i$. This is why they state $\epsilon_i \sim N(0,1)$, but do not state a distribution for $x_i$, which are simply your measured explanatory variables. $\endgroup$ – Patty Jul 5 '17 at 0:24
  • $\begingroup$ But the problem did state a distribution. The exact wording of the problem is "...where covariates $x_i's$ and noise $\epsilon_i's$ are iid random variables with standard normal distribution. $\endgroup$ – user164144 Jul 5 '17 at 0:29
  • 1
    $\begingroup$ It's stating that the $x_i's$ are covariates used to explain $y_i's$ and the $\epsilon_i's$ are random noise which are standard normal random variables. It would make no sense if you also assume $x_i$ are random in this context. $\endgroup$ – Patty Jul 5 '17 at 0:31
  • $\begingroup$ Just saying that that is the exact wording of the problem. It states that both the $x_i's$ and the $\epsilon_i's$ are iid standard normal. Is this really not possible to solve as the problem is stated? $\endgroup$ – user164144 Jul 5 '17 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.