1
$\begingroup$

I think I know in general how to derive the maximum likelihood estimation for a parameter given a distribution. But I can't wrap my head around this one!

We have the observations: $(y_1, x_1),...,(y_n, x_n)$ from a random sample where: $Y_i \sim N(\beta x_i, 1)$ for $i = 1,2,...,n$. We are interested in the expected value of $y_i$ given $x_i$.

How do i derive the ML-estimate for the parameter $\beta?$

Don't know how to begin when the parameter $\beta$ shows up in that way.

EDIT 1

I set:

$$\tilde{\mu} = x_i\beta$$

The Likelihood is:

$$\prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2}(\frac{y_i - \tilde{\mu}}{\sigma})^2} \iff (\frac{1}{\sqrt{2\pi\sigma^2}})^n e^{-\sum_{i=1}^{n}\frac{1}{2\sigma^2}(y_i - \tilde{\mu})^2}$$

Then the log-likelihood is:

$$-\frac{n}{2}\ln{2\pi} -\frac{n}{2}\ln{\sigma^2} -\frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i - \tilde{\mu})^2$$

If we take the derivative of the log-likelihood we get:

$$-\frac{1}{2\sigma^2}\frac{d}{d\tilde{\mu}}(\sum_{i=1}^{n}(y_i - \tilde{\mu})^2)$$

Which is:

$$-\frac{1}{2\sigma^2}\frac{d}{d\tilde{\mu}}(\sum_{i=1}^{n}(y_i^2 - 2y_i\tilde{\mu} + \tilde{\mu}^2))$$

Should i now insert $\tilde{\mu} = x_i\beta$?

If i do i get:

$$-\frac{1}{2\sigma^2}\frac{d}{d\beta}(\sum_{i=1}^{n}(y_i^2 - 2y_i x_i\beta + x_i^2\beta^2))$$

What should i do now, this approach below seems bad?

$$-\frac{1}{2\sigma^2}\sum_{i=1}^{n}\frac{d}{d\beta}(y_i^2 - 2y_i x_i\beta + x_i^2\beta^2)$$

$$-\frac{1}{2\sigma^2}\sum_{i=1}^{n} - 2y_i x_i + 2 x_i^2\beta$$

$$\frac{1}{2\sigma^2} 2n\bar{x} (n\bar{y} - n\bar{x}\beta)$$

And set to zero for finding the estimate: $$\frac{1}{2\sigma^2} 2n\bar{x} (n\bar{y} - n\bar{x}\beta) = 0$$

$$\bar{y} = \bar{x}\beta$$

$$\frac{\bar{y}}{\bar{x}} = \beta$$

Which is wrong.

EDIT 2

I do try my best to understand so please explain if you downvote.

$\endgroup$
11
  • 2
    $\begingroup$ Start by writing out the likelihood. If you don't know how to do it for $\beta x_i$, write it as a product of $n$ factors each using $\mu_i$, then replace $\mu_i$ with $\beta x_i$ for all $i$. $\endgroup$
    – Sycorax
    Commented Dec 1, 2022 at 19:01
  • 3
    $\begingroup$ 1. treat X as known, not a random variable. 2. Recall the chain rule. 3. It's always easiest to calculate MLEs from a single observation. $\endgroup$
    – AdamO
    Commented Dec 1, 2022 at 19:07
  • 3
    $\begingroup$ In the expression for the normal distribution, you have $g(\beta)=(y_i - x_i \beta)^2$. What's the derivative of $g$ wrt $\beta$? $\endgroup$
    – Sycorax
    Commented Dec 1, 2022 at 19:15
  • 2
    $\begingroup$ Nope. Intuitively, that doesn't make sense because it implies $\beta$ is not a real number when $\bar x$ is 0, and that the estimation would become very volatile for small changes in $\bar x$ around 0. Write out the steps from the start and don't skip any any of the algebra. $\endgroup$
    – Sycorax
    Commented Dec 1, 2022 at 19:43
  • 2
    $\begingroup$ You go astray at the outset: $\tilde\mu$ depends on a single observation $x_i.$ That's not going to be useful and, as a notation, it's misleading you, because $\tilde\mu$ needs a subscript $i.$ You also introduce an undefined and superfluous quantity "$\sigma,$" further confusing the issue. Since the log likelihood of $y_i$ is $-(y_i-\beta x_i)^2/2$ (up to an additive constant), use that to form the log likelihood for an independent set of $(x_i,y_i)$ data and go on from there. The rest can be done with elementary algebra -- you don't even need Calculus. $\endgroup$
    – whuber
    Commented Dec 1, 2022 at 20:39

1 Answer 1

1
$\begingroup$

The derivation is basically fine until here. After this step, you introduce $\bar x, \bar y$ and the result is not correct. $$0=-\frac{1}{2\sigma^2}\sum_{i=1}^{n} - 2y_i x_i + 2 x_i^2\beta.$$ It's given that $\sigma=1$. We can multiply through by the leading fraction. The rest is rearranging. $$\begin{align} 0 &= \sum_{i=1}^{n} y_i x_i - x_i^2\beta\\ &= \sum_{i=1}^{n} y_i x_i - \sum_{j=1}^n x_j^2\beta \\ \sum_{i=1}^{n} y_i x_i &= \beta \sum_{j=1}^n x_j^2 \\ \frac{\sum_{i=1}^{n} y_i x_i}{ \sum_{j=1}^n x_j^2} &= \beta \end{align}$$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.