I have an exercise where I have to use a probabilistic model to fit the data. Here is the original problem (Problem 1): enter image description here

(Problem 2 with solution) At the lecture we learnt to derive Bayesian linear regression solution. We have dataset $\textbf{x} = (x^{(1)},...,x^{(N)}) $ and $\textbf{y} = (y^{(1)},...,y^{(N)}) $, parameters $\textbf{w} = (w_0,w_1)^T $.

We also have $\phi(x) = (1,x)^T$ and $ y = h^w(x) + \epsilon$ with $\epsilon \sim N(0,\sigma^2)$

Then we have the Bayesian solution : enter image description here

The question is how to use insight from Problem 2 to solve Problem 1. As I realize problem 1 is just problem 2 version without any predictors $x$ (in which $\mu = w_0$ and $e^{(i)} = \epsilon$ ). My best guess would be $\mu = y^{(N+1)} = \text{Mean}(y^{(i)})$ but I'm not sure how to prove it

  • $\begingroup$ Is it necessary to use Problem 2 as a 'substep' in the solution? For example, since the shift by a constant of a normal distribution is again a normal distribution we have $p(y_i) = N(\mu, 1)$ and so we can just write down the likelihood and minimize $- \log L$ directly which will lead you to $\mu = \text{mean of the $y_i$}$ i believe... $\endgroup$ Nov 2 '17 at 10:59
  • $\begingroup$ Not necessarily. Actually I think what u just said is exactly what I'm looking for. Could you elaborate/derive what is the log-likelihood form and how to minimize it to get the answer ? That would be great ! $\endgroup$ Nov 2 '17 at 11:02
  • $\begingroup$ I have to go to lunch now :-) If you can't solve it on your own within the next 45 minutes then I will write a complete answer... deal? $\endgroup$ Nov 2 '17 at 11:10
  • $\begingroup$ Yes sir. I'll try to :) $\endgroup$ Nov 2 '17 at 11:20

Well I think I got the answer for this. Thanks to Fabian Werner and based on this wikipedia article https://en.wikipedia.org/wiki/Maximum_likelihood_estimation:

So we have $p(y_i) = N(\mu,1)$ and log-likelihood $$\log(L(\mu,\sigma)) = (-n/2)log(2\pi)-(1/2)\sum(y^{i}-\mu)^2$$ Taking derivatives: $$ 0 = \frac{\partial}{\partial\mu}\log(L(\mu,\sigma)) = 0 - \frac{-2n(\bar{y} - \mu)}{2}$$ And we get the answer $\mu = \bar{y} $ (Mean of y)

  • $\begingroup$ So, what about the other two questions? ;-) $\endgroup$ Nov 2 '17 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.