Suppose we wish to model two variables $x$ and $y$, as having an underlying linear relation with added errors. That is, with data $(x,y)_i: i = 1,...,n$, we model: $$ \begin{pmatrix} x_i \\ y_i \end{pmatrix} \sim \mathcal N\left(\begin{pmatrix}u_i\\v_i\end{pmatrix},\Sigma\right) $$ and $v_i = a + bu_i$ with some parameters ($a,b)$ and $\Sigma$ is a diagonal.

Assume that the value $u_i$ follow a normal distribution with mean $\mu$ and variance $\tau^2$. Write the likelihood of data given the parameters; you can do this by integrating over $u_1,...,u_n$ or by working with the multivariate normal distribution.

Given density function:

\begin{align} f(x,y|u,a,b,\Sigma) &= \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2}\left[\frac{(x-u_i)^2}{\sigma_x^2}+\frac{(y-a-bu_i)^2}{\sigma_y^2}\right]\right) \\ p(u_i)&= \frac{1}{\sqrt{2\pi}\tau} \exp\left(-\frac{1}{2}\frac{(u_i-\mu)^2}{\tau^2}\right) \\ f(x,y|a,b,\Sigma) &= \int ...\int f(x,y|u,a,b,\Sigma)p(u_i)du_1...du_n \end{align}

My question is how to get the integration above? What distribution does it follow?

  • $\begingroup$ Don't you need a product over $i$ in there somewhere? $\endgroup$ – Glen_b May 3 '13 at 0:33

You need to multiply $f(x,y|u_i,a,b,\Sigma)$ and $p(u_i)$ to get a joint distribution of $f(x,y,u_i|a,b, \Sigma)$:

$f(x,y,u_i|a,b, \Sigma) = f(x,y|u_i,a,b,\Sigma) * p(u_i) = \frac{1}{\sqrt{2\pi}}exp(-\frac{1}{2}[\frac{(x-u_i)^2}{\sigma_x^2}+\frac{(y-a-bu_i)^2}{\sigma_y^2}])* \frac{1}{\sqrt{2\pi}\tau}exp(-\frac{1}{2}\frac{(u_i-\mu)^2}{\tau^2})$

Note, the exponential part can be re-organized to have a form similar to $ \frac{ (u_i - xxx)^2 }{ yyy}$, you will need to figure out what is $xxx$ and $yyy$.

Then you can integrate $u_i$ out using:

$\int_{-\infty}^{\infty} f(x,y,u_i|a,b, \Sigma) \mathrm{d} u_i$

Also note the above illustrated 1 sample, but you have $N$ observations (you will need to multiply the likelihood together).

  • $\begingroup$ Thanks for clarify my question. The problem I'm having is how to do the integration. I tried to expand and group $u_i$ together but no luck. I can't get familiar form. Anyone has idea? $\endgroup$ – bayesnwb May 4 '13 at 1:54
  • $\begingroup$ another hint: integrate(exp(-x^2), (x, -infinity, infinity)) = sqrt(pi) refer: wolframalpha.com/input/… $\endgroup$ – zhanxw May 4 '13 at 14:33
  • $\begingroup$ u_i is supposed to be integrated out not x $\endgroup$ – bayesnwb May 6 '13 at 0:44
  • $\begingroup$ you're right, but idea is the same. So you can substitute x with u_i. The exponent part in the integral can be written as a quadratic form. $\endgroup$ – zhanxw May 7 '13 at 6:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.