# Integrating unknown parameter out of the likelihood function

Suppose we wish to model two variables $x$ and $y$, as having an underlying linear relation with added errors. That is, with data $(x,y)_i: i = 1,...,n$, we model: $$\begin{pmatrix} x_i \\ y_i \end{pmatrix} \sim \mathcal N\left(\begin{pmatrix}u_i\\v_i\end{pmatrix},\Sigma\right)$$ and $v_i = a + bu_i$ with some parameters ($a,b)$ and $\Sigma$ is a diagonal.

Assume that the value $u_i$ follow a normal distribution with mean $\mu$ and variance $\tau^2$. Write the likelihood of data given the parameters; you can do this by integrating over $u_1,...,u_n$ or by working with the multivariate normal distribution.

Given density function:

\begin{align} f(x,y|u,a,b,\Sigma) &= \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{1}{2}\left[\frac{(x-u_i)^2}{\sigma_x^2}+\frac{(y-a-bu_i)^2}{\sigma_y^2}\right]\right) \\ p(u_i)&= \frac{1}{\sqrt{2\pi}\tau} \exp\left(-\frac{1}{2}\frac{(u_i-\mu)^2}{\tau^2}\right) \\ f(x,y|a,b,\Sigma) &= \int ...\int f(x,y|u,a,b,\Sigma)p(u_i)du_1...du_n \end{align}

My question is how to get the integration above? What distribution does it follow?

• Don't you need a product over $i$ in there somewhere? – Glen_b May 3 '13 at 0:33

You need to multiply $f(x,y|u_i,a,b,\Sigma)$ and $p(u_i)$ to get a joint distribution of $f(x,y,u_i|a,b, \Sigma)$:

$f(x,y,u_i|a,b, \Sigma) = f(x,y|u_i,a,b,\Sigma) * p(u_i) = \frac{1}{\sqrt{2\pi}}exp(-\frac{1}{2}[\frac{(x-u_i)^2}{\sigma_x^2}+\frac{(y-a-bu_i)^2}{\sigma_y^2}])* \frac{1}{\sqrt{2\pi}\tau}exp(-\frac{1}{2}\frac{(u_i-\mu)^2}{\tau^2})$

Note, the exponential part can be re-organized to have a form similar to $\frac{ (u_i - xxx)^2 }{ yyy}$, you will need to figure out what is $xxx$ and $yyy$.

Then you can integrate $u_i$ out using:

$\int_{-\infty}^{\infty} f(x,y,u_i|a,b, \Sigma) \mathrm{d} u_i$

Also note the above illustrated 1 sample, but you have $N$ observations (you will need to multiply the likelihood together).

• Thanks for clarify my question. The problem I'm having is how to do the integration. I tried to expand and group $u_i$ together but no luck. I can't get familiar form. Anyone has idea? – bayesnwb May 4 '13 at 1:54
• another hint: integrate(exp(-x^2), (x, -infinity, infinity)) = sqrt(pi) refer: wolframalpha.com/input/… – zhanxw May 4 '13 at 14:33
• u_i is supposed to be integrated out not x – bayesnwb May 6 '13 at 0:44
• you're right, but idea is the same. So you can substitute x with u_i. The exponent part in the integral can be written as a quadratic form. – zhanxw May 7 '13 at 6:51