3
$\begingroup$

I have: $X \sim \mathcal{N}(\mu_x, \sigma_x^2)$ and $Y \sim \mathcal{N}(\mu_y, \sigma_y^2)$. $X$ and $Y$ are independent.
$\mu_x$ and $\mu_y$ are not known and I want to learn about them (Bayesian inference).
However, I don't observe draws from $X$ and $Y$ independently, I only observes draws of $Z = X + Y$, with $Z \sim \mathcal{N}(\mu_x + \mu_y, \sigma_x^2 + \sigma_y^2)$.

Given some prior about $\mu_x$ and $\mu_y$ (e.g. $\mu_x \sim N(m^x_0, s^x_0)$ and $\mu_y \sim N(m^y_0, s^y_0)$), can I get a posterior that would have a closed form (and which would also be normal - i.e. keep the conjugacy property) ?

I tried to do the computations, basically I'm stuck. If you know $\mu_x$ or $\mu_y$ it's quite straightforward because then you just change $Z$ into $Z - \mu_x$ for example.
But with both unknown I cannot get a normal closed form for the posterior (of $\mu_x$ or $\mu_y$) because the posterior depends on the other mean (which is also unknown).
e.g. for $\mu_x$ for example, the posterior is $\mu_x \sim N(m^x_1, s^x_1)$ where:
$s^x_1 = (1/\sigma^2_x + 1/s_x^2)^{-1}$
$m^x_1 = s^x_1 \Big( m^x_0/s_x^2 + (z - \mu_y)/\sigma^2_x \Big)$
But obviously, $\mu_y$ is not known either, hence the problem. The posterior must be weighted by your belief about $\mu_y$ somehow. And I'm not sure how it works. (Because similarly, your belief about $\mu_y$ is updated exactly in the same way). As suggested in the comments, there might be some identification problem here. But I'm not sure because the prior's precision limits the scope of your posterior about both distributions.

My questions are:
- Am I missing a simplification in the computations? i.e. maybe there is an additional simplification at the end.
- If no, is there another distribution that I could use (e.g. lognormal instead of normal? I thought maybe with the log properties I could factorize something) for which I would keep the conjugacy property?
- Does anyone have any source on this specific kind of learning setup?

EDIT: more details.

$\endgroup$
  • $\begingroup$ Isn't there an identifiability problem in this setup? $\endgroup$ – Demetri Pananos Aug 16 at 16:51
  • $\begingroup$ Yep I need an analytic form. In fact I need to frame the problem (choice of distributions, etc.) in order to get an analytic form. $\endgroup$ – G. Ander Aug 16 at 16:55
  • $\begingroup$ I'm not sure about identification. The thing is, you still start with some prior about both distributions. If one of your prior is like perfect information (e.g. knowledge of $\mu_x$), there's no identification problem. So for prior which are more noisy I believe you can still do something. But it yields some weird system. I'm going to edit the first post. $\endgroup$ – G. Ander Aug 16 at 16:59
3
$\begingroup$

You are correct that $Z\mid \mu_x,\mu_y,\sigma^2_x,\sigma^2_y \sim \mathcal{N}(\mu_x + \mu_y, \sigma_x^2 + \sigma_y^2)$. First note that this likelihood is not identifiable. This is because $\mu_x,\mu_y$ will evaluate to the same function as $\mu_x-1,\mu_y+1$. Actually, you can add and subtract any number and get the same likelihood. This is why the likelihood has a “ridge.”

However, the posterior is identifiable. This means that any kind of inference about a single parameter will definitely be very sensitive to the prior you choose.

Assuming all of the variances are known, \begin{align*} p(z \mid \mu_x, \mu_y) &\propto p(z \mid \mu_x, \mu_y) p(\mu_x, \mu_y) \\ &\propto \exp\left[-\frac{(z- \mu_x - \mu_y)^2}{2\sigma^2_x + 2 \sigma^2_y} \right] \exp\left[-\frac{(\mu_x - m_0^x)^2}{2s_0^x} \right]\exp\left[-\frac{(\mu_y - m_0^y)^2}{2s_0^y} \right] \\ \end{align*} which is proportional to the exponential of $$ -\frac{1}{2(\sigma^2_x + \sigma^2_y)(s_0^x s_0^y)}\left\{ (z- \mu_x - \mu_y)^2s_0^xs_0^y + (\mu_x - m_0^x)^2s_0^y(\sigma^2_x + \sigma^2_y) + (\mu_y - m_0^y)^2s_0^x(\sigma^2_x + \sigma^2_y) \right\}. $$ This means that your posterior is bivariate normal, and after you collect the terms with $\mu_x\mu_y$, you'll see you'll get a negative correlation. This makes intuitive sense, because if $\mu_x$ is really "big", then that will probably mean $\mu_y$ is really "small."

Here's some R code that helps show this. It plots the log of the unnormalized posterior

# library(devtools)
# devtools::install_github("tbrown122387/mmcmc")
library(mmcmc)
logUnNormPosterior <- function(mux,muy){
  z <- 2
  sigmaSqX <- sigmaSqY <- sx <- sy <- 1
  mx <- my <- 0
  dnorm(x = z, mux+muy, sqrt(sigmaSqY + sigmaSqY), log = T)+
    dnorm(mux, mx, sx, log = T) + dnorm(muy, my, sy, log = T)
}
mmcmc::plotSurface(lowerFirst = -20, upperFirst = 20, 
                   lowerSecond = -20, upperSecond = 20,
                   numGridPointsOnEachAxis = 10, 
                   f = logUnNormPosterior, contour = T)

Here's the plot:

plot

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.