I'm considering a problem that has been partially addressed elsewhere: Bayesian updating with conjugate priors using the closed form expressions

but now I have an added twist. My samples are drawn from distributions of unknown mean and variance, and I wish to estimate this mean and variance. However, instead of direct access to the samples, I only have noisy observations of them, which I model by addition of Gaussian noise of ~N(0,s) to the each sample independently, and s is known. How do I incorporate this additional noise term to the posterior worked out for the original problem above?

  • $\begingroup$ The additional noise may jeopardize conjugacy. It actually does for the variance. $\endgroup$ – Xi'an Nov 29 '16 at 19:46

If the samples have a normal distribution then the added noise simply adds the known s to the variance of the samples. So this affects the likelihood function. Then the posterior can be computed in the usual way. If you are assuming a non-normal distribution things are complicated because you need to determine the distribution of the observation which is now the sample and the independent Gaussian noise term.


You cannot because you do not have an additional variable correlated either with the noise or the true value of $\sigma$ for the underlying.

Let me give you an example. Imagine that you had perfect measurements when you started your experiment, but at some point the machine doing the measurement broke. The problem was that it added noise, but did not impact the center of location. Again, for simplicity, we will assume normality throughout.

Your problem would include $\sigma_{true}$, $\sigma_{noise}$, and a parameter $k$ which is an estimate of the timing of the break.

To the left of the partition you would estimate $\mathcal{N}(\mu,\sigma_{true}^2)$. To the right of the partition you would model $\mathcal{N}(\mu,\sigma_{true}^2+\sigma_{noise}^2)$ and your partition would follow some distribution as well. There are several candidate distributions depending on how you want to model where the break happens.

There is no conjugacy based solution, but without a to distinguish, the most you know is that $\sigma_{true}\le\sigma_{measured}$.

Without a solution to filter the results, you have an ordinary conjugacy problem. It might be important to note that as Bayesian probability is $\Pr(\mu;\sigma|X)$, where $X$ is data, it treats data as fixed. Randomness does not exist, it is rather just uncertainty as to the transform of the real data.

You are arguing that $X_{observed}=f(X_{true})$, but that will require some way to measure $f$ with no additional source of data, which cannot be done.

Your problem is that $\sigma_{observed}^2=\sigma_{true}^2+\sigma_{noise}^2.$ This is no different than the generic problem $c=a+b$, with $a;b>0$ There are an infinite number of solutions to the set of a and b.

It doesn't upset the conjugacy calculation because you cannot differentiate the two. If you could differentiate signal and noise, even weakly, conjugacy would not apply and you would have to do numerical integration.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.