# Practically, how to update a Bayesian distribution for a single parameter

First of all let me say that I have looked at this very helpful answer, but I don't believe it fully answers my question.

Background

Let's say I have a single parameter $$\theta$$ that I want to know the value of. At the moment I am unsure on the true, physical value $$\theta$$, and so I quantify this uncertainty via a probability distribution $$p(\theta)$$. For this question, we can just assume it to be Gaussian ,$$N(\mu, \sigma^2)$$.

This parameter causes some signal $$s$$ which I can measure. Given this signal I can update my prior via Bayes' as,

$$p(\theta| s) = \frac{p(s|\theta) p(\theta)}{p(s)}$$

i.e. what is the probability of the parameter $$\theta$$ given $$s$$?

Question

How exactly do I update the distribution and obtain the posterior? $$p(\theta)$$ is a distribution, but $$p(s|\theta)$$ and $$p(s)$$ are just scalars right? So I pick some value of $$\theta$$, calculate the probability $$p(s|\theta)$$, multiply the corresponding value in the distribution $$p(\theta)$$, repeat for all values of $$\theta$$? And then divide by $$p(s)$$?

I am a bit confused and a clear worked example (or pointing to the relevant material) would be much appreciated

• If you would like a worked example, perhaps you could tell us the sampling distribution $p(s|\theta)$ that you are using, and then we can show you how to solve for that case.
– Ben
Mar 5, 2020 at 0:37
• @Ben-ReinstateMonica My main concern is application to gravitational systems. See Eq. 1 of arxiv.org/pdf/gr-qc/0703086v1.pdf Mar 5, 2020 at 1:26
• $p(s|\theta)$ is a function of $\theta$, specifically, the likelihood function Mar 5, 2020 at 1:47

In Bayesian inference, $$p(s|\theta)$$ is called the likelihood function, which is a distribution but seen as a function of the sample (in this case the signal $$s$$). $$p(s)$$ is indeed a scalar (as a function of $$\theta$$), called the normalizing constant which makes $$p(\theta|s)$$ a proper density. You can see this easily by writing $$p(s) = \int p(s|\theta)p(\theta).$$

Now, to make inferences about $$\theta$$, we are are not usually interested in the distribution $$p(\theta|s)$$, but in a quantity related to this distribution (e.g. the mean).

In a textbook example you would find a distribution proportional to $$p(\theta|s)$$ by multiplying the likelihood and the prior. This posterior depends on the observed signal $$s$$. Ideally, you would then identify a parametric family for the posterior from which you can easily evaluate the quantity of interest. If the posterior is not easily obtainable, methods like MCMC are used to generate samples from this distribution and estimate the quantity of interest.

You can read more about this in Probabilistic Programming and Bayesian Methods for Hackers. This book is not so theoretical and I think it's a good reference to understand the basics of Bayesian inference.

It really depends on what $$s$$ is.

We call $$p(s \vert \theta)$$ the likelihood. To do Bayesian inference with pen and paper, the likelihood has to have a specific form (in particular, it has to be Gaussian -- well, it doesn't HAVE to be, but it makes things easier than if it weren't).

That question you linked to does a good job of explaining the updating when the likelihood is Gaussian. In the case it is not Gaussian, we have to resort to some computational methods, which are not relevant at this time.

I'm also going to be a bit pedantic for a moment. $$p(s\vert \theta)$$ is only a scalar when you know $$\theta$$. Since you don't know theta, then $$p(s\vert \theta)$$ is not a scalar; it is a probability distribution parameterized by $$\theta$$. So, your conceptualization of the problem (namely, that $$p(s\vert \theta)$$ and $$p(s)$$ are just scalars) is flawed. This is likely where the misunderstanding is stemming from.