Practically, how to update a Bayesian distribution for a single parameter

Question

First of all let me say that I have looked at this very helpful answer, but I don't believe it fully answers my question.

Background

Let's say I have a single parameter $\theta$ that I want to know the value of. At the moment I am unsure on the true, physical value $\theta$, and so I quantify this uncertainty via a probability distribution $p(\theta)$. For this question, we can just assume it to be Gaussian ,$N(\mu, \sigma^2)$.

This parameter causes some signal $s$ which I can measure. Given this signal I can update my prior via Bayes' as,

$$ p(\theta| s) = \frac{p(s|\theta) p(\theta)}{p(s)}$$

i.e. what is the probability of the parameter $\theta$ given $s$?

Question

How exactly do I update the distribution and obtain the posterior? $p(\theta)$ is a distribution, but $p(s|\theta)$ and $p(s)$ are just scalars right? So I pick some value of $\theta$, calculate the probability $p(s|\theta)$, multiply the corresponding value in the distribution $p(\theta)$, repeat for all values of $\theta$? And then divide by $p(s)$?

I am a bit confused and a clear worked example (or pointing to the relevant material) would be much appreciated

Answer 1

It really depends on what $s$ is.

We call $p(s \vert \theta)$ the likelihood. To do Bayesian inference with pen and paper, the likelihood has to have a specific form (in particular, it has to be Gaussian -- well, it doesn't HAVE to be, but it makes things easier than if it weren't).

That question you linked to does a good job of explaining the updating when the likelihood is Gaussian. In the case it is not Gaussian, we have to resort to some computational methods, which are not relevant at this time.

I'm also going to be a bit pedantic for a moment. $p(s\vert \theta)$ is only a scalar when you know $\theta$. Since you don't know theta, then $p(s\vert \theta)$ is not a scalar; it is a probability distribution parameterized by $\theta$. So, your conceptualization of the problem (namely, that $p(s\vert \theta)$ and $p(s)$ are just scalars) is flawed. This is likely where the misunderstanding is stemming from.

Answer 2

In Bayesian inference, $p(s|\theta)$ is called the likelihood function, which is a distribution but seen as a function of the sample (in this case the signal $s$). $p(s)$ is indeed a scalar (as a function of $\theta$), called the normalizing constant which makes $p(\theta|s)$ a proper density. You can see this easily by writing $$p(s) = \int p(s|\theta)p(\theta).$$

Now, to make inferences about $\theta$, we are are not usually interested in the distribution $p(\theta|s)$, but in a quantity related to this distribution (e.g. the mean).

In a textbook example you would find a distribution proportional to $p(\theta|s)$ by multiplying the likelihood and the prior. This posterior depends on the observed signal $s$. Ideally, you would then identify a parametric family for the posterior from which you can easily evaluate the quantity of interest. If the posterior is not easily obtainable, methods like MCMC are used to generate samples from this distribution and estimate the quantity of interest.

You can read more about this in Probabilistic Programming and Bayesian Methods for Hackers. This book is not so theoretical and I think it's a good reference to understand the basics of Bayesian inference.

Answer 3

What about reading a Bayesian textbook:

What is the best introductory Bayesian statistics textbook?

or even an on-line tutorial

Bayesian statistics tutorial