Probability density vs probability in Bayesian statistics

Question

I have started learning Bayesian statistics recently, but I am confused about probability densities and probability. I do know they are different, and I can usually identify both, but in one problem, I am confused if I am calculating a probability density or probability.

Here is the description:

I have a variable,

$m \sim N(\frac{a+b}2, 1)$

The prior distribution for $a$ and $b$ are as follows- both $a$ and $b$ are uniformly distributed over $[-1,1]$.

I wanted to calculate,

$P(0 \leq b \leq 1 |0 \leq a \leq 1, m=0)$.

If I calculate this, will I have a probability density or probability? If it is a probability density, can its value exceed 1?

Especially, if I find that the value of the expression is greater than $0.5$, does that mean that given the conditions, the probability of getting a positive $b$ is greater than $0.5$? And that the probability of $b$ being less than $0$ is less than $0.5$?

Answer 1

The distinction between probability density and probability of an event has little to relate with Bayesian statistics.

Given a random variable $X$, with distribution $P^X$, this distribution can be characterised by the definition of $P^X(A)$ for all measurable sets $A$, or equivalently by the definition of a density $p^X$ wrt a measure $\text{d}\mu$ if the distribution is absolutely continuous wrt this measure, the connection being that $$P^X(A) = \int_A p^X(x)\text{d}\mu(x)$$ For instance, if $\text{d}\mu$ is the Lebesgue measure and $P^X$ the Uniform distribution on $(-1,1)$, $$P^X([0,1])=\int_0^1 \frac{1}{2}\text{d}x=\frac{1}{2}$$ When dealing with conditional distributions, things are pretty much the same, namely that given $m=0$ and $0\le a\le 1$, $b$ is a random variable with a probability distribution denoted by e.g. $$P^b(\cdot|0\le a\le 1,m=0)$$ which is associated with a probability coverage of all measurable sets $A$ in $[-1,1]$, $$P^b(A|0\le a\le 1,m=0)$$and which enjoys a probability density $$p^b(\cdot|0\le a\le 1,m=0)$$ wrt the Lebesgue (uniform) measure on $[-1,1]$, such that $$P^b(A|0\le a\le 1,m=0)=\int_A p^b(x|0\le a\le 1,m=0)\text{d}x$$The only part where Bayes has a say is in defining this conditional distribution, since Bayes' theorem states that $$P^b(A|0\le a\le 1,m=0)=\frac{P^b(A,0\le a\le 1|m=0)}{P^b(0\le a\le 1|m=0)}$$

Answer 2

When you are dealing with continuous random variables, one way that you can tell the difference between probability and probability density is that the former applies to events involving random variables, whereas the latter applies to individual values of continuous random variables. Another way to tell is that we usually denote probabilties using an upper-case symbol like $\mathbb{P}$, $\text{P}$, $\text{Pr}$ or $P$, whereas we usually denote densities using a lower-case symbol like $f$ or $p$.

Now, if you look at your $P$ statement, you will see firstly that it is an upper-case symbol. But more importantly, you will notice that the variable of interest is stipulated to fall within a particular range, rather than being at an individual value. This gives you the event $\{ 0 \leqslant b \leqslant 1 \}$ so you are looking at the conditional probability of that event. This conditional probability can be computed from the underlying densities as:

$$\begin{aligned} \mathbb{P}(0 \leqslant b \leqslant 1 | 0 \leqslant a \leqslant 1, m=0) &= \frac{\mathbb{P}(0 \leqslant b \leqslant 1, 0 \leqslant a \leqslant 1| m=0)}{\mathbb{P}(0 \leqslant a \leqslant 1| m=0)} \\[6pt] &= \frac{\int_0^1 \int_0^1 p(a, b|m=0) \ da \ db}{\int_0^1 p(a|m=0) \ da}. \\[6pt] \end{aligned}$$

Since this is a probability, its value must be between zero and one. (And yes, a probability density value can exceed one.) Also, note that the distinction between probability and probability density is not specific to Bayesian statistics --- it occurs within probability theory, and arises in all statistical methodologies.