How to calculate the probability of the parameters?

Question

I am reading the Wikipedia article on posterior probability and I note the expression:

$$P(\theta|X) = \frac{P(X|\theta)P(\theta)}{P(X)}$$

I understand that $\theta$ represents the parameters of the probability model underlying $P$. However, what is actually meant by $P(\theta)$ and how would I calculate this?

In some sense, when I am analyzing data relative to a model, aren't the parameters under my control? And if so, wouldn't I use something like a uniform distribution for $P(\theta)$? If the parameters are continuous, wouldn't this value be 0? How do I connect $P(\theta)$ to ground truth?

Answer 1

$P(\theta)$ is a prior, the assumed distribution of the parameter $\theta$. It is not something that you estimate¹, but something that describes what you know, or assume, about the parameter before seeing the data. If you refuse to make prior assumptions, then you can use so-called "uninformative" priors (e.g. uniform), but no prior is truly uninformative (check here for examples of different "uninformative" priors, that lead to slightly different results). Bayesian would also argue that you always have and can make some prior assumptions. You choose the prior according to your best knowledge, e.g. you know that it should be continuous, positive-valued and skewed, so you choose gamma distribution with parameters that fit your assumption (most of the probability mass is centered on the values you consider most likely etc.). As about choosing your priors, check for example the Eliciting priors from experts thread.

It is not true that for continuous random variables the prior is zero, the probability is zero, but the probability density function is non-zero.

_{¹ Unless you are using empirical Bayes approach, where the priors are estimated from the data.}