In the Bayes formula, $$ p(\theta|D) = \frac{ p(D|\theta) p(\theta) }{ p(D) } $$

  • which of $D, \theta$ should we regard as given and which as variables,

  • which of the factors $p(\theta|D), p(D|\theta), p(\theta), p(D)$ are valid probabilities

  • which of $D, \theta$ can refer to more than one data point?

This is not for a class.

To explain the question:

$P(D|\theta)$ is the likelihood, which is described as proportional to a probability, but not itself a probability. I believe it is true that for fixed $\theta$, $\int p(D|\theta) d D = 1$, seems like this must be true. But with $D$ fixed and $\theta$ variable, the integral $\int p(D|\theta) d\theta$ is not necessarily one.

I assume that $p(\theta|D)$ is a probability, otherwise the normalizing constant in the denominator woudl not be necessary. I assume that $\int p(\theta|D) d\theta = 1$, but integrating the posterior with respect to $D, \int p(\theta|D) d D$, is not necessarily onee. Is that true?

Often $D$ is written in font $\mathcal{D}$ (look closely), which I believe is a hint that $D$ can describe one data point or multiple data points or all the data. For example, the likelihood of the data is often assumed to factor across individuald data points. However I believe $\theta$ is a single thing (though possibly a vector) regardless of the data.


1 Answer 1

  • which of $D, \theta$ should we regard as given and which as variables,

Both are random variables, otherwise they wouldn't have probability distributions. Yet, it may be the case that some of the parameters are given, e.g. you estimate mean of normal distribution given the data, prior and known variance, so you have only two, not three, random variables to consider ($D$ and $\mu$, but not $\sigma^2$).

  • which of the factors $p(\theta|D), p(D|\theta), p(\theta), p(D)$ are valid probabilities

All of them are valid probabilities, or probability densities. The likelihood $p(D|\theta)$ in Bayesian setting is conditioned on random variable (not fixed point, as in maximum likelihood), so it is a valid conditional distribution.

  • which of $D, \theta$ can refer to more than one data point?

$D, \theta$ are random variables, not the observed points. You apply the Bayes theorem to two random variables and their distributions.

This may be more clear, it I give you an example, e.g. the (obligatory) coin example. Say that you've flipped a coin $n$ times and observed $k$ heads, given this data, you want to estimate the probability of throwing heads. This translates to the following model

$$ k|\pi \sim \mathcal{Bin}(n, \pi) \\ \pi \sim \mathcal{Beta}(\alpha,\beta) $$

where $k$ follows the binomial distribution (the likelihood) and we use beta distribution as a prior for $\pi$. In terms of Bayes theorem, this is

$$ p(\pi|k) \propto p(k|\pi) \; p(\pi) $$

where all the three components are random variables and all have distributions. In this case $n$ is fixed, so different authors would use different notational conventions and either include $n$ in the Bayes theorem formula, or not (I didn't). There are also two hyperparameters $\alpha,\beta$ that are fixed (assumed a priori).

Notice that this is a simple case and most of the textbooks, blogs, or tutorials, will describe such simple examples, while most of the real-life examples would get much more complicated (yet the same general principles apply). Also notice that Bayesian updating lets us process data all at once, or one point at a time, and both approaches would lead to same results. Many examples would describe simple case of updating given single, so instead of describing a model for some number of i.i.d. random variables ("you have ten samples of..."), it would be described in terms of single variable for simplicity.

  • $\begingroup$ I am mostly ok with this, but not comfortable yet with the statement about the likelihood being a valid probability. Textbooks show how the likelikhood does not integrate to one, and seem to not make any distinct for the Bayesian case, it is the same likelihood as before. $\endgroup$ Apr 11, 2018 at 13:19
  • $\begingroup$ @isolatedstudent likelihood is a conditional probability, same as any other conditional probability, there is nothing invalid about it. $\endgroup$
    – Tim
    Apr 11, 2018 at 13:23
  • $\begingroup$ I hesitate to disagree because I am a "newb", but: In the Bishop PRML textbook, p.22, it says "note that the likelihood is not a probability distribution over w, and its integral with respect to w does not (necessarily) equal 1". $\endgroup$ Apr 12, 2018 at 5:05
  • $\begingroup$ So I think the issue is considering p(D|w) as a function of D, versus considering it as a function of w. But, this is what I hope to have clarified in this question above! $\endgroup$ Apr 12, 2018 at 5:06
  • $\begingroup$ @isolatedstudent frequentist likelihood is a function of $D$ given fixed $\theta$, while Bayesian likelihood is a conditional probability $p(D|\theta)$ where both arguments are random variables. Conditional probability involves two variables, by definition $p(A|B)= p \frac{p(A,B)}{p(B)}$. Bayes theorem is e theorem about probabilities, it let's you calculate $p(A|B)$ given $p(B|A)$ and vice versa. See stats.stackexchange.com/questions/224037/… $\endgroup$
    – Tim
    Apr 12, 2018 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.