3
$\begingroup$

In linear regression in order to calculate the posterior distribution $ $$p(\mathbf{w}|\mathcal{D})$ we need to perform the Bayes rule using the likelihood, prior and evidence as follows (taken from Bishop's book):

$$p(\mathbf{w}|\mathcal{D}) = \frac{p(\mathcal{D}|\mathbf{w})p(\mathbf{w})}{p(\mathcal{D})}$$

Then, by modelling prior likelihood with Gaussans using as precision $\alpha, \beta$ we can write it as:

$$ p(\mathbf{w}|\mathcal{D})= = \frac{\mathcal{N} \left( \mathbf{w} | \mathbf{0}, \frac{1}{\alpha} I \right) \mathcal{N} \left( \mathbf{t} | \mathbf{\Phi}\mathbf{w}, 1/\beta I\right) }{ \int \mathcal{N} \left( \mathbf{w} | \mathbf{0}, \frac{1}{\alpha} I \right) \mathcal{N} \left( \mathbf{t} | \mathbf{\Phi}\mathbf{w}, 1/\beta I \right)dw } $$

So more compactly we can write the following form: $$ = \frac{\mathcal{N} \left( \mathbf{w} | \mathbf{0}, \frac{1}{\alpha} I \right) \mathcal{N} \left( \mathbf{t} | \mathbf{\Phi}\mathbf{w}, 1/\beta I\right) }{ p(\mathbf{t} | \mathbf{\Phi, \alpha, \beta})}$$

The numerator and the likelihood/prior are clear and straightforward. My question is why the evidence is conditioned to precision parameters $\alpha, \beta$. Is that actually correct?

Is it that $\mathbf{w}$ is marginalized out and therefore, not part of the dependencies?

$\endgroup$
5
  • $\begingroup$ Have you checked MacKay's Bayesian interpolation (also here)? It does a very insightful, principled, step-by-step analysis of the inference and the parameters involved. Recommended reading! It is as you say: there is an assumed underlying model $M$ (stating eg that likelihood and prior are normals etc), which has fixed parameters $\alpha,\beta$. It should be explicitly included in the conditional of all probabilities of your first equation. $\endgroup$
    – pglpm
    Commented Jun 25, 2021 at 9:01
  • $\begingroup$ These parameters make sense to be part of the prior and the likelihood since I am modeling them using these parameters. However, the evidence seems likely to be independent of these parameters. Is that right? $\endgroup$
    – Jose Ramon
    Commented Jun 25, 2021 at 10:01
  • $\begingroup$ So I guess my confusion part is that while the evidence is conditioned to these parameters, it is not conditioned to $\mathbf{w}$ which is marginalized out. Is that right? $\endgroup$
    – Jose Ramon
    Commented Jun 25, 2021 at 11:24
  • $\begingroup$ Thanks for the recommendation also I will have a closer look. $\endgroup$
    – Jose Ramon
    Commented Jun 25, 2021 at 11:25
  • $\begingroup$ One comment: in your second equation, you write the product $p(D|w)\,p(w)$ in reverse order: $N(w|0,\dotsc)$ corresponds to $p(w)$, and $N(\pmb{t}|\dotso)$ corresponds to $p(D|w)$. $\endgroup$
    – pglpm
    Commented Sep 13, 2023 at 9:02

1 Answer 1

3
+50
$\begingroup$

Short answer

The denominator is the evidence for your model (as compared to different models), and since your model is defined by specific values of $\alpha,\beta$, they must implicitly or explicitly be contained in the conditional of the evidence.

Long answer

The evidence in your question must be conditional on the parameters $\alpha,\beta$. This is clear if we ask: evidence for what?

The evidence that data $D$ give to a hypothesis or model $M$ in some context $K$ is simply the probability for the data given the model and the context: $$\mathrm{P}(D \mid M, K) $$ where the comma stands for "and" ($\land$). It is also called the "likelihood or marginal likelihood of the model in view of the data". "Marginal" because it is often calculated using the marginalization rule on other probabilities. All this is just a matter of terminology; what matters is to specify clearly the probability of what given what.

The evidence is important when calculating and comparing the probabilities of mutually exclusive hypotheses or models $M',M'',\dotsc$ given some data, because it enters these probabilities directly: $$\mathrm{P}(M' \mid D, K) = \frac{\mathrm{P}(M'\mid D, K)\ \mathrm{P}(M' \mid K)}{ \mathrm{P}(M'\mid D, K)\ \mathrm{P}(M' \mid K) + \mathrm{P}(M''\mid D, K)\ \mathrm{P}(M'' \mid K) + \dotsb} $$ in particular if all models have equal probabilities on $K$, then $$ \frac{\mathrm{P}(M' \mid D, K)}{\mathrm{P}(M'' \mid D, K)} = \frac{\mathrm{P}(D \mid M', K)}{\mathrm{P}(D \mid M'', K)} $$ and similarly for all pairwise comparisons.

For these matters see for example


It's maybe useful to make clear also what we mean by "model" – a word widely used but often poorly defined. A model $M$ is a set of assumptions that, in some fixed context $K$, allows us to give definite numerical probabilities to a set, often infinite, of observations, or measurement outcomes, or other empirically verifiable statements that we're interested in. Such probabilities are often called "predictive".

In your case, for example, you are ultimately interested in a potentially infinite set of outcomes $\{Y_i = y_i \mid i\in\{1,2,\dotsc\}\}$ given some inputs $\{X_i = x_i\}$. Your model $M$, somewhat simplified, leads to the following probabilities, among others: \begin{multline} \mathrm{P}(Y_1 = y_1,\, Y_2 = y_2 \mid X_1 = x_1,\, X_2 = x_2, M, K) = {}\\ \int \mathrm{N}(y_1 \mid w x_1, \beta)\ \mathrm{N}(y_2 \mid w x_2, \beta)\ \mathrm{N}(w \mid 0, \alpha)\ \mathrm{d}w \ , \end{multline} \begin{multline} \mathrm{P}(Y_1 = y_1,\, Y_2 = y_2,\, Y_5 = y_5 \mid X_1 = x_1,\, X_2 = x_2,\, X_5 = x_5, M, K) = {}\\ \int \mathrm{N}(y_1 \mid w x_1, \beta)\ \mathrm{N}(y_2 \mid w x_2, \beta)\ \mathrm{N}(y_5 \mid w x_5, \beta)\ \mathrm{N}(w \mid 0, \alpha)\ \mathrm{d}w \ , \end{multline} and so on, for all possible values of $y_1, x_1,y_2,x_2,\dotsc$. Here $\mathrm{N}$ represents a (multivariate) normal with given mean and precision, and $\alpha$ and $\beta$ are given numbers (or positive-definite matrices).

This model $M$ consists of many, many assumptions, difficult to express verbally: assumptions of linear dependencies, assumption of normality of variations with particular standard deviations $\alpha,\beta$, and so on. Note that if we say that we don't know $\alpha,\beta$, then this is not a model, because we cannot give definite numerical values to the probabilities above. This definition of "model" is implicit in MacKay above, in some of Fortini & al's papers, and partially also in

  • Jaynes above, for example chapters 4, 6, 20;
  • Bernardo, Smith: Bayesian Theory, chapters 4–6.

Particular values of $\alpha,\beta$ are part of the specification of your model, just like normality, linearity, and so on, and in principle they are no more nor less important than all other specifications.

But now suppose we want to compare this model with another model $M'$ that is identical in all assumptions, except for different values $\alpha', \beta'$. This is a different model, because it leads to different values of the predictive probabilities. Instead of $M,M'$ we could then use the notation $M_{\alpha,\beta}, M_{\alpha',\beta'}$ to emphasize the actual difference between these two models. But note that if we were to compare $M$ with another model $M''$ that differs, say, in the use of Gumbel distributions instead of normals, but with same precisions $\alpha,\beta$, then we could use the notation "$M_{\textrm{N}}$, "$M_{\textrm{G}}$" instead.

Given some data $D$ and equal probabilities on $K$, the evidence for model $M_{\alpha,\beta}$ is $$ \mathrm{P}(D \mid M_{\alpha,\beta}, K) \ , $$ which is the expression in the denominator in your question. Similarly for the evidence of model $M_{\alpha',\beta'}$. You see that this evidence must in particular depend on the specific values of $\alpha$ and $\beta$, because it is the evidence for the model having those specific values.

Actually the notation "${}\mid \pmb{\phi},\pmb{\alpha},\pmb{\beta})$" is just a shorthand, because the conditioning is not only on these specific parameter values, but also on all other assumptions underlying model $M$.

It's maybe good to emphasize that this is not evidence for some specific value of $w$ within a comparison with other possible values of that parameter. Rather, it is evidence in a comparison between model $M \equiv M_{\alpha,\beta}$, model $M_{\alpha',\beta'}$, and maybe other similar models. If no other models are considered besides $M$, then it's a bit pointless to call $\mathrm{P}(D \mid M_{\alpha,\beta}, K)$ "evidence": our model $M$ is effectively just part of a fixed context $K' = M \land K$.

We could, however, also call "evidence" the first factor in the numerator of your expression. It is the evidence for a model $Q$ characterized by predictive probabilities such as \begin{multline} \mathrm{P}(Y_1 = y_1,\, Y_2 = y_2 \mid X_1 = x_1,\, X_2 = x_2, Q, K) = {}\\ \mathrm{N}(y_1 \mid w x_1, \beta)\ \mathrm{N}(y_2 \mid w x_2, \beta) \ , \end{multline} \begin{multline} \mathrm{P}(Y_1 = y_1,\, Y_2 = y_2,\, Y_5 = y_5 \mid X_1 = x_1,\, X_2 = x_2,\, X_5 = x_5, Q, K) = {}\\ \mathrm{N}(y_1 \mid w x_1, \beta)\ \mathrm{N}(y_2 \mid w x_2, \beta)\ \mathrm{N}(y_5 \mid w x_5, \beta) \end{multline} and so on, with a specific value of $w$. Such value is part of the specification of this model, which can also be denoted by "$Q_w$" if we plan to compare it with other models that are identical but for different values $w'$, $w''$, etc. From this point of view your equation can be re-interpreted as $$\mathrm{p}(Q_w \mid D, K') = \frac{\mathrm{P}(D \mid Q_w, K')\ \mathrm{p}(Q_w \mid K')}{\int \mathrm{P}(D \mid Q_w, K')\ \mathrm{p}(Q_w \mid K')\ \mathrm{d}w} $$ and clearly $\mathrm{P}(D \mid Q_w, K')$ is the evidence for $Q_w$ given by the data in the context $K'$.

From this alternative point of view you see that it is important to make clear what is the evidence for, and in which context.

$\endgroup$
4
  • $\begingroup$ Why de do have p(Y|X, a, b) instead of p(X,Y|a, b) since D is the X, Y and not just the target values Y? $\endgroup$
    – Jose Ramon
    Commented Sep 12, 2023 at 21:12
  • 1
    $\begingroup$ That's a very smart and relevant question in my opinion! The cool thing is that you can use either: the result of the calculation will be the same (it's an example of the self-consistency of Bayesian theory). The reason is that the models have nothing to say on the plausibility of $X$ values, so their probabilities factorize out: $p(X,Y|M,K) = p(Y|X,M,K) p(X|M,K)$ ${} = p(Y|X,M,K) p(X|K)$ and then simplify in the fractions. $\endgroup$
    – pglpm
    Commented Sep 12, 2023 at 22:30
  • $\begingroup$ Thanks for your answer! I understand that the one parameter $M$ is dropped since it models just the prior. However, the second parameter $K$ models the conditional (that stems from likelihood). I am not really confident about why we are dropping it ;)) $\endgroup$
    – Jose Ramon
    Commented Sep 13, 2023 at 6:43
  • $\begingroup$ Mmm... wait, I don't follow you, maybe we're misunderstanding each other? I used "$M$" for "model", that is, the parameters $\pmb{w}$ in your case; and "$K$" for the background information outside of the model, that is, the parameters $(\alpha,\beta,\pmb{\varPhi})$. $\endgroup$
    – pglpm
    Commented Sep 13, 2023 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.