Given a classical linear regression model

$$y = X\beta + \varepsilon,$$

$$\varepsilon\sim N(0,\sigma^2I_n),$$

the posterior density is proportional to the product of the likelihood and the selected prior density,

$$f(\beta,\sigma^2\mid X,y)\propto f(y\mid X,\beta,\sigma^2)f(\beta,\sigma^2).$$

For ease of notation, I write $\theta = (\beta, \sigma^2)$. So,

$$f(\theta\mid X,y)\propto f(y\mid X, \theta) f(\theta). \tag 1$$ I'm interested in finding the integrating constant, i.e. what makes the "proportional to" in $(1)$ to an "is equal to".

As every joint pdf can be factorized into its conditionals, it follows that

$$f(\theta,X,y)=f(\theta)f(X\mid \theta) f(y\mid X,\theta) \tag 2$$

From Eq. $(2)$ I would follow that

$$f(\theta\mid X,y) = \frac{f(\theta) f(X\mid\theta) f(y\mid X,\theta)}{f(X,y)}.$$

So what makes the relation $(1)$ equal is the division of the right-hand side of $(1)$ by

$$\frac{f(X,y)}{f(X\mid \theta)}.\tag 3$$

However, the term $(3)$ does depend on $\theta$ and thus cannot be the integrating constant. This is what puzzles me right now. I think I have some faulty reasoning here, but I can't see it.

Is it maybe that $X$ and $\theta$ must be assumed to be stochastically independent so that $f(X\mid \theta) = f(X)$ and in this case, $(3)$ would become

$$\frac{f(X,y)}{f(X)}=f(y\mid X). \tag{3'}$$

In this case, $(3')$ would be the integrating constant of the right-hand-side of $(1)$, because $(3')$ does not depend on $\theta$. The posterior density would then be given by

$$f(\theta\mid X,y)=\frac{f(y\mid X,\theta)f(\theta)}{f(y\mid X)}. $$

I would appreciate any help.

  • $\begingroup$ I extensively cleaned up your MathJax usage. But I left intact the abominable usage by which the same symbol is used for both a random variable and the argument to its density function, and the same symbol for multiple different densisty functions. $\endgroup$ – Michael Hardy Mar 3 at 17:37

Without any context, the proportional relationship is as follows: $f(\theta\mid X,y)\propto f(y\mid X,\theta)f(\theta\mid X)$, and the integrating constant is $p(y\mid X)$ if $X$ is given; but in (1), you write $f(\theta\mid X,y)\propto f(y\mid X,\theta)f(\theta)$ with implicit assumption that $f(\theta\mid X)=f(\theta)$, which also validates your reasoning: $f(X\mid \theta)=f(\theta)$, i.e. $X$ and $\theta$ are assumed to be stochastically independent. This is a natural assumption since one belongs to the model and the other is the data (without the output variable).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.