I have been reading Tony Lancaster's 2004 book "An Introduction to Modern Bayesian Econometrics." On pages 116-117, Lancaster derives a conditional joint distribution for the data $p(y,X|\boldsymbol{\beta})$ for a linear model $y=X\boldsymbol{\beta}+\epsilon$ using a joint distribution of $X$ and the residuals, $p(\epsilon, X)$. But the substitutions he uses en route to his result don't always seem justified. Are they justified?

He starts out with the goal: posterior inference about $\boldsymbol{\beta}$, given $X$ and $y$, using Bayes Theorem

$$p(\boldsymbol{\beta}|y,X)\propto p(y,X|\boldsymbol{\beta})p(\boldsymbol{\beta})$$

Which requires deriving a conditional joint distribution for the data


His strategy is to first obtain a joint distribution for $\epsilon$ and $X$, under the assumption that $\epsilon$ and $X$ are independent, not just uncorrelated

$$p(\epsilon, X)=p_{\epsilon}(\epsilon)p(X)$$

His next step is to make substitutions into this joint distribution using $y=X\boldsymbol{\beta}+\epsilon$, and then condition on $\boldsymbol{\beta}$, yielding the required conditional joint distribution


I understand the right hand side of this result, simply using $\epsilon=y-X\boldsymbol{\beta}$. But what about the left hand side? It can't be a substitution simply of $y=\epsilon$, as that doesn't make sense. Can someone help me better understand how this result is justified? Am I missing something basic?


1 Answer 1


The left-hand side of your final equation, $p(y,X|\beta$), can be written as $p_y(y|X,\beta)p(X|\beta)$, and writing the right-hand side more explicitly then gives $$ p_y(y|X,\beta)p(X|\beta) = p_\epsilon(y - X\beta|X,\beta)p(X|\beta) $$ Your intuition is right in that generally $p_y(y|X,\beta) \neq p_\epsilon(\epsilon|X,\beta)$. Instead, you have $$ p_y(y|X,\beta) = p_\epsilon(\epsilon|X,\beta) \left| \frac{\mathrm{d} \epsilon}{\mathrm{d} y} \right| $$ where $\left| \frac{\mathrm{d} \epsilon}{\mathrm{d} y} \right|$ is known as the Jacobian of the transformation. But in linear regression, $\epsilon = y - X\beta$ and therefore $\frac{\mathrm{d} \epsilon}{\mathrm{d} y} =1$, which means the Jacobian can be ignored.

I think that's pretty much all there is to say about this. Things would look different if your model was non-linear (for instance, a Box-Cox regression), or if your model was autoregressive, but in linear regression that transformation of variables is trivial.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.