In the linked resource, the author defines the posterior probability of Bayesian linear regression as:

P(B|y,X) = P(y|B,X)*P(B|X)/P(y|X)

I have a couple questions/issues with this.

First, B represents the weight(s) vector, but it shouldn't include the bias term, b, in y=mx+b. Does this model just forget to include a bias or is it somehow factored in B?

Second, I'm used to seeing Bayes Rule in a very specific form:

P(A|B) = P(B|A)*P(A)/P(B)

So it's my naive guess that the posterior would take the following form:

P(B|y,X) = P(y,X|B)*P(B)/P(y,X)

Do the equations, mine & the source, equate to the same value? If not, why am I wrong? I gather that there is something about conditional probabilities and the chain rule that is going over my head.

EDIT: According to this video my interpretation is correct! However, I'm still unclear where the bias term comes into play. Likewise, I'd like to what likelihood and priors are typically used. Wikipedia shows: likelihood

This looks somewhat like a multivariate guassian, without some scaling terms outside of the exponent, and most notably, no precision matrix (aka inverse covariance) in the exponentiated term.

Does this likelihood have a name? Where did it come from? Etc.

  • $\begingroup$ Include the unit vector as a column of $X$. $\endgroup$ – Xi'an Sep 11 '20 at 5:37

To quote from our Bayesian Essentials with R book (Chapter 3, p.67):

The ordinary normal linear regression model is such that $$ \mathbf y|\beta,\sigma^2,X\sim\mathscr{N}_n(X\beta,\sigma^2I_n) \tag{1}$$ and thus $$ \mathbb{E}[y_i|\beta,X]=\beta_0+\beta_1x_{i1}+\ldots+\beta_kx_{ik}\,,\quad \mathbb{V}(y_i|\sigma^2,X)=\sigma^2\,. $$ In particular, the presence of an intercept $\beta_0$ explains why a column of $1$'s is necessary in the matrix $X$ to preserve the compact formula $X\beta$ in the conditional mean of $\mathbf y$.

for the inclusion of an intercept in the regression and (Chapter 3, p.66)

A large proportion of statistical analyses deal with the representation of dependences among several observed quantities. For instance, which social factors influence unemployment duration and the probability of finding a new job? Which economic indicators are best related to recession occurrences? Which physiological levels are most strongly correlated with aneurysm strokes? From a statistical point of view, the ultimate goal of these analyses is thus to find a proper representation of the conditional distribution, $f(y|\theta,\mathbf x)$, of an observable variable $y$ given a vector of observables $\mathbf x$, based on a sample of $\mathbf x$ and $y$.

to stress that the entire analysis is run conditional on $\mathbf X$. With a further excerpt (Chapter 3, p.72):

We stress here that conditioning on $\mathbf X$ is valid only when $\mathbf X$ is exogenous, that is, only when we can write the joint distribution of $(\mathbf y,\mathbf X)$ as $$ f(\mathbf y,\mathbf X|\alpha,\beta,\sigma^2,\delta)=f(\mathbf y|\alpha,\beta,\sigma^2,\mathbf X)f(\mathbf X|\delta)\,, $$ where $(\alpha,\beta,\sigma^2)$ and $\delta$ are fixed parameters. We can thus ignore $f(\mathbf X|\delta)$ if the parameter $\delta$ is only a nuisance parameter since this part is independent of $(\alpha,\beta,\sigma^2)$. The practical advantage of using a regression model as above is that it is much easier to specify a realistic conditional distribution of one variable given $p$ others rather than a joint distribution on all $p+1$ variables. Note that if $\mathbf X$ is not exogenous, for instance when $\mathbf X$ involves past values of $\mathbf y$,the joint distribution must be used instead.

Concerning the likelihood function, it stems from (1) with $\mathbf y$ being indeed a Normal vector with $\sigma^2 \mathbf I_n$ as its covariance matrix. This follows from the Normality assumption on the noise $\mathbf \epsilon$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.