Confused with the fundamental assumptions of Frequentist and Bayesian Linear Regression

Question

In Frequentist Linear Regression, I have seen 2 approaches which lead to basically similar models. We have $W,y,X,\epsilon$ related as $y=W^TX+\epsilon$, where $y$ is the dependent random variable, and $X$ is assumed to be a constant (first approach), or random (second) independent variable. $\epsilon$ is assumed to be the Gaussian error. Now let us say we assume $X$ as a random variable, of which, we don't know the probability distribution. (At least the sources I've read don't talk about its distribution)
We also write the the data as $\{(x_i,y_i)_n\}$, and this notation is widely used: $$p(D)=p(y|X)\tag{i}$$ where "$D$" is often called the Data. ((1)Is it a random variable?)
Well then, we get (assuming parameters of $\epsilon$ to be constant), $$p(y|W,X)=N(W^TX,\sigma_\epsilon^2)$$ and compute the MLE.
Now coming to the semi Bayesian, we know a prior distribution of $W$. Now we wish to know the posterior, given the data D. That is, $$p(W|D)=_{\text{Def of Conditional Probability}} \frac{p(W,D)}{p(D)}=\frac{p(D|W)p(W)}{p(y|X)}$$ (2)Now, how do we compute $p(D|W)$? (All we are given is $(i)$, about $p(D)$)

Now coming to the pure Bayesian, we basically want $$p(y|X,D) =\frac{p(y,X,D)}{p(x,D)}$$ I was told at school that this equals $$\int_Wp(y|W,X)p(W|D)dW$$ (3)How to arrive at this?
Please use only basic stuff like definitions to derive this. I have found so many interpretations and ways of these, that I literally have no idea what is the correct way to look at it. So please provide answers to the questions in bold.
Also, if there is any mistake in the above reasoning, please point out.

Answer 1

Is $D=\{(X_i,y_i)_n\}$ a random variable?

Yes, since $y_i$ are random variables. Are $X_i$ random variables? Theoretically yes, but is not always useful to consider them as such. Since the regression weights $W$ only affect the distribution of $y_i|X_i$, we can say:

$$p(W|D)=p(W|X,y)=\frac{p(W,x,y)}{\int\int p(W,x,y)dxdy}=\frac{p(W)p(X)p(y|X,w)}{\int p(W')p(X)p(y|X,W')dW'}=\frac{p(W)p(y|X,w)}{\int p(W')p(y|X,W')dW'}$$

Since $p(X)$ cancels in the calculation of the posterior $p(W|D)$, we can safely ignore the distribution of $X$ and consider it fixed instead of random.

There are, of course, some exceptions: when $X$ includes some missing data, considering $X$ as random allows us to manage imputation in a quite principled way: we just marginalize the missing values out when finding the posterior. But in the absence of missing values, I do not see any advantages to considering $X$ as random with some distribution $p(X)$.

Also, note that $p(D)=p(y|X)$ when $X$ is fixed. When $X$ is considered random, $p(D)=p(y,X)=p(y|X)p(X)$.

How do I calculate $p(D|W)$?

Since you assume a normal model, where $p(y|X,W)$ is normal, you have simply: $$p(D|W)=\prod_{i=1}^n p(X_i,y_i|W)=\prod_{i=1}^n p(y_i|X_i,W)p(x_i)$$

If you take $x$ as fixed, you can safely ignore $p(X)$ in the above equation.

How to arrive at $p(y^\star|X^\star,D)=\int p(y^\star|W,X^\star)p(W|D)dW$?

This equation refers to the posterior predictive distribution: after we fit our model on data $D$, how do I predict the value $y^\star$ given the covariate vector $X^\star$? (I added $\star$ here to emphasize that $X^\star$ and $y^\star$ refer to new observations not included in the original data $D$).

Then, the equation you wrote follow by applying the law of total probability. In details, we have:

$$p(y^\star|X^\star,D)=p(y^\star|X^\star,D)\cdot\int p(W|D)dW \\= \int p(y^\star|X^\star,D)\cdot p(W|D)dW=\int p(y^\star,W|X^\star,D)dW\\=\int p(y^\star|W, X^\star)p(W|D)dW$$

You might find these questions about the law of total probability useful:

• Can we prove the law of total probability for continuous distributions?
• How to think about Continuous Total Probability Theorem applied to a conditional?

Hope it was helpful!