2
$\begingroup$

In Frequentist Linear Regression, I have seen 2 approaches which lead to basically similar models. We have $W,y,X,\epsilon$ related as $y=W^TX+\epsilon$, where $y$ is the dependent random variable, and $X$ is assumed to be a constant (first approach), or random (second) independent variable. $\epsilon$ is assumed to be the Gaussian error. Now let us say we assume $X$ as a random variable, of which, we don't know the probability distribution. (At least the sources I've read don't talk about its distribution)
We also write the the data as $\{(x_i,y_i)_n\}$, and this notation is widely used: $$p(D)=p(y|X)\tag{i}$$ where "$D$" is often called the Data. ((1)Is it a random variable?)
Well then, we get (assuming parameters of $\epsilon$ to be constant), $$p(y|W,X)=N(W^TX,\sigma_\epsilon^2)$$ and compute the MLE.
Now coming to the semi Bayesian, we know a prior distribution of $W$. Now we wish to know the posterior, given the data D. That is, $$p(W|D)=_{\text{Def of Conditional Probability}} \frac{p(W,D)}{p(D)}=\frac{p(D|W)p(W)}{p(y|X)}$$ (2)Now, how do we compute $p(D|W)$? (All we are given is $(i)$, about $p(D)$)

Now coming to the pure Bayesian, we basically want $$p(y|X,D) =\frac{p(y,X,D)}{p(x,D)}$$ I was told at school that this equals $$\int_Wp(y|W,X)p(W|D)dW$$ (3)How to arrive at this?
Please use only basic stuff like definitions to derive this. I have found so many interpretations and ways of these, that I literally have no idea what is the correct way to look at it. So please provide answers to the questions in bold.
Also, if there is any mistake in the above reasoning, please point out.

$\endgroup$
2
  • $\begingroup$ In (i) if D is (x,y) then $p(D)=p(x,y) \neq p(x|y)$ $\endgroup$ Sep 19, 2020 at 20:06
  • $\begingroup$ @user0, okay, but I still don't see how it solves the other issues. Can you help me with that? $\endgroup$ Sep 19, 2020 at 20:32

1 Answer 1

5
$\begingroup$

Is $D=\{(X_i,y_i)_n\}$ a random variable?

Yes, since $y_i$ are random variables. Are $X_i$ random variables? Theoretically yes, but is not always useful to consider them as such. Since the regression weights $W$ only affect the distribution of $y_i|X_i$, we can say:

$$p(W|D)=p(W|X,y)=\frac{p(W,x,y)}{\int\int p(W,x,y)dxdy}=\frac{p(W)p(X)p(y|X,w)}{\int p(W')p(X)p(y|X,W')dW'}=\frac{p(W)p(y|X,w)}{\int p(W')p(y|X,W')dW'}$$

Since $p(X)$ cancels in the calculation of the posterior $p(W|D)$, we can safely ignore the distribution of $X$ and consider it fixed instead of random.

There are, of course, some exceptions: when $X$ includes some missing data, considering $X$ as random allows us to manage imputation in a quite principled way: we just marginalize the missing values out when finding the posterior. But in the absence of missing values, I do not see any advantages to considering $X$ as random with some distribution $p(X)$.

Also, note that $p(D)=p(y|X)$ when $X$ is fixed. When $X$ is considered random, $p(D)=p(y,X)=p(y|X)p(X)$.

How do I calculate $p(D|W)$?

Since you assume a normal model, where $p(y|X,W)$ is normal, you have simply: $$p(D|W)=\prod_{i=1}^n p(X_i,y_i|W)=\prod_{i=1}^n p(y_i|X_i,W)p(x_i)$$

If you take $x$ as fixed, you can safely ignore $p(X)$ in the above equation.

How to arrive at $p(y^\star|X^\star,D)=\int p(y^\star|W,X^\star)p(W|D)dW$?

This equation refers to the posterior predictive distribution: after we fit our model on data $D$, how do I predict the value $y^\star$ given the covariate vector $X^\star$? (I added $\star$ here to emphasize that $X^\star$ and $y^\star$ refer to new observations not included in the original data $D$).

Then, the equation you wrote follow by applying the law of total probability. In details, we have:

$$p(y^\star|X^\star,D)=p(y^\star|X^\star,D)\cdot\int p(W|D)dW \\= \int p(y^\star|X^\star,D)\cdot p(W|D)dW=\int p(y^\star,W|X^\star,D)dW\\=\int p(y^\star|W, X^\star)p(W|D)dW$$

You might find these questions about the law of total probability useful:

Hope it was helpful!

$\endgroup$
4
  • $\begingroup$ Amazing. Just an observation, there's no need to "assume" that $X$ is fixed in the second part. The $p(X_i)$ term just cancels out from the denominator! $\endgroup$ Sep 21, 2020 at 13:08
  • $\begingroup$ Great answer :). Just a question. In the third right-hand side term of the expression that defines $p(W|D)$, would it be correct to consider in the numerator $p(W|X)p(X)$ instead of $p(W)p(X)$? $\endgroup$
    – Javier TG
    Sep 21, 2020 at 18:06
  • 2
    $\begingroup$ @JavierTG Good point! It would indeed be correct to consider p(W|X)p(X). When I write it as p(W)p(X), I am assuming than W and X are independent. It makes sense: if I didn't know the values of Y and someone told me the values of the the covariates X, I wouldn't learn anything about W. $\endgroup$
    – PedroSebe
    Sep 22, 2020 at 0:47
  • $\begingroup$ See also (frequentist ...) stats.stackexchange.com/questions/144826/… and stats.stackexchange.com/questions/215230/… $\endgroup$ Oct 29, 2020 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.