why in Gaussian linear regression they put $y_i-\theta^Tx_i$ instead of $x$?

Question

I am new to the machine learning area. Then I am studying a paper that considers the problem of logistic regression in Bayesian. In general, in regression or logistic regression when they assume the prior or likelihood is Gaussian why in the formula of Gaussian $p(x)= \frac{1}{\sqrt{2\pi \sigma^2}}\mathrm{exp}(-\frac{(x-\mu)^2}{2\sigma^2})$ instead of $x$ they put $y_i-\theta^tx_i$?

Answer 1

The Gaussian assumption, if any, is on the errors of the process, not the regressors. In the simplest case the model is $$y_i=\alpha+\beta x_i+\varepsilon_i$$, where $\varepsilon_i\sim\mathcal N(0,\sigma^2)$, i.e. errors are Gaussian.

If you knew true parameters of the model you could infer the errors $\varepsilon_i=y_i-(\alpha+\beta x_i)$. You could interpret this as measured value $y_i$ being a random variable with mean $\alpha+\beta x_i$, i.e. $y_i\sim\mathcal N(\alpha+\beta x_i,\sigma^2)$. Notice how we replace a constant $\mu$ in a textbook Gaussian distribution expression with the model value $(\alpha+\beta x_i)$.

The errors are not observed, so we can estimate them as residuals $\hat \varepsilon_i\equiv r_i=y_i-(a+b x_i)$. Then we write the MLE equation and estimate $a,b,\hat\sigma$ from it. The estimates are of true parameters $\alpha,\beta,\sigma$ that you denoted collectively as $\theta$.

Actually, in many cases we don’t need the errors to be Gaussian IID, because it is enough that they have finite variance and are uncorrelated, plus some other assumptions. We then show that for large enough sample due to CLT the parameter estimates are Gaussian around true parameters. This way the requirements on the dataset are less restrictive and more realistic. Gaussian iid requirement on errors is often unrealistic, and it's a very strong assumption that is hard to test.

Requiring Gaussian assumption on regressors would render the model almost useless in most applications. Imagine you are conducting an experiment and setting X at a few chosen values, e.g. a temperature of the oven at $x=100,200,300,400F$. First, X is not random at all. Second, it’s distribution is not Gaussian. However, the regression can be applied in these type of settings.