I am new to the machine learning area. Then I am studying a paper that considers the problem of logistic regression in Bayesian. In general, in regression or logistic regression when they assume the prior or likelihood is Gaussian why in the formula of Gaussian $p(x)= \frac{1}{\sqrt{2\pi \sigma^2}}\mathrm{exp}(-\frac{(x-\mu)^2}{2\sigma^2})$ instead of $x$ they put $y_i-\theta^tx_i$?

  • $\begingroup$ The part that gets squared is the difference from the mean. In linear regression the conditional mean of $Y|X=x$ is $Y - \theta x$. $\endgroup$ Jun 2, 2020 at 12:59
  • $\begingroup$ Your edit doesn't characterize the use of the formula correctly. In the formula, which is a function of three variables $x,\mu,\sigma,$ you plug in "$y_i$" for "$x$" and "$\theta^\top x_i$" for "$\mu.$" (Note, also, that $\sigma^2,$ not $\sigma,$ should appear under the square root in the normalizing factor.) $\endgroup$
    – whuber
    Jun 2, 2020 at 14:41
  • $\begingroup$ you mean in $p(x)$, when we are dealing with Gaussian and regression, we consider x as $y_i$, and mean as $\mu$? Then, why the book write suppose mean is zero? $\endgroup$
    – Raz
    Jun 2, 2020 at 14:48
  • $\begingroup$ The formula uses "$\mu,$" not $0,$ for the mean. The only thing going on here is that the formula you quote is being applied directly to the supposition that the mean of the random variable "$y_i$" is $\theta^\top x_i.$ This isn't even a matter of statistics or mathematics; it's purely a matter of notation: that is, applying the formula through substitution. $\endgroup$
    – whuber
    Jun 2, 2020 at 16:39

1 Answer 1


The Gaussian assumption, if any, is on the errors of the process, not the regressors. In the simplest case the model is $$y_i=\alpha+\beta x_i+\varepsilon_i$$, where $\varepsilon_i\sim\mathcal N(0,\sigma^2)$, i.e. errors are Gaussian.

If you knew true parameters of the model you could infer the errors $\varepsilon_i=y_i-(\alpha+\beta x_i)$. You could interpret this as measured value $y_i$ being a random variable with mean $\alpha+\beta x_i$, i.e. $y_i\sim\mathcal N(\alpha+\beta x_i,\sigma^2)$. Notice how we replace a constant $\mu$ in a textbook Gaussian distribution expression with the model value $(\alpha+\beta x_i)$.

The errors are not observed, so we can estimate them as residuals $\hat \varepsilon_i\equiv r_i=y_i-(a+b x_i)$. Then we write the MLE equation and estimate $a,b,\hat\sigma$ from it. The estimates are of true parameters $\alpha,\beta,\sigma$ that you denoted collectively as $\theta$.

Actually, in many cases we don’t need the errors to be Gaussian IID, because it is enough that they have finite variance and are uncorrelated, plus some other assumptions. We then show that for large enough sample due to CLT the parameter estimates are Gaussian around true parameters. This way the requirements on the dataset are less restrictive and more realistic. Gaussian iid requirement on errors is often unrealistic, and it's a very strong assumption that is hard to test.

Requiring Gaussian assumption on regressors would render the model almost useless in most applications. Imagine you are conducting an experiment and setting X at a few chosen values, e.g. a temperature of the oven at $x=100,200,300,400F$. First, X is not random at all. Second, it’s distribution is not Gaussian. However, the regression can be applied in these type of settings.

  • 1
    $\begingroup$ The language in this answer may mislead readers. We do not assume the residuals are iid Gaussian as suggested by the notation. We assume the errors are iid Gaussian, as you state, and we estimate their common variance with $\hat\sigma^2.$ Mathematics shows that the residuals are correlated: that's an ineluctable consequence of the OLS or ML estimates. The likelihood is expressed in terms of the distribution of the errors, not the residuals. $\endgroup$
    – whuber
    Jun 2, 2020 at 13:41
  • $\begingroup$ @whuber, clarified language $\endgroup$
    – Aksakal
    Jun 2, 2020 at 15:40
  • $\begingroup$ @Aksakal Thank you so much, I guess I understand it. $\endgroup$
    – Raz
    Jun 2, 2020 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.