I have seen this stated in multiple sources, where if the errors in a linear model ($y_i = \beta x_i + \epsilon_i$) follow $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$, then $x_i|\beta \sim \mathcal{N}(0, \sigma^2)$, the same distribution. Here is one link https://www.youtube.com/watch?v=_-Gnu498s3o that states this, starting at around 2:20.

If the error terms are gaussian distributed, why does this imply that the independent and dependent variables are also gaussian distributed?

  • $\begingroup$ I've watched the video and I believe the creator of the video made a typo when they wrote the likelihood. He wrote $f(x_i | \beta, \sigma^2)$ but he should have wrote $f(y_i | \beta, \sigma^2)$. That is, $$y_i | \beta, \sigma^2 \sim \mathcal N (\beta x_i , \sigma^2)$$ $\endgroup$ – SOULed_Outt May 25 '20 at 6:39
  • $\begingroup$ Also, depending on how you'd like to treat $x_i$ you may want to write the likelihood as $f(y_i | x_i, \beta, \sigma^2)$. $\endgroup$ – SOULed_Outt May 25 '20 at 6:40
  • $\begingroup$ @SOULed_Outt Ah, maybe that's what we meant. I think the latter form you commented is the one that I see the most often. Although, I think I usually see the semicolon usage $f(y_i | x_i ; \beta)$. Andrew NG's CS229 notes uses the semicolon notation to indicate that we're not conditioning on $\beta$. $\endgroup$ – user5965026 May 25 '20 at 6:43
  • $\begingroup$ Perhaps the notation is to make it clearer that you're treating the parameters as fixed values (i.e. not random variables). Then it would be better to say $$y_i | x_i; \beta, \sigma^2 \sim \mathcal N (\beta x_i, \sigma^2)$$ and $$f(y_i | x_i; \beta, \sigma^2)$$ $\endgroup$ – SOULed_Outt May 25 '20 at 8:06

It doesn’t imply anything about the predictors (independent variables) or the response (dependent variable). It is a comment about the conditional distribution of $y$, conditioned on some specified value of $x$.

The idea is that you’re sliding a bell curve up and down the regression line. For example, enter image description here The regression line gives the expected value, but then you draw an observation from the conditional distribution of $y$ given that $x$-value. That’s where the error comes from.

Remember that this framework posits that the conditional distribution is $N(\hat{y}_i, \sigma^2)$.

  • $\begingroup$ Sorry, I'm having a hard time visualizing "sliding a bell curve up and down the regression line." Is this bell curve parallel to the x axis or to the regression line? $\endgroup$ – user5965026 May 25 '20 at 5:52
  • $\begingroup$ blogs.sas.com/content/iml/files/2015/09/GLM_normal_identity.png I’m on my phone right now, but if you edit that into my post, I’ll accept the edit and you’ll get a rep point or two. Otherwise I’ll add it tomorrow. $\endgroup$ – Dave May 25 '20 at 5:53
  • $\begingroup$ Ah I think I get it. So basically the idea is by assuming the errors are normally distributed with zero mean, means that on average, our $y_i$ will fall on the regression line. Is this regression line determined from population parameters or from sample parameters? Also do you know why the video states $f(x_i|\beta)$ is normally distributed. I was really confused by that. $\endgroup$ – user5965026 May 25 '20 at 5:58
  • 1
    $\begingroup$ I meant to add the picture directly in the post, not just a link to it. I’ll address your other comments in the morning. $\endgroup$ – Dave May 25 '20 at 6:04
  • 1
    $\begingroup$ Exactly! The regression line is your predictions, which come from the the fitted equation. I hope this answers your question. If not, please do move this to chat. $\endgroup$ – Dave May 25 '20 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.