4
$\begingroup$

I am studying simple linear regression for the very first time, and I'm having a little trouble understanding something. If someone can clarify this for me and perhaps lead the explanation to a little introduction/motivation behind simple linear models that would be really helpful.

What I've understood is that we have a random variable $Y$ we think is linearly related to a random variable $X$.

But then I've seen the "model" that we use is $Y = \beta_0 + \beta_1 X + \epsilon $. If our assumption was that $X$ and $Y$ are linear, then why did we add an error term? Aren't they exactly linearly related, under our assumption?

I would understand that $y_i = \beta_0 + \beta_1 x_i + \epsilon_i $ was used, perhaps indicating that "While $X$ and $Y$ are perfectly linearly correlated, when we are observing values we have measurement errors and other factors affect this from being a perfect observation.

I thought that maybe our assumption is not that $Y$ and $X$ are exactly linear transforms of each other, but rather just "correlated". If this is a more accurate explanation of our assumption, then the epsilon would make more sense.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

The more correct "model" is that $\mathbb{E}(Y|X) = \beta_0 + \beta_1 X$. Since $Y$ is a random variable, there's no way to know for sure what $Y$ will be. That's why there's a need for the (conditional) expectation.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Yes the model you provide makes sense. Can I please confirm something: Are we saying "we don't know the bivariate distribution between $X$ and $Y$, but we guess it to be linear." So $Y \approx \beta_0 + \beta_1 X$, alternatively written as $Y = \beta_0 + \beta_1 X + \epsilon$? $\endgroup$
    – user523384
    Sep 17, 2019 at 7:10
  • $\begingroup$ Is the goal then to find $\beta_0$ and $\beta_1$ such that $\epsilon$ is a minimum? $\endgroup$
    – user523384
    Sep 17, 2019 at 7:11
  • $\begingroup$ We don't know anything. We assume that $X$ and $Y$ have the joint distribution such that when you restrict $X$ to be some value $x_0$, the distribution of $Y$ is given by a normal distribution with mean $\beta_0 + \beta_1 x_0$. The goal is to minimize the sum of squares of $\varepsilon$'s. $\endgroup$
    – Art
    Sep 17, 2019 at 7:13
  • $\begingroup$ Thanks for that comment, it makes a lot of sense. So now that we have made this normal distribution assumption, what is the role of sample data? It is used to estimate $\beta_0$ and $\beta_1$ which the sum of squares of $\epsilon$? How would you relate this sample data with the original model containing the expectation? $\endgroup$
    – user523384
    Sep 17, 2019 at 7:27
  • $\begingroup$ To be more clear, say now I'm trying to model $\mathbb{E}(Y|X) = \beta_0 + \beta_1 X$. I have data points $(x_i,y_i)_{i=1,...,n}$. Would you replace $X$ and $Y$ with these data points in the equation $\mathbb{E}(Y|X) = \beta_0 + \beta_1 X$ to get a system of equations to in turn estimate $\beta$'s? $\endgroup$
    – user523384
    Sep 17, 2019 at 7:29
3
$\begingroup$

If our assumption was that X and Y are linear, then why did we add an error term? Aren't they exactly linearly related, under our assumption?

Linear here implies linear plus noise.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thank you! So would it be right to say "we assume $X$ and $Y$ are "almost" linear, meaning $Y$ is pretty much $a+bX$, but it also has some dependence on other unknown/expensive to identify factors leading to some disruption and we correct $a+bX$ by adding a random variable $\epsilon$" Our additional assumptions about $\epsilon$ complete our model. It is still a model despite our "correction factor $\epsilon$" because we can't accurately claim what $Y$ will be given we know $X$? We can only predict what $\epsilon$ might be? $\endgroup$
    – user523384
    Sep 17, 2019 at 7:37
  • $\begingroup$ The way I see it, linear regression is blind to the fact that you're data follows an "almost linear" trend. A more precise view is to say, if your data is linear plus some additive perturbation (preferably Gaussian), then linear regression is able to provide a "good" approximation of your data. This confirms the second part of what you said: "it is still a model despite our ..." $\endgroup$
    – idnavid
    Sep 18, 2019 at 4:10
  • $\begingroup$ Oh wow I think that makes sense. Would it be fair of me to say this is somewhat analogous to linear taylor approximation? Any (reasonable) function can be approximated with a linear approximation, but whether it's "good" or depends on how close to linear your data actually is. The taylor approximation doesn't care how close to linear you are, it's just more accurate when you're linear? $\endgroup$
    – user523384
    Sep 19, 2019 at 12:12
  • $\begingroup$ I guess you can think of it that way. I personally never made that connection, but if it helps you "project" this to your world-view (pun intended :)) ), then sure. $\endgroup$
    – idnavid
    Sep 20, 2019 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.