I have recently commenced a 2nd-year course on linear models and have been a little overwhelmed by either the abuse of notation or the lack of clarity behind what a linear model is. I've read multiple related MSE Q's, websites, gone to many books, seen the lecture notes but I still don't feel like I've understood precisely "what is what" in the model. I've summarised below some doubts I've come across,

  1. What exactly are we trying to approximate/investigate? Is it a relationship between random variables $X$ and $Y$? Is it between $x_i$ and $y_i$ for a set of data points $\{(x_i,y_i): i=1,..,n \}$ that we have? Are $x_i$ and $y_i$ realisations of the random variables $X$ and $Y$? Does this mean we are trying to infer a relationship between the random variables $X$ and $Y$ based on the relationship we see between a sample of their realisations $(x_i,y_i)$?

  2. What is a precise statement of our model equation? I've seen a few equations, and i'm not sure whether one follows from the other, or if they're two equivalent statements of the same concept?: For example: $Y = \beta_0 + \beta_1 X + \epsilon$, I've also seen $\mathbb{E}(y) = \beta_0 + \beta_1 x$ (which I'm unclear what is random or not, cause they are lowercase). I've also seen a model dealing with data points and not random variables, for example $y_i = \beta_0 + \beta_1 x_i + \epsilon$. I've also seen $\mathbb{E}(Y|X) = \beta_0 + \beta_1 X$.

  3. What variables exactly are random, and what variables are "realisations" of random variables, and what variables are actually random but we "treat" them as constants because the model is conditional?

  4. Is the error term $\epsilon$ random or is it fixed? I realise almost everywhere I've seen this to be "random", but if the model we are using is one with data points $y_i = \beta_0 + \beta_1 x_i + \epsilon$, then how can $\epsilon$ be random if $y_i$ is the $y$ value of a observation which should be fixed? For this equation, would it not be more correct to have the residual there rather than an "error" term?

  5. Is a residual a realisation of the error term $\epsilon$ for a specific data point?

Request: if someone could provide a precise description of a linear model, specifically referencing the above doubts, I would really really appreciate it. Please be specific about whether a variable is random or not etc.

Apologies for including multiple questions in one, but I thought it would be nice for an overall synopsis of what's going on here.

I realise that some parts of the above questions might be in other MSE asked questions, I have read almost all of them, but I don't yet feel I've understood it yet. If you wish to copy-paste and reference other articles, that's completely fine, but I would please prefer a holistic answer rather than references to multiple articles

Thank you so much


Original Question.

  1. You never estimate something about your data. You estimate something about the populations from which your data are drawn. Therefore, what you want to explore is the relationship between $X$ and $Y$. You do this by examining your observed values of $X$ and $Y$. $x_i$ and $y_i$ are realizations of $X$ and $Y$ that we use to infer something about the relationship between $X$ and $Y$.

  2. Let's unpack what each of those four equations says.

$$Y = \beta_0 + \beta_1 X + \epsilon$$

"The random variable $Y$ is equal to some number plus a multiple of the $X$ variable, plus some random error term."

This describes the phenemonon at the population level, even if the left side would be more technically correct if it said $Y\vert X$. $\epsilon$ gives the shape of the distribution, while $\beta_0 + \beta_1X$ slides the distribution along the real line.

(I'm now seeing this way of writing it as flawed. We look at a particular value of $X$. The overall distribution of $X$ is not so important when we predict $Y$ when $X = 2$.)

$$\mathbb{E}(y) = \beta_0 + \beta_1 x$$

"The expected value of a vector $y$ of observations of $Y$ is equal to some number plus a multiple of of vector $x$ of observations of $X$."

It may help you to write this with vector arrows on top: $\mathbb{E}(\vec{y}) = \beta_0 + \beta_1 \vec{x}$.

$$y_i = \beta_0 + \beta_1 x_i + \epsilon$$

"The i$^{th}$ observation of $Y$ is equal to some number plus a multiple of the i$^{th}$ observation of $X$, plus some random error term." (That random error term ought to be $\epsilon_i$ as the i$^{th}$ observation of that random error term.)

This is just the observation-by-observation version of the previous equation.

$$\mathbb{E}(Y|X) = \beta_0 + \beta_1 X$$

"The expected value of Y, given some other information ($X$) that we know, is equal to some number plus a multiple of the value of that other information."

We get the $\epsilon$ to go away in the equations with expectations because we assume that $\mathbb{E}\big[\epsilon \big] = 0$. (This is a standard assumption.)

  1. The data are observations and are not random. They are whatever you happen to observe. The populations are the random variables. I don't think we treat any random variables as constants. (Did you mean the $\beta_0$ term?)

  2. "Error" and "residual" do NOT have the same meaning. In the population, there is an error term, which is a random variable. Your data give you residuals, which you use to estimate something about the population distribution of the error.

  3. Yes! It's common to speak of errors when we mean residuals. Perhaps "mean squared error" should be "mean squared residual". But we're used to writing MSE.

The gist of regression is that we have too much variability in $Y$, but we know something else ($X$) about that distribution, so we look at the conditional distributions $Y\vert X$. However, there is a whole distribution, not just the point estimate that our regression model gives (which is the estimated expected value of that conditional distribution).

It takes a while to wrap the brain on the idea of having a whole conditional distribution when you're used to getting a single number as your prediction. The idea is that some distribution is sliding up and down the regression line. When $X$ has some value $x_0$, draw a value of the distribution $Y\vert X=x_0$. When $X$ has some value $x_1$, draw a value of the distribution $Y\vert X=x_1$. But there is some distribution sliding up and down the regression line.

I suspect you will have questions. We'll deal with them in the comments or maybe in chat.


(The first response alone was too long for a comment, and I want these in the main answer, too, not buried in the comments or in a chat.)

  1. $Y\vert X=x = \beta_0 + \beta_1 x + \epsilon$ is the most correct way to write it. The full distribution of $X$ isn't so important. (In particular, the Gauss-Markov theorem that you either know or will learn as you study regression more makes no assumptions about the distribution of $X$, even if it appears that way when you first see it.) We don't care what the distribution of $X$ is, only that we want to know about $Y$ when we have some other information. For example, if you want to know about human heights, it would make sense not to compare the heights of adults and children. We don't really care about the distributions of ages, only that $\mathbb{E}\big[\text{Height} \vert \text{adult} \big] \ne \mathbb{E}\big[\text{Height} \vert \text{child} \big] $.

  2. I wouldn't say that they're equivalent, but the math to get from one to the other is easy. We assume $\mathbb{E}\big[ \epsilon \big] = 0$. Then the expectation is an integral and therefore linear. Let's assume that $ Y \vert X$ is shorthand for $ Y \vert X=x$ for an interesting value of $X$ (say an adult subject rather than a child).

$$ Y \vert X = \beta_0 + \beta_1 x + \epsilon$$

  • (The $x$ is intentionally lowercase, since we care about a particular value of $X$ and not the entire distribution.)

$$ \mathbb{E} \big[Y \vert X\big] = \mathbb{E} \big[\beta_0 + \beta_1 x + \epsilon \big]$$

$$ \mathbb{E} \big[Y \vert X\big] = \mathbb{E} \big[\beta_0 \big]+ \mathbb{E} \big[\beta_1 x \big]+ \mathbb{E} \big[\epsilon \big]$$

$$ \mathbb{E} \big[Y \vert X\big] = \beta_0 + \beta_1 x + 0$$

  1. Yes, you want to minimize something about the residuals. $y_i - \hat{y}_i$ is a residual. For many regressions, you minimize $\sum (y_i - \hat{y}_i)^2$, though there are alternatives if you want to get fancy (such as the sum of absolute values instead of squares).
| cite | improve this answer | |
  • $\begingroup$ Thank you so much for your help. Your suspicion will probably be true haha! But can I please start with two short questions first: 1. What in your opinion would be the most "technically correct" model equation? If I've understood correctly, it should be $Y|X = \beta_0 + \beta_1 X + \epsilon$. Is it because technically $Y = \beta_0 + \beta_1 X + W$ where $Y,X,W$ are random? (So the error term should technically be different if you condition on $X$)? $\endgroup$ – user523384 Sep 21 '19 at 7:01
  • $\begingroup$ 2. Is $\mathbb{E}(Y|X) = \beta_0 + \beta_1 X$ equivalent to $Y|X = \beta_0 + \beta_1 X + \epsilon$? I can see how taking the expectation would lead us from $Y|X$ to $\mathbb{E}(Y|X)$ but I'm not sure if I can see how we go the other way around. Thank you! $\endgroup$ – user523384 Sep 21 '19 at 7:01
  • $\begingroup$ And one more, 3. Thanks for explaining the residuals vs error difference. To confirm, when estimating our parameters, are we really trying to minimise the residuals, in a mean square sense? Rather than the errors? I guess I'm having trouble grasping the idea behind "minimising a random variable" if we do indeed mean minimising $\epsilon$ $\endgroup$ – user523384 Sep 21 '19 at 7:36
  • 1
    $\begingroup$ @user523384 Please see my "Responses". $\endgroup$ – Dave Sep 21 '19 at 15:26
  • 1
    $\begingroup$ @user523384 1) It’s more like shorthand. 2) Expected value is precisely what we aim to examine. Yes, that’s a model. Ordinary least squares that you’re learning at the beginning of your regression class also makes the assumption of normality with constant variance, though you’re still estimating the conditional expectation. $\endgroup$ – Dave Sep 22 '19 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.