Provide a precise and concise statement on what a simple linear model is I have recently commenced a 2nd-year course on linear models and have been a little overwhelmed by either the abuse of notation or the lack of clarity behind what a linear model is. I've read multiple related MSE Q's, websites, gone to many books, seen the lecture notes but I still don't feel like I've understood precisely "what is what" in the model. I've summarised below some doubts I've come across, 


*

*What exactly are we trying to approximate/investigate? Is it a relationship between random variables $X$ and $Y$? Is it between $x_i$ and $y_i$ for a set of data points $\{(x_i,y_i): i=1,..,n \}$ that we have? Are $x_i$ and $y_i$ realisations of the random variables $X$ and $Y$? Does this mean we are trying to infer a relationship between the random variables $X$ and $Y$ based on the relationship we see between a sample of their realisations $(x_i,y_i)$?

*What is a precise statement of our model equation? I've seen a few equations, and i'm not sure whether one follows from the other, or if they're two equivalent statements of the same concept?:
For example: $Y = \beta_0 + \beta_1 X + \epsilon$, I've also seen $\mathbb{E}(y) = \beta_0 + \beta_1 x$ (which I'm unclear what is random or not, cause they are lowercase). I've also seen a model dealing with data points and not random variables, for example $y_i = \beta_0 + \beta_1 x_i + \epsilon$. I've also seen $\mathbb{E}(Y|X) = \beta_0 + \beta_1 X$.

*What variables exactly are random, and what variables are "realisations" of random variables, and what variables are actually random but we "treat" them as constants because the model is conditional?

*Is the error term $\epsilon$ random or is it fixed? I realise almost everywhere I've seen this to be "random", but if the model we are using is one with data points $y_i = \beta_0 + \beta_1 x_i + \epsilon$, then how can $\epsilon$ be random if $y_i$ is the $y$ value of a observation which should be fixed? For this equation, would it not be more correct to have the residual there rather than an "error" term?

*Is a residual a realisation of the error term $\epsilon$ for a specific data point? 

Request:
  if someone could provide a precise description of a linear model,
  specifically referencing the above doubts, I would really really
  appreciate it. Please be specific about whether a variable is random
  or not etc.

Apologies for including multiple questions in one, but I thought it would be nice for an overall synopsis of what's going on here.
I realise that some parts of the above questions might be in other MSE asked questions, I have read almost all of them, but I don't yet feel I've understood it yet. If you wish to copy-paste and reference other articles, that's completely fine, but I would please prefer a holistic answer rather than references to multiple articles
Thank you so much
 A: Original Question.


*

*You never estimate something about your data. You estimate something about the populations from which your data are drawn. Therefore, what you want to explore is the relationship between $X$ and $Y$. You do this by examining your observed values of $X$ and $Y$. $x_i$ and $y_i$ are realizations of $X$ and $Y$ that we use to infer something about the relationship between $X$ and $Y$.

*Let's unpack what each of those four equations says.
$$Y = \beta_0 + \beta_1 X + \epsilon$$

"The random variable $Y$ is equal to some number plus a multiple of
  the $X$ variable, plus some random error term." 
This describes the phenemonon at the population level, even if the
  left side would be more technically correct if it said $Y\vert X$.
  $\epsilon$ gives the shape of the distribution, while $\beta_0 +
 \beta_1X$ slides the distribution along the real line.
(I'm now seeing this way of writing it as flawed. We look at a
  particular value of $X$. The overall distribution of $X$ is not so
  important when we predict $Y$ when $X = 2$.)

$$\mathbb{E}(y) = \beta_0 + \beta_1 x$$

"The expected value of a vector $y$ of observations of $Y$ is equal to
  some number plus a multiple of of vector $x$ of observations of $X$."
It may help you to write this with vector arrows on top:
  $\mathbb{E}(\vec{y}) = \beta_0 + \beta_1 \vec{x}$.

$$y_i = \beta_0 + \beta_1 x_i + \epsilon$$

"The i$^{th}$ observation of $Y$ is equal to some number plus a
  multiple of the i$^{th}$ observation of $X$, plus some random error
  term." (That random error term ought to be $\epsilon_i$ as the
  i$^{th}$ observation of that random error term.)
This is just the observation-by-observation version of the previous
  equation.

$$\mathbb{E}(Y|X) = \beta_0 + \beta_1 X$$

"The expected value of Y, given some other information ($X$) that we
  know, is equal to some number plus a multiple of the value of that
  other information." 
We get the $\epsilon$ to go away in the equations with expectations
  because we assume that $\mathbb{E}\big[\epsilon \big] = 0$. (This is a
  standard assumption.)



*The data are observations and are not random. They are whatever you happen to observe. The populations are the random variables. I don't think we treat any random variables as constants. (Did you mean the $\beta_0$ term?)

*"Error" and "residual" do NOT have the same meaning. In the population, there is an error term, which is a random variable. Your data give you residuals, which you use to estimate something about the population distribution of the error.

*Yes! It's common to speak of errors when we mean residuals. Perhaps "mean squared error" should be "mean squared residual". But we're used to writing MSE.

The gist of regression is that we have too much variability in $Y$,
  but we know something else ($X$) about that distribution, so we look
  at the conditional distributions $Y\vert X$. However, there is a whole
  distribution, not just the point estimate that our regression model
  gives (which is the estimated expected value of that conditional
  distribution).
It takes a while to wrap the brain on the idea of having a whole
  conditional distribution when you're used to getting a single number
  as your prediction. The idea is that some distribution is sliding up
  and down the regression line. When $X$ has some value $x_0$, draw a
  value of the distribution $Y\vert X=x_0$. When $X$ has some value
  $x_1$, draw a value of the distribution $Y\vert X=x_1$. But there is
  some distribution sliding up and down the regression line.

I suspect you will have questions. We'll deal with them in the comments or maybe in chat.
Responses
(The first response alone was too long for a comment, and I want these in the main answer, too, not buried in the comments or in a chat.)


*

*$Y\vert X=x = \beta_0 + \beta_1 x + \epsilon$ is the most correct way to write it. The full distribution of $X$ isn't so important. (In particular, the Gauss-Markov theorem that you either know or will learn as you study regression more makes no assumptions about the distribution of $X$, even if it appears that way when you first see it.) We don't care what the distribution of $X$ is, only that we want to know about $Y$ when we have some other information. For example, if you want to know about human heights, it would make sense not to compare the heights of adults and children. We don't really care about the distributions of ages, only that $\mathbb{E}\big[\text{Height} \vert \text{adult} \big] \ne \mathbb{E}\big[\text{Height} \vert \text{child} \big] $.

*I wouldn't say that they're equivalent, but the math to get from one to the other is easy. We assume $\mathbb{E}\big[ \epsilon \big] = 0$. Then the expectation is an integral and therefore linear. Let's assume that $ Y \vert X$ is shorthand for $ Y \vert X=x$ for an interesting value of $X$ (say an adult subject rather than a child).
$$ Y \vert X = \beta_0 + \beta_1 x + \epsilon$$


*

*(The $x$ is intentionally lowercase, since we care about a
particular value of $X$ and not the entire distribution.)
$$ \mathbb{E} \big[Y \vert X\big] = \mathbb{E} \big[\beta_0 + \beta_1 x + \epsilon \big]$$
$$ \mathbb{E} \big[Y \vert X\big] = \mathbb{E} \big[\beta_0 \big]+ \mathbb{E} \big[\beta_1 x \big]+ \mathbb{E} \big[\epsilon \big]$$
$$ \mathbb{E} \big[Y \vert X\big] = \beta_0 + \beta_1 x + 0$$


*Yes, you want to minimize something about the residuals. $y_i - \hat{y}_i$ is a residual. For many regressions, you minimize $\sum (y_i - \hat{y}_i)^2$, though there are alternatives if you want to get fancy (such as the sum of absolute values instead of squares).

