Foundations behind Linear Regression / Statistical Modelling I've always struggled with the foundations behind the concept of modelling (and specifically regression) - what is random, what is not, what we are modelling.

I think I have a grasp of it - but I'd love if someone could please
confirm if this matches with what they understand as well. Otherwise,
if they have anything to add or correct - I would really appreciate
this!


*

*We have a random variable $Y$, for example the weather, that we want to understand. However, it is a little too variable if we know absolutely nothing else. If someone says "What's the weather like" with no other context about when/where etc, it's really hard to say anything so far.


*However, to shrink this problem, and to perhaps better understand $Y$ relative to some other variables that are easier to observe (if assuming random) or control (if assuming non-random controlled) some other variables $\mathbf{X}\in \mathbb R^p$. For example, we might have predictors as the location and the month. This information would help us understand the season which is now something we can talk about - for example if it's January in Australia - you can start to imagine it the weather would probably be hot and sunny.
This has a few benefits/aims:

*

*The variance of $Y$ given this new information $\mathbf{X}$ is significantly reduced. Before having infinite possibilities for the weather all with pretty even chance, now that we know something, we can start visualising what the nature (i.e. the distribution) of $Y$ might be like given $\mathbf{X}$.


*We can understand the relationship between $Y$ and other variables $\mathbf{X}$. The relationship with not be deterministic because $Y$ is random (intuitively, there are an uncountably infinite number of factors that come together to determine what $Y$ will be), so we cannot deterministically know what $Y$ might be just based on a finite (or even countably infinite) number of predictors $\mathbf{X}$. But depending how relevant $\mathbf{X}$ is to the data-generating process for $Y$, it might explain a good majority - leading to a visible trend when we observe data $\{(x_i, y_i): i=1,2,...,n\}$.
(I have used the terms "uncountably infinite" and "countably infinite" a little recklessly. They are not meant to be literally accurate - I don't have any sources for this. But this is how I intuitively understand what something purely random is in real life, and I wonder whether this analogy is suitable?)


*By making assumptions on the nature of the part of $Y$ unexplained by $\mathbf{X}$ (called the random error term $\epsilon$ - being collective influence of all other factors part from $\mathbf{X}$ on $Y$), we can say even more. If we assume that $\mathbb{E}(\epsilon)=0$, then we can say that while there is no deterministic relationship between $\mathbf{X}$ and $Y$, there is a deterministic relationship between $\mathbf{X}$ and $\mathbb{E}(Y|\mathbf{X})$ - i.e. there is a deterministic relationship between $\mathbf{X}$ and the average value of $Y$. If we further assume a distribution for $\epsilon$ then we can formulate a probabilistic model (i.e. a model for the distribution) for $Y$. For example, in simple linear regression, we assume that $\epsilon\sim \mathcal N(0,1)$ which leads to $Y\sim \mathcal N(\beta_0 + \beta_1 X, \sigma^2)$.

After contemplating, I also think the following interpretation is not accurate (do you agree?)

*

*There is some true deterministic underlying relationship between $Y$ and $\mathbf{X}$ but our data is noisy (for example due to measurement errors etc) and doesn't let us see this.

I feel like this is inaccurate - it's not just the data that's noisy, but the relationship itself between $Y$ and $\mathbf{X}$ is noisy. This is because $\mathbf{X}$ does not completely determine $Y$ (for if it did, that would mean we could observe $Y$ when we have $\mathbf{X}$, and prediction would not be necessary). We assume that the collective influence of uncollected information $\epsilon$ - all the other factors unobserved will symmetrically fault this relationship above and below. (however this assumption that $\mathbb E (\epsilon)=0$ is fairly arbitrary though.)
Thanks in advance, I would love to hear your suggestions/modifications/corrections and any parts you feel are accurate :)
 A: In short
Indeed, regression lines and correlations do not necessarily correspond one-to-one to a causal relationship
Platonic/causal/deterministic model

After contemplating, I also think the following interpretation is not accurate (do you agree?)

*

*There is some true deterministic underlying relationship between $Y$ and $\mathbf{X}$ but our data is noisy (for example due to measurement errors etc) and doesn't let us see this.


That interpretation relates to a platonic idea where there is a true (causal) relationship between $\mathbf{X}$ and $Y$. But those X and Y are not our observations.
This may be the case in many physical experiments or some other experiments with a good foundation in the underlying mechanistic model such that we can think about the 'real' world behaving deterministicly according to our model, but our observations of the real world are imperfect so we need a statistical model to relate our observations.
The model like $y_i = \alpha + \beta x_i + \epsilon_i$ is a model that describes how the data is generated, the data points $y_i$ are considered as caused by the deterministic part $\alpha + \beta x_i$ and in addition with some non-deterministic part $\epsilon_i$ (measurement error or variations from measurement to measurement that are yet unexplained).
(In order to do regression you do not need to have that causal/idealistic underlying interpretation. Later we will see that it might be even wrong)
This interpretation works when the deterministic model is considered good enough such that the only statistical variations are variations related to measurements. With weather models this is not the case. There will be broader variations from experiment to experiment which is due to bias in the model (the weather models are far from a platonic ideal) and not just due to variations from measurement to measurement.
Contrast with regression
The concept of regression that you describe seems to relate to regression as an empirical model but not as a mechanistic model. Sure, we can relate some variable $Y$ with some regressors $\mathbf{X}$ with some linear or polynomial function or with some other curve. But in this case we are predicting $E(Y\vert \mathbf{X})$ with some approximate model, it is not (necessarily) relating to a deterministic/idealistic/platonic/causal model. The fitted model is just describing some 'trend' or correlation between measurements $Y$ and $\mathbf{X}$ and does not relate to any realistic/mechanistic underlying 'true' relationship (it may be even wrong to consider the regression fit as a causal relationship).
You are right that regression is not exactly like that platonic underlying deterministic relationship. For instance, when we switch the role of $X$ and $Y$ then we get a different result. If there would be some true causal relationship then one might intuitively expect to get the same result no matter what direction you perform the regression. You do not get this because regression does not determine the deterministic underlying (causal) relationship but it is finding the relationship to predict $Y$ based on $X$.
When regression relates to the platonic model
However, in some cases of well controlled experiments we can consider that there is a true causal relationship between $Y$ and $\mathbf{X}$ and that the only/most discrepancy is due to measurement errors in $Y$. In that case the regression model will coincide with the causal model (but it will fall apart when there are measurement errors in $X$ as well)
A: $\newcommand{\E}{\operatorname{E}}$Here's a more formal perspective which I think can be helpful for being clear about what's random and what isn't. We have a probability space $(\Omega,\mathscr F, P)$ and random variables $X$ and $Y$ with $X: \Omega \to \mathbb R^{p}$ and $Y:\Omega\to\mathbb R$.
You're asking about regression specifically so I'm going to focus on that, rather than different notions of modeling like selecting a measure from an indexed collection $\{P_\theta : \theta\in\Theta\}$. We want to come up with some function $h$ that "explains" $Y$ using $X$, thus we seek a $(\mathbb B^p, \mathbb B)$-measurable $h : \mathbb R^p\to\mathbb R$ such that $h\circ X$ is "close" to $Y$. It can be shown that $h\circ X$ is $(\sigma(X),\mathbb B)$-measurable.
This addresses part of your question (1): by going from $Y$ to $h\circ X$, we have changed from being $(\mathscr F, \mathbb B)$-measurable to $(\sigma(X),\mathbb B)$-measurable. It is always the case that $\sigma(X)\subseteq\mathscr F$ but if $X$ is not very complex then this can provide a great simplification. I think this is a more precise way to look at the "information" here.
In order to actually produce such an $h$ we'll need some way to measure its performance. We can appeal to decision theory and do this via a loss function $L(Y, f(X))$, and since this is a random variable we'll actually use the risk functional
$$
R[h] = \E[L(Y, h(X))].
$$
In practice we'd never want to minimize this over all $(\mathbb B^p, \mathbb B)$-measurable functions as those functions can be quite complicated and that would be a hopeless business (we also would have many functions with identical values on the training set and we wouldn't be guaranteed to have our empirical risk minimizer converge on the true minimizer). Instead we'll want to restrict our attention to some nicer function space $\mathcal F$ and then pick $\hat h$ from there. Our choice of $\mathcal F$ is a modeling decision. For example, we could fix some basis functions $h_1,\dots,h_m$ and take
$$
\mathcal F = \text{span}\{h_1,\dots,h_m\}
$$
so we're considering functions of the form
$$
x\mapsto \sum_{i=1}^m \beta_ih_i(x).
$$
In this case we can reasonably select a $\hat h$ based on a finite sample and then we're modeling $Y$ as $\hat h\circ X$. This includes linear regression and fancier things like splines. If we allow the basis functions to also have parameters in them (i.e. be "adaptive") then we can view neural networks and many other models from this perspective too. Note that if the $h_i$ are nice enough (i.e. continuous) then if $f,g\in \mathcal F$ are equal almost surely they are in fact equal everywhere, so we don't need to deal with issues of functions being defined almost everywhere.
This also touches on how there are two approximations happening here: first we're restricting the true $h$ to be in $\mathcal F$, and then we're approximating it with $\hat h$, the one we actually found. If no element of $\mathcal F$ is actually a good fit then we'll have a large error in that step even if $\hat h$ is really the best element in $\mathcal F$.

Assuming $Y$ is integrable, it can be shown that $\E(Y|X)$ is the a.s.-unique minimizer of $\E((Y-Z)^2)$ over $(\sigma(X), \mathbb B)$-measurable $Z$. It also can be shown that there is a Borel $h$ such that $\E(Y|X) = h\circ X$; we can use this result to define $\E(Y|X=x) = h(x)$ which means we don't need to refer to $\Omega$ (see e.g. Lemma 1.2 in section 1.4.1 of Jun Shao's Mathematical Statistics for more on this). Thus if we choose to use squared loss, the actual minimizer is the conditional expectation. And since $\E(Y|X=x) = h(x)$, when we restrict $h$ to being in our friendly $\mathcal F$ we're actually directly modeling $\E(Y|X=x)$ as belonging to this space.
A lot of this has been from a machine learning perspective since I think that kind of signal modeling is intuitive. But if we want to think of making distributional assumptions, saying $Y = h\circ X$ induces the distribution of $Y$ based on $X$ so we could approach things from that way.
Making distributional assumptions on $\varepsilon$ is not necessary for doing this. We can always run our algorithm and get a model. It's more that without understanding the error we won't have a sense of when our procedure is doing well or not.
A: This is an interesting perspective, but perhaps you are making regression to be a bit more complex than it needs to be. Let's focus on your weather example.
Suppose I'm a meteorologist. I want to predict the weather (more specifically, let's say temperature). Based on the research I have done, I think that a reasonable model of temperature is:
\begin{align}
y &= \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \varepsilon \\
  &= \beta^\top \mathbf{x} + \varepsilon
\end{align}
where $y$ is temperature, $x_1$ is latitude, $x_2$ is longitude, $x_3$ is time of year, and $\varepsilon \sim \mathcal{N}(0,\sigma^2)$ is a zero-mean Gaussian noise term that captures for all of the other variations I am not specifically accounting for.
Then, consequently, $y \sim \mathcal{N}(\beta^\top\mathbf{x},\sigma^2)$. That is:
$$
\mathbb{E}[Y\mid X=\mathbf{x}] = \beta^\top\mathbf{x}.
$$
I don't think I've said anything so far you haven't already stated, but I wanted to lay out the process because it may be helpful.
Specifically, I (as an expert meteorologist in this scenario) proposed a probabilistic model which I thought was reasonable based the research I've done and expertise I have. The specific probabilistic model has a linear relationship (i.e., $\beta \mathbf{x}$) with a "wiggle-room" term $\varepsilon$ which captures what I don't want to/cannot model explicitly. This probabilistic model has the interpretation that the expected value given the data contained in $\mathbf{x}$ is equal to $\beta \mathbf{x}$.
Importantly, this expected value happens to minimize the mean squared error between the prediction and the true value (given the stated probabilistic model). Consequently, once I have determined the $\beta$ coefficients (for example, based on some historical data), I can use them along with a new $\mathbf{x}^*$ test point and predict the corresponding temperature $y^*$ in a principled way.
Whether or not the model I have proposed is a good approximation to the true model is a completely different (and very important) question.
I just want to outline the process of creating a regression model as several steps:

*

*Propose a probabilistic model;

*Do inference (e.g., compute the $\beta$ coefficients in this case);

*Use the model for future predictions.

I wanted to bring up this applied way of thinking about regression (which is theoretically grounded) because perhaps it emphasizes the most important parts of what regression is about.
Hope this (at least partially) helps.
