# Foundations behind Linear Regression / Statistical Modelling

I've always struggled with the foundations behind the concept of modelling (and specifically regression) - what is random, what is not, what we are modelling.

I think I have a grasp of it - but I'd love if someone could please confirm if this matches with what they understand as well. Otherwise, if they have anything to add or correct - I would really appreciate this!

• We have a random variable $$Y$$, for example the weather, that we want to understand. However, it is a little too variable if we know absolutely nothing else. If someone says "What's the weather like" with no other context about when/where etc, it's really hard to say anything so far.

• However, to shrink this problem, and to perhaps better understand $$Y$$ relative to some other variables that are easier to observe (if assuming random) or control (if assuming non-random controlled) some other variables $$\mathbf{X}\in \mathbb R^p$$. For example, we might have predictors as the location and the month. This information would help us understand the season which is now something we can talk about - for example if it's January in Australia - you can start to imagine it the weather would probably be hot and sunny.

This has a few benefits/aims:

1. The variance of $$Y$$ given this new information $$\mathbf{X}$$ is significantly reduced. Before having infinite possibilities for the weather all with pretty even chance, now that we know something, we can start visualising what the nature (i.e. the distribution) of $$Y$$ might be like given $$\mathbf{X}$$.

2. We can understand the relationship between $$Y$$ and other variables $$\mathbf{X}$$. The relationship with not be deterministic because $$Y$$ is random (intuitively, there are an uncountably infinite number of factors that come together to determine what $$Y$$ will be), so we cannot deterministically know what $$Y$$ might be just based on a finite (or even countably infinite) number of predictors $$\mathbf{X}$$. But depending how relevant $$\mathbf{X}$$ is to the data-generating process for $$Y$$, it might explain a good majority - leading to a visible trend when we observe data $$\{(x_i, y_i): i=1,2,...,n\}$$.

(I have used the terms "uncountably infinite" and "countably infinite" a little recklessly. They are not meant to be literally accurate - I don't have any sources for this. But this is how I intuitively understand what something purely random is in real life, and I wonder whether this analogy is suitable?)

1. By making assumptions on the nature of the part of $$Y$$ unexplained by $$\mathbf{X}$$ (called the random error term $$\epsilon$$ - being collective influence of all other factors part from $$\mathbf{X}$$ on $$Y$$), we can say even more. If we assume that $$\mathbb{E}(\epsilon)=0$$, then we can say that while there is no deterministic relationship between $$\mathbf{X}$$ and $$Y$$, there is a deterministic relationship between $$\mathbf{X}$$ and $$\mathbb{E}(Y|\mathbf{X})$$ - i.e. there is a deterministic relationship between $$\mathbf{X}$$ and the average value of $$Y$$. If we further assume a distribution for $$\epsilon$$ then we can formulate a probabilistic model (i.e. a model for the distribution) for $$Y$$. For example, in simple linear regression, we assume that $$\epsilon\sim \mathcal N(0,1)$$ which leads to $$Y\sim \mathcal N(\beta_0 + \beta_1 X, \sigma^2)$$.

After contemplating, I also think the following interpretation is not accurate (do you agree?)

• There is some true deterministic underlying relationship between $$Y$$ and $$\mathbf{X}$$ but our data is noisy (for example due to measurement errors etc) and doesn't let us see this.

I feel like this is inaccurate - it's not just the data that's noisy, but the relationship itself between $$Y$$ and $$\mathbf{X}$$ is noisy. This is because $$\mathbf{X}$$ does not completely determine $$Y$$ (for if it did, that would mean we could observe $$Y$$ when we have $$\mathbf{X}$$, and prediction would not be necessary). We assume that the collective influence of uncollected information $$\epsilon$$ - all the other factors unobserved will symmetrically fault this relationship above and below. (however this assumption that $$\mathbb E (\epsilon)=0$$ is fairly arbitrary though.)

Thanks in advance, I would love to hear your suggestions/modifications/corrections and any parts you feel are accurate :)

• @SextusEmpiricus Yes, I agree! Do you agree with the explanation overall? Or do you have a different view? Jun 17 '20 at 8:12
• @SextusEmpiricus I realised, however, that I don't quite understand what's going on when the predictors are fixed/set. How are we supposed to condition $Y$ knowing a scalar/constant? Would you have any idea? Jun 17 '20 at 8:14
• Ah yes. This would also be possible for sure. I suppose I was considering the scenario where we don't know whether there is a deterministic relationship in the first place - and it seems most probably not. Usually I suspect if the response is a fairly complex phenomenon - even like Sales in a business, one would not expect the only deciding factor would be money spent in TV advertising, despite how determining it seems to be Jun 17 '20 at 8:23
• Jun 17 '20 at 8:26
• I have placed my comments in an answer. Jun 17 '20 at 8:36

### In short

Indeed, regression lines and correlations do not necessarily correspond one-to-one to a causal relationship

### Platonic/causal/deterministic model

After contemplating, I also think the following interpretation is not accurate (do you agree?)

• There is some true deterministic underlying relationship between $$Y$$ and $$\mathbf{X}$$ but our data is noisy (for example due to measurement errors etc) and doesn't let us see this.

That interpretation relates to a platonic idea where there is a true (causal) relationship between $$\mathbf{X}$$ and $$Y$$. But those X and Y are not our observations.

This may be the case in many physical experiments or some other experiments with a good foundation in the underlying mechanistic model such that we can think about the 'real' world behaving deterministicly according to our model, but our observations of the real world are imperfect so we need a statistical model to relate our observations.

The model like $$y_i = \alpha + \beta x_i + \epsilon_i$$ is a model that describes how the data is generated, the data points $$y_i$$ are considered as caused by the deterministic part $$\alpha + \beta x_i$$ and in addition with some non-deterministic part $$\epsilon_i$$ (measurement error or variations from measurement to measurement that are yet unexplained).

(In order to do regression you do not need to have that causal/idealistic underlying interpretation. Later we will see that it might be even wrong)

This interpretation works when the deterministic model is considered good enough such that the only statistical variations are variations related to measurements. With weather models this is not the case. There will be broader variations from experiment to experiment which is due to bias in the model (the weather models are far from a platonic ideal) and not just due to variations from measurement to measurement.

### Contrast with regression

The concept of regression that you describe seems to relate to regression as an empirical model but not as a mechanistic model. Sure, we can relate some variable $$Y$$ with some regressors $$\mathbf{X}$$ with some linear or polynomial function or with some other curve. But in this case we are predicting $$E(Y\vert \mathbf{X})$$ with some approximate model, it is not (necessarily) relating to a deterministic/idealistic/platonic/causal model. The fitted model is just describing some 'trend' or correlation between measurements $$Y$$ and $$\mathbf{X}$$ and does not relate to any realistic/mechanistic underlying 'true' relationship (it may be even wrong to consider the regression fit as a causal relationship).

You are right that regression is not exactly like that platonic underlying deterministic relationship. For instance, when we switch the role of $$X$$ and $$Y$$ then we get a different result. If there would be some true causal relationship then one might intuitively expect to get the same result no matter what direction you perform the regression. You do not get this because regression does not determine the deterministic underlying (causal) relationship but it is finding the relationship to predict $$Y$$ based on $$X$$.

### When regression relates to the platonic model

However, in some cases of well controlled experiments we can consider that there is a true causal relationship between $$Y$$ and $$\mathbf{X}$$ and that the only/most discrepancy is due to measurement errors in $$Y$$. In that case the regression model will coincide with the causal model (but it will fall apart when there are measurement errors in $$X$$ as well)

• This is personally such a clarifying, enlightening answer. I have been struggling with understanding this distinction for quite some time. Thank you for this! It makes so much how the DGP makes a casual statement but regression just makes a statement of association Jun 17 '20 at 11:38
• Could you please clarify something for me? If, in our formulation, the regressors are considered deterministic, is it possible to conduct regression? For how could a random variable be correlated with a "constant" predictor $x_i$? Is regression only possible with random $\mathbf{X}$? Jun 17 '20 at 11:40
• Relating to your last comment: With empirical regression you make no assumptions about the distribution of Y conditional on X (at least not in terms of a parametric distribution). So the causality is not the contrast, ie. the 'parametric regression' can be still non-causal (and actually the empirical regression can be causal but just with unknown error distribution). For example, it may be correct to assume that $Y\vert X$ is normal distributed, without there being a causal model like X causing Y. Jun 17 '20 at 12:07
• Relating to your before last comment: Correlation is more used to describe how a parameter Y varies as we vary parameter X. This variation in X can be random but that is not necessary. In many experiments we control X (up to some small measurement error). That does not mean that X is constant. For instance, in your weather model $X$ could be the height at which we measure $Y$ and we might take measurements at different heights. Jun 17 '20 at 12:14
• It is indeed not like the height causes the weather. However you might still formulate it as causal, changing the height causes your measurements to change. Take for instance pressure as function of hight. You might consider the pressure in a column of water a very well described physical function, 'pressure is a function of height (plus some error)'. You can see it as 'the height is a parameter of the causal/physical/mechanistic model' (ie the model that describes the observations in terms of first principles, causal principles). Jun 17 '20 at 12:26

$$\newcommand{\E}{\operatorname{E}}$$Here's a more formal perspective which I think can be helpful for being clear about what's random and what isn't. We have a probability space $$(\Omega,\mathscr F, P)$$ and random variables $$X$$ and $$Y$$ with $$X: \Omega \to \mathbb R^{p}$$ and $$Y:\Omega\to\mathbb R$$.

You're asking about regression specifically so I'm going to focus on that, rather than different notions of modeling like selecting a measure from an indexed collection $$\{P_\theta : \theta\in\Theta\}$$. We want to come up with some function $$h$$ that "explains" $$Y$$ using $$X$$, thus we seek a $$(\mathbb B^p, \mathbb B)$$-measurable $$h : \mathbb R^p\to\mathbb R$$ such that $$h\circ X$$ is "close" to $$Y$$. It can be shown that $$h\circ X$$ is $$(\sigma(X),\mathbb B)$$-measurable.

This addresses part of your question (1): by going from $$Y$$ to $$h\circ X$$, we have changed from being $$(\mathscr F, \mathbb B)$$-measurable to $$(\sigma(X),\mathbb B)$$-measurable. It is always the case that $$\sigma(X)\subseteq\mathscr F$$ but if $$X$$ is not very complex then this can provide a great simplification. I think this is a more precise way to look at the "information" here.

In order to actually produce such an $$h$$ we'll need some way to measure its performance. We can appeal to decision theory and do this via a loss function $$L(Y, f(X))$$, and since this is a random variable we'll actually use the risk functional $$R[h] = \E[L(Y, h(X))].$$

In practice we'd never want to minimize this over all $$(\mathbb B^p, \mathbb B)$$-measurable functions as those functions can be quite complicated and that would be a hopeless business (we also would have many functions with identical values on the training set and we wouldn't be guaranteed to have our empirical risk minimizer converge on the true minimizer). Instead we'll want to restrict our attention to some nicer function space $$\mathcal F$$ and then pick $$\hat h$$ from there. Our choice of $$\mathcal F$$ is a modeling decision. For example, we could fix some basis functions $$h_1,\dots,h_m$$ and take $$\mathcal F = \text{span}\{h_1,\dots,h_m\}$$ so we're considering functions of the form $$x\mapsto \sum_{i=1}^m \beta_ih_i(x).$$ In this case we can reasonably select a $$\hat h$$ based on a finite sample and then we're modeling $$Y$$ as $$\hat h\circ X$$. This includes linear regression and fancier things like splines. If we allow the basis functions to also have parameters in them (i.e. be "adaptive") then we can view neural networks and many other models from this perspective too. Note that if the $$h_i$$ are nice enough (i.e. continuous) then if $$f,g\in \mathcal F$$ are equal almost surely they are in fact equal everywhere, so we don't need to deal with issues of functions being defined almost everywhere.

This also touches on how there are two approximations happening here: first we're restricting the true $$h$$ to be in $$\mathcal F$$, and then we're approximating it with $$\hat h$$, the one we actually found. If no element of $$\mathcal F$$ is actually a good fit then we'll have a large error in that step even if $$\hat h$$ is really the best element in $$\mathcal F$$.

Assuming $$Y$$ is integrable, it can be shown that $$\E(Y|X)$$ is the a.s.-unique minimizer of $$\E((Y-Z)^2)$$ over $$(\sigma(X), \mathbb B)$$-measurable $$Z$$. It also can be shown that there is a Borel $$h$$ such that $$\E(Y|X) = h\circ X$$; we can use this result to define $$\E(Y|X=x) = h(x)$$ which means we don't need to refer to $$\Omega$$ (see e.g. Lemma 1.2 in section 1.4.1 of Jun Shao's Mathematical Statistics for more on this). Thus if we choose to use squared loss, the actual minimizer is the conditional expectation. And since $$\E(Y|X=x) = h(x)$$, when we restrict $$h$$ to being in our friendly $$\mathcal F$$ we're actually directly modeling $$\E(Y|X=x)$$ as belonging to this space.

A lot of this has been from a machine learning perspective since I think that kind of signal modeling is intuitive. But if we want to think of making distributional assumptions, saying $$Y = h\circ X$$ induces the distribution of $$Y$$ based on $$X$$ so we could approach things from that way.

Making distributional assumptions on $$\varepsilon$$ is not necessary for doing this. We can always run our algorithm and get a model. It's more that without understanding the error we won't have a sense of when our procedure is doing well or not.

• I am very grateful for your answer. For some reason, I failed to find mathematical rigour in the foundations of regression/modelling in any of the texts I've seen. They always jump straight to $Y=\beta_0 + \beta_1 x + \epsilon$ but I have a good amount of trouble understanding what's going on Jun 16 '20 at 1:55
• I have a few questions to follow up: 1. When you write $\text{E}(Y|X)(x) = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3$, doesn't $x\in \mathbb{R}$?. But I thought $\text{E}(Y|X):\Omega \to \mathbb{R}$? (and $\Omega$ is not necessarily $\mathbb{R}$ right?) Jun 16 '20 at 1:56
• 2. Are we trying to model the relationship between $Y$ and $X$, or the average value of $Y$ given $X$? I thought the goal was to understand the relationship between the actual temperature $Y$, rather than the average temperature? But knowing the relationship between expected value of $Y|X$ and $X$ would not be useful unless we had some bounded (and hopefully quite small) variance for $Y|X$ as well right? Jun 16 '20 at 2:00
• 3. and if $\text{E}(Y|X)$ is also a random variable, why is it that we try to model $\text{E}(Y|X)$ belonging to some family of functions $\mathcal{F}$ instead of assuming $Y|(X=x)$, $Y$ or $(X,Y)$ belonging to some family of functions? Jun 16 '20 at 3:08
• @user523384 for (1) good point re: $\text{E}(Y\mid X)$ being a function on $\Omega$. The formal definition of $\text{E}(Y|X=x)$ uses an additional result that $\text{E}(Y|X)$ can be written as $h\circ X$ for a measurable $h : \mathbb R^p\to\mathbb R$ so this lets us define $\text{E}(Y|X=x) = h(x)$ and then the inputs to $\text E(Y|X)$ are outputs of $X$. I'll add this to the answer in the morning and I'll also address (2) and (3).
– jld
Jun 16 '20 at 5:22

This is an interesting perspective, but perhaps you are making regression to be a bit more complex than it needs to be. Let's focus on your weather example.

Suppose I'm a meteorologist. I want to predict the weather (more specifically, let's say temperature). Based on the research I have done, I think that a reasonable model of temperature is:

\begin{align} y &= \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \varepsilon \\ &= \beta^\top \mathbf{x} + \varepsilon \end{align}

where $$y$$ is temperature, $$x_1$$ is latitude, $$x_2$$ is longitude, $$x_3$$ is time of year, and $$\varepsilon \sim \mathcal{N}(0,\sigma^2)$$ is a zero-mean Gaussian noise term that captures for all of the other variations I am not specifically accounting for.

Then, consequently, $$y \sim \mathcal{N}(\beta^\top\mathbf{x},\sigma^2)$$. That is: $$\mathbb{E}[Y\mid X=\mathbf{x}] = \beta^\top\mathbf{x}.$$

I don't think I've said anything so far you haven't already stated, but I wanted to lay out the process because it may be helpful.

Specifically, I (as an expert meteorologist in this scenario) proposed a probabilistic model which I thought was reasonable based the research I've done and expertise I have. The specific probabilistic model has a linear relationship (i.e., $$\beta \mathbf{x}$$) with a "wiggle-room" term $$\varepsilon$$ which captures what I don't want to/cannot model explicitly. This probabilistic model has the interpretation that the expected value given the data contained in $$\mathbf{x}$$ is equal to $$\beta \mathbf{x}$$.

Importantly, this expected value happens to minimize the mean squared error between the prediction and the true value (given the stated probabilistic model). Consequently, once I have determined the $$\beta$$ coefficients (for example, based on some historical data), I can use them along with a new $$\mathbf{x}^*$$ test point and predict the corresponding temperature $$y^*$$ in a principled way.

Whether or not the model I have proposed is a good approximation to the true model is a completely different (and very important) question.

I just want to outline the process of creating a regression model as several steps:

1. Propose a probabilistic model;
2. Do inference (e.g., compute the $$\beta$$ coefficients in this case);
3. Use the model for future predictions.

I wanted to bring up this applied way of thinking about regression (which is theoretically grounded) because perhaps it emphasizes the most important parts of what regression is about.

Hope this (at least partially) helps.

• Hi @jcreinhold I really appreciate your answer, thanks for this! And welcome to stack exchange :) I definitely agree, what you have mentioned are the more practical and important elements of regression which are used. However I think what I'm looking for is more so the story and journey that brings us to the model $y =\beta^T\mathbf{x} + \epsilon$. There are many questions that come to my mind when I see this: - Jun 16 '20 at 2:13
• - is this equality an assumption or a true equality? (if we don't assume a distribution or mean for $\epsilon$) - why is there noise between $Y$ and $X$, and what is the "true underlying relationship" that we are after when modelling? (For example, we have the "true regression line" as a target but I don't not understand how "true" this true regression line is. Is it true given our assumption, or is it the real life relationship between $Y$ and $X$) Jun 16 '20 at 2:14
• Hi @user523384. The story I laid out was an attempt to show where the linear model came from; that is, a scientist/researcher/statistician or whoever—let's call them a modeler—_assumes_ a model like a linear relationship between $\mathbf{x}$ and $y$. The modeler reasonably assumes also that there are other factors which may affect the exact value of $y$ given the data $x$. However, the modeler makes the naive assumption that all of these other unaccounted for factors cancel each other out (on average, hence the zero mean) and, due to the CLT, the distribution of this term is Gaussian. Jun 16 '20 at 14:27
• The linear relationship is very much an assumption of the modeler. A modeler can assume a more complicated relationships—e.g., a polynomial relationship—and a modeler could also assume a non-Gaussian distribution for $\varepsilon$ (see GLMs). But any model about the real world is a (flawed) simplification; there is a famous saying by George Box that "all models are wrong, some models are useful." So to directly answer your questions: No, in real world data, the model $y=\beta^\top \mathbf{x}+\varepsilon$ is not a true equality. Jun 16 '20 at 14:36