# From a theoretical standpoint, why is linear regression useful? [closed]

I am not asking about how linear regression is derived. I am asking why oftentimes linear regression is the most useful to predict data. I understand that there are different types of regression, but what conditions must data satisfy in order for it to be represented by a line? There is always a hypothetical line that can go through a set of points with infinite accuracy but it is not necessarily a good predictor, why? In essence, why is precision counterproductive? And what specific mathematics must I learn about to answer this question? All the resources I have been exposed to merely go into the derivation of the line of best fit. I apologize if my thoughts are messy, but even a guess of where to look is much appreciated.

• There is always a hypothetical line that can go through a set of points with infinite accuracy but it is not necessarily a good predictor There is? And if the hypothetical line has infinite accuracy, how it is not a good predictor?
– Dave
Commented Nov 16, 2023 at 20:33
• To give an oversimplified answers: it's both useful and harmful in the same sense any linear approximation to a nonlinear function strikes the delicate balance between tractability and oversimplification. Linear regression is saying $E(y|x)$ is (at least roughly) linear. Commented Nov 16, 2023 at 20:43
• Any question about "useful to predict data" is about applications, not theory, making it difficult to understand what you might mean by "theoretical standpoint."
– whuber
Commented Nov 16, 2023 at 20:51
• Is it possible that you are assuming that the word "line" in this context means ANY kind of line (for example an arbitrarily curvy or squiggly line?)? In a linear regression context "line" means a STRAIGHT line. There is always a hypothetical (arbitrarily) curvy line that can go thought points with infinite accuracy but that's obviously not true of the straight line that we are talking about in linear regression Commented Nov 16, 2023 at 20:51
• It seems to me that this question is largely about noise and overfitting? I.e., you can fit a complicated squiggly line to your data points, but it will not work well on future observations because you are overfitting to the noise in your training data. Noise on future data points will be different. You want a curve that fits all the systematic variations in your data but none of the random variations. Linear regression misses some of the systematic variation due to its simplicity, but it definitely does not fit the noise. That yields high robustness. Commented Nov 16, 2023 at 20:58

It seems that Robert has already addressed the assumptions required in regression and some of the theoretical underpinnings of regression. Along with Robert's useful answer, I would like to highlight some additional reasons, which are similarly related to each other:

Regression predicts rather than describes. The whole reason regression was invented was to understand sweet pea genetics. If Galton could predict with some reliability how the plants would grow, he could determine what would be useful and what would be wasteful. The reason this method became useful was due to it's ability to model in variation. A naive estimate of the predicted value of an outcome is generally considered the mean (but could be other values). A regression makes this idea conditional upon variation based off different inputs. This was one of the beginning problems that Galton faced, which was taking rather uninformative descriptive data like the mean and transforming it into a modeling tool which makes the mean conditional (see plots of the mean and conditional mean regression below).

If we consider this a prediction problem, we can see how the first plot is bad and the second plot is at least in some part better. Let's say we are trying to predict $$y$$ here. If we just guess that it will be the mean of $$y$$, we will obviously miss by a lot, but it won't be totally wrong...any given $$y$$ here should hover around the mean. However, if we make that conditional on $$x$$, we can make better guesses. For example, we can probably assume that $$x=1$$ should yield an outcome of $$y=2$$. This leads into my next point.

Fitting is not the same as overfitting. I gather from the question that part of your issue is with overfitting, where any naive line is fit to data without consideration of the data generating process (DGP). Lets say we have an archer who is shooting arrows at a target:

Visually inspecting this relationship, we should predict to a degree that the relationship has a downward-facing parabolic function...the arrow should go up in height, then go down in height. For simplicity, we pretend the height starts and ends at zero feet from the ground, and hits a maximum around 7 feet:

We could just fit any line here:

But you can see both of these are poor approximations of the DGP (I made a more extreme version here than my original answer for a better illustration). The first linear fit would make for terrible predictions. We would expect the arrow to shoot downward and never stop. The other is overfit and makes similarly bad predictions (its a 20th order polynomial which interpolates badly), so where we predict the arrow is all over the place. However, if we simply understand that this is a downward parabola, we can fit a much simpler and reasonable relationship that makes for better predictions:

Our assumptions allow these predictions to be better. Part of your question was about which conditions allow for a regression to function well. Robert already noted some, but I'll highlight that linearity is an important one which you can see from my example (note that we wish for linearity in the parameters, even with nonlinear data). Another example is heterogeneity of variance. Here you can see from the Engel data from Koenker & Hallock, 2001 that the dispersion of data points is rather unequal. The data first clusters to the bottom and then fans out across the scatterplot. Once again, we fit a linear regression:

Because our data doesn't meet the heterogeneity of variance assumption, predictions are once again potentially poor. We can guess with considerable certainty what low values of income should give us in terms of food expenditure, but that becomes a lot more uncertain as we move to the top right. Thus our predictions are only as good as our regressions meet the minimum standards for fitting.

#### Edit

I just saw that you also asked about what math is necessary to learn the intuition behind regression. A lot can be learned from basic algebra (such as what a "function" is), calculus (how things like the slope work for linear and nonlinear functions), linear algebra (learning things like how the coefficients are constructed), probability (understanding hypothesis testing in regression) and of course statistics. At a higher level, you may need to take classes that combine all this math in something like a linear models class.

However, much of this can also be simply understood by experience with data and simulation. In fact, much of what I know now about regression is a combination of math I have learned and simulation based around that math.

• (+1) But that's not a parabola - some kind of bell curve. Commented Nov 17, 2023 at 0:14
• Yeah I originally modeled a different curve but was too lazy to change it back and felt it looked similar enough to demonstrate the main goal here. I went back and fixed it, with some sparse data at the ends to make the 20th order polynomial more extreme than I had before. Commented Nov 17, 2023 at 0:29

From a theoretical standpoint, why is linear regression useful?

I'm not sure what a theoretical standpoint would be since linear regression is about fitting a model to data, which is very much a practical thing. Anyway here are a few reasons why I think it is useful:

1. Quite often we want to model a real-world data-generating process, either to investigate possible causal relations, or to make predictions. It turns out that many such processes are well approximated by a straight line, at least over a relatively small part of the domain (note that extrapolation can be very dangerous)

2. Even when there are curvilinear associations, these can often be accommodated in a linear regression model by using higher order terms or splines - the model is still linear in the parameters.

3. Ease of interpretation. Compared to many other types of models, the results from linear regression are fairly easy to understand for non-technical people. This is very useful in business when explaining

4. The linear regression model can be extended to generalised linear models, linear mixed effects models, ridge and Lasso regression, quantile regression, robust regression, among others.

5. One-way ANOVA, two-way ANOVA (and as many ways as you like) as well as ANCOVA are just special cases of linear regression.

what conditions must data satisfy in order for it to be represented by a line?

I won't go into this, since it has been discussed on this site many times. Here are a couple of threads:

What is a complete list of the usual assumptions for linear regression?

Understanding the assumptions of Linear Regression

• Your fourth point is an underrated one. I believe a large reason that t-tests and ANOVAs are given poor interpretations by psychologists is because they are not taught that they have equivalence to categorical regressions. On multiple occasions I've been told by a researcher that ANCOVA was used to control for covariates solely because they "couldn't control for the covariate in a regression." Commented Nov 16, 2023 at 23:05
• @ShawnHemelstrand thanks for that, I have updated this answer :) Commented Nov 17, 2023 at 9:04

I am going to answer this question, "I understand that there are different types of regression, but what conditions must data satisfy in order for it to be represented by a line?".

The field of statistics is (largely) about making predictions from data. It is inherently an applied subject. In the "real world", what matters more than anything else are how accurate your predictions are. Sometimes we use prediction methods that are mathematically flawed, but we use them anyway because they are easier to work. In the end what really matters is how accurate it is.

There is a classification techinique called "Naive Bayes Classifier". It is called "naive", because you naively assume that all the predictor variables are independent. This is usually not true. But you just boldly assume it anyway. And guess what, despite the abuse of the math, it still can be a good classifier.

Your question sounds like you are asking what is the mathematical basis for doing linear regression? The answer is, you often do not need to justify your method. You just try it and see how well it works.

By the way, there are other types of regression methods, not just least squares. Some of them might work better than others. In the end it is just real-life experimentation about what prediction method works better for a specific problem.

The math behind it might describe some perfect ideal situation, but in the real-world those assumptions are never valid, and a statistician usually does not care. The statistician will just try something and see how well the prediction works.

• Theirs is an element of truth in this answer. However, I fear that it will be easy for a beginner to read it and determine that the theory doesn’t matter if they can just torture the data into confessing.
– Dave
Commented Nov 17, 2023 at 4:17
• @Dave I find that theory is more valuable in coming up with a prediction-method than in actually justifying it. People who have a solid grasp of probability theory (especially multivariable distributions) are much more comfortable with trying alternative methods that will work better. The theory expands what we can try out. But the justification, in the real world, will come down not to theory but in how accurate the predictions are. Commented Nov 17, 2023 at 5:44
• I was hoping there would be some sort of well defined connection between n-degree functions, and the number of factors some arbitrarily large number has. In essence, a linear relationship is more likely to exist than an exponential or hyperbolic. A conjecture that seems intuitively correct but I am too inexperienced to define this structure. I am under the impression that this question is related to the precise complexities that influence the direction of data. No idea how to begin asking such a question. Commented Nov 18, 2023 at 3:34

I would like to add a couple of remarks from a non-statistician point of view:

Many real-world relations can be approximated by a linear dependence
Mathematicaly it simply means that a differentiable function can be approximated by Taylor series: $$y=f(x) \approx f(0) + f'(0)x +...$$ and for many practical purposes this is good enough. In fact, in sciences such linear approximations even get to be called "laws", although they are certainly not fundamental laws. Just a few examples known from high school:

• Ohm's law (linearized current-voltage dependence)
• Hooke's law - linearized dependence between stress and strain
• Modification of electric field in a media $$\mathbf{E}\rightarrow \epsilon\mathbf{E}$$ or polarizability of molecules $$\mathbf{P}=\chi \mathbf{E}$$.

Here is, e.g., the actual dependence of stress on strain:
but the initial linear region is sufficient for many applications.

Linear regression is not necessarily a line
Term Linear regression is somewhat ambiguous. In some cases it implies the linear dependence between a response variable and a number of predictor variables, as $$y=\beta_0 + \sum_{j=1}^n\beta_j x_j.$$ Whoever, the predictor variables may be themselves functions of another variable. E.g., by choosing $$x_j=x^j$$ the above equation becomes fitting data with a polynomial of power $$n$$, which can pass through every point of the data (if $$n$$ is the number of data points.) Yet, the mathematical procedure to find coefficients $$\beta_j$$ and test their significance is the same.

• This is a very good answer (+1) and I was about to type mine when I saw yours. My problem with "linear approximations" is that they often (if not usually, if not "almost always") approximate something without any reason to approximate linearly. As an ex-physicist, the only moment I can approximate with a linear function is when there is a law that says so (with all the caveats of your, once again excellent, answer). No linear law, no linear approximation.
– WoJ
Commented Nov 17, 2023 at 12:18