From a theoretical standpoint, why is linear regression useful? [closed]

Question

I am not asking about how linear regression is derived. I am asking why oftentimes linear regression is the most useful to predict data. I understand that there are different types of regression, but what conditions must data satisfy in order for it to be represented by a line? There is always a hypothetical line that can go through a set of points with infinite accuracy but it is not necessarily a good predictor, why? In essence, why is precision counterproductive? And what specific mathematics must I learn about to answer this question? All the resources I have been exposed to merely go into the derivation of the line of best fit. I apologize if my thoughts are messy, but even a guess of where to look is much appreciated.

Answer 1

It seems that Robert has already addressed the assumptions required in regression and some of the theoretical underpinnings of regression. Along with Robert's useful answer, I would like to highlight some additional reasons, which are similarly related to each other:

Regression predicts rather than describes. The whole reason regression was invented was to understand sweet pea genetics. If Galton could predict with some reliability how the plants would grow, he could determine what would be useful and what would be wasteful. The reason this method became useful was due to it's ability to model in variation. A naive estimate of the predicted value of an outcome is generally considered the mean (but could be other values). A regression makes this idea conditional upon variation based off different inputs. This was one of the beginning problems that Galton faced, which was taking rather uninformative descriptive data like the mean and transforming it into a modeling tool which makes the mean conditional (see plots of the mean and conditional mean regression below).

If we consider this a prediction problem, we can see how the first plot is bad and the second plot is at least in some part better. Let's say we are trying to predict $y$ here. If we just guess that it will be the mean of $y$, we will obviously miss by a lot, but it won't be totally wrong...any given $y$ here should hover around the mean. However, if we make that conditional on $x$, we can make better guesses. For example, we can probably assume that $x=1$ should yield an outcome of $y=2$. This leads into my next point.

Fitting is not the same as overfitting. I gather from the question that part of your issue is with overfitting, where any naive line is fit to data without consideration of the data generating process (DGP). Lets say we have an archer who is shooting arrows at a target:

Visually inspecting this relationship, we should predict to a degree that the relationship has a downward-facing parabolic function...the arrow should go up in height, then go down in height. For simplicity, we pretend the height starts and ends at zero feet from the ground, and hits a maximum around 7 feet:

We could just fit any line here:

But you can see both of these are poor approximations of the DGP (I made a more extreme version here than my original answer for a better illustration). The first linear fit would make for terrible predictions. We would expect the arrow to shoot downward and never stop. The other is overfit and makes similarly bad predictions (its a 20th order polynomial which interpolates badly), so where we predict the arrow is all over the place. However, if we simply understand that this is a downward parabola, we can fit a much simpler and reasonable relationship that makes for better predictions:

Our assumptions allow these predictions to be better. Part of your question was about which conditions allow for a regression to function well. Robert already noted some, but I'll highlight that linearity is an important one which you can see from my example (note that we wish for linearity in the parameters, even with nonlinear data). Another example is heterogeneity of variance. Here you can see from the Engel data from Koenker & Hallock, 2001 that the dispersion of data points is rather unequal. The data first clusters to the bottom and then fans out across the scatterplot. Once again, we fit a linear regression:

Because our data doesn't meet the heterogeneity of variance assumption, predictions are once again potentially poor. We can guess with considerable certainty what low values of income should give us in terms of food expenditure, but that becomes a lot more uncertain as we move to the top right. Thus our predictions are only as good as our regressions meet the minimum standards for fitting.

Edit

I just saw that you also asked about what math is necessary to learn the intuition behind regression. A lot can be learned from basic algebra (such as what a "function" is), calculus (how things like the slope work for linear and nonlinear functions), linear algebra (learning things like how the coefficients are constructed), probability (understanding hypothesis testing in regression) and of course statistics. At a higher level, you may need to take classes that combine all this math in something like a linear models class.

However, much of this can also be simply understood by experience with data and simulation. In fact, much of what I know now about regression is a combination of math I have learned and simulation based around that math.

Answer 2

From a theoretical standpoint, why is linear regression useful?

I'm not sure what a theoretical standpoint would be since linear regression is about fitting a model to data, which is very much a practical thing. Anyway here are a few reasons why I think it is useful:

1. Quite often we want to model a real-world data-generating process, either to investigate possible causal relations, or to make predictions. It turns out that many such processes are well approximated by a straight line, at least over a relatively small part of the domain (note that extrapolation can be very dangerous)

2. Even when there are curvilinear associations, these can often be accommodated in a linear regression model by using higher order terms or splines - the model is still linear in the parameters.

3. Ease of interpretation. Compared to many other types of models, the results from linear regression are fairly easy to understand for non-technical people. This is very useful in business when explaining

4. The linear regression model can be extended to generalised linear models, linear mixed effects models, ridge and Lasso regression, quantile regression, robust regression, among others.

5. One-way ANOVA, two-way ANOVA (and as many ways as you like) as well as ANCOVA are just special cases of linear regression.

what conditions must data satisfy in order for it to be represented by a line?

I won't go into this, since it has been discussed on this site many times. Here are a couple of threads:

What is a complete list of the usual assumptions for linear regression?

Understanding the assumptions of Linear Regression

Answer 3

I am going to answer this question, "I understand that there are different types of regression, but what conditions must data satisfy in order for it to be represented by a line?".

---

The field of statistics is (largely) about making predictions from data. It is inherently an applied subject. In the "real world", what matters more than anything else are how accurate your predictions are. Sometimes we use prediction methods that are mathematically flawed, but we use them anyway because they are easier to work. In the end what really matters is how accurate it is.

There is a classification techinique called "Naive Bayes Classifier". It is called "naive", because you naively assume that all the predictor variables are independent. This is usually not true. But you just boldly assume it anyway. And guess what, despite the abuse of the math, it still can be a good classifier.

Your question sounds like you are asking what is the mathematical basis for doing linear regression? The answer is, you often do not need to justify your method. You just try it and see how well it works.

By the way, there are other types of regression methods, not just least squares. Some of them might work better than others. In the end it is just real-life experimentation about what prediction method works better for a specific problem.

The math behind it might describe some perfect ideal situation, but in the real-world those assumptions are never valid, and a statistician usually does not care. The statistician will just try something and see how well the prediction works.

Answer 4

I would like to add a couple of remarks from a non-statistician point of view:

Many real-world relations can be approximated by a linear dependence
Mathematicaly it simply means that a differentiable function can be approximated by Taylor series: $$ y=f(x) \approx f(0) + f'(0)x +... $$ and for many practical purposes this is good enough. In fact, in sciences such linear approximations even get to be called "laws", although they are certainly not fundamental laws. Just a few examples known from high school:

• Ohm's law (linearized current-voltage dependence)
• Hooke's law - linearized dependence between stress and strain
• Modification of electric field in a media $\mathbf{E}\rightarrow \epsilon\mathbf{E}$ or polarizability of molecules $\mathbf{P}=\chi \mathbf{E}$.

Here is, e.g., the actual dependence of stress on strain:
but the initial linear region is sufficient for many applications.

For the discussion of Ohm's law see: Please provide me with a proof for the formula : $R(T)=\rho(1+\alpha T)$ which relates the change of resistivity with temperature

Linear regression is not necessarily a line
Term Linear regression is somewhat ambiguous. In some cases it implies the linear dependence between a response variable and a number of predictor variables, as $$ y=\beta_0 + \sum_{j=1}^n\beta_j x_j. $$ Whoever, the predictor variables may be themselves functions of another variable. E.g., by choosing $x_j=x^j$ the above equation becomes fitting data with a polynomial of power $n$, which can pass through every point of the data (if $n$ is the number of data points.) Yet, the mathematical procedure to find coefficients $\beta_j$ and test their significance is the same.