Why study Linear Regression?

Question

Given two random variables $\xi$ and $\eta$ we can compute their "correlation coefficient" $c$, and form the line of best fit between these two random variables. My question is why?

1) There are random variables, $\xi$ and $\eta$ which are dependent in the worst possible way, i.e. $\xi = f(\eta)$ and despite this $c=0$. If one only thinks along linear regression, then one would be totally blinded to this.

2) Why linear specifically? There are other kinds of relationships that can exist between random variables. Why single that one out of all others?

Answer 1

I agree not all relations are linear in itself but quite a lot of relations can be linearly approximated. We have seen many such cases in mathematics such as the Taylor series or Fourier series etc. The key point here is, geomatt22 said in the comment, you can in general transform the nonlinear data and apply some kind of transformation with basis functions and linearize the relationship. The reason universities only address 'multiple linear regression models' (including simple regression models) is because they are the building block to models of a more advanced level which are also linear.

Mathematically speaking, as long as you can prove that a certain linear approximation is dense in a Hilbert space, then you will be able to use the approximation to represent a function in the space.

Answer 2

The model you are referring to, simple linear regression, a.k.a. "the line of best fit" (I am confusing model and estimation method here), is admittedly very simple (as the name says). Why studying it? I can see a lot of reasons. In the following I assume that the concept of random variable has been at least informally introduced, because you mentioned it in your question.

1. pedagogical: of course, for you it's obvious that real-valued random variables with finite second order moments form an Hilbert space. Maybe it was already obvious when you first studied probability theory. But statistics is not only teached to math students: there is a wider public, from physics to economics, to computer science, to social science, etc. These students may encounter statistics early in their course of study. They may or may not have been expoused to linear algebra, and even in the first case, they may not have seen it from the more abstract point of view of a math course. For these students, the very concept of approximating a random variable by another random variable is not so immediate. Even the basic property of the simple linear model, i.e., the fact that the error and the predictor are orthogonal random variables, is sometimes surprising to them. The fact that you can define an "angle" between random variables ("nasty" objects! measurable functions from a probability space to a measurable space) may be obvious to you, but not necessarily to a freshman. Thus, if the study of vector spaces starts with the good ol' Euclidean plane, doesn't it make sense to start the study of statistical models with the simplest one?
2. procedural: with simple linear regression you can introduce the concept of parameter estimation, and thus the method of least squares, standard errors, etc. in its simplest case. If you think this is trivial, keep in mind that a lot of professionals, who use statistics in their job/research but are not statisticians, are deeply confused about the frequentist confidence interval! Anyway, once the easiest case has been covered, you can go to multiple linear regression. Once this is mastered, then all the linear models are available for estimation. In other words, if I can fit the model $\xi = \beta_0+\sum_{i=1}^N \beta_i \eta_i +\epsilon$ (by OLS, or LARS in case regularization is needed, etc.), then I can fit all models of the kind $\xi = \sum_{i=0}^N \beta_i \phi(\eta_i) +\epsilon$. This is a really powerful class of models, which, as noted by @DaeyoungLim, can approximate all functions in the Hilbert space, if you have an infinite set of basis functions, and if they generate a vector subspace which is dense in the Hilbert space.
3. practical: there are numerous successful applications of simple linear regression. Okun's law in economics, Hooke's law, Ohm's law and Charles's law in physics, the relationship between blood systolic pressure and age in medicine (I have no idea if it has a name!) are all examples of simple linear regression, with varying degrees of accuracy.

Answer 3

A further reason is the lovely way regression gives a unified treatment of techniques like ANOVA. To me, the usual 'elementary' treatment of ANOVA seems quite obscure, yet a regression-based treatment is crystal clear. I suspect this has much to do with the way regression models make explicit some assumptions that in 'elementary' treatments are tacit and unexamined. Furthermore, the conceptual clarity offered by such a unifying perspective is accompanied by similar practical benefits when time comes to implement methods in statistical software.

This principle applies not only to ANOVA, but to extensions like restricted cubic splines--which notably address your second question.

Answer 4

Linear Regression's popularity is due in part to it's interpretability - that is, non-technical people can understand the parameter coefficients with just a little bit of explanation. This adds a great deal of value in business situations, where end users of the output or predictions may not have a deep understanding of math/statistics.

Yes, there are assumptions and limitations with this technique (as with all approaches), and it may not provide the best fit in many cases. But Linear Regression is very robust, and can often perform quite well even when assumptions are violated.

For these reasons, it is definitely worth studying.

Answer 5

Something might not be dirctly related.

If you have two series $x$ and $y$ that $cov(x,y) = 0$, and if you suspect there are relationship between $x$ and $y$. You could make a plot between $y$ and $x$ to examine their relationship.