I am currently in a linear regression class, but I can't shake the feeling that what I am learning is no longer relevant in either modern statistics or machine learning. Why is so much time spent on doing inference on simple or multiple linear regression when so many interesting datasets these days frequently violate many of the unrealistic assumptions of linear regression? Why not instead teach inference on more flexible, modern tools like regression using support vector machines or Gaussian process? Though more complicated than finding a hyperplane in a space, wouldn't this give students a much better background for which to tackle modern day problems?
$\begingroup$
$\endgroup$
11
-
12$\begingroup$ Do screwdrivers make hammers obsolete? Or does each perform a different task? $\endgroup$– Sycorax ♦Sep 26, 2017 at 21:45
-
7$\begingroup$ I have a multitool that functions as a knife, a saw, a couple of different screwdrivers, a pair of pliers, and probably a couple of other things, but when I need any of those tools it's the last thing I'd reach for. It's only useful in a pinch, it's never the "best tool for the job". $\endgroup$– DarrenSep 26, 2017 at 23:12
-
8$\begingroup$ Many, many situations faced by real people involve very small data sets with high noise; in many cases more complex models are not feasible while at least a good fraction of the time a plain linear model is at least tenable. While large data sets (and their associated issues) will continue to grow as a proportion of the total data analysis that goes on, very small data sets and the relatively simple analyses they rely on will never go away. Added to that the more sophisticated tools are built directly on top of simpler ones, not just historically but conceptually. $\endgroup$– Glen_bSep 27, 2017 at 1:22
-
7$\begingroup$ In addition to the many situations where linear regression is of continued practical use, it's also worth pointing out that it is foundational in learning about a broad class of more sophisticated additive models. In that respect, this question is sorta like asking whether calculus makes arithmetic obsolete. $\endgroup$– Jacob SocolarSep 27, 2017 at 2:03
-
1$\begingroup$ @Aksakal Please elaborate. What about use in Bayesian optimization? $\endgroup$– Mark L. StoneSep 27, 2017 at 13:29
|
Show 6 more comments
5 Answers
$\begingroup$
$\endgroup$
3
It is true that the assumptions of linear regression aren't realistic. However, this is true of all statistical models. "All models are wrong, but some are useful."
I guess you're under the impression that there's no reason to use linear regression when you could use a more complex model. This isn't true, because in general, more complex models are more vulnerable to overfitting, and they use more computational resources, which are important if, e.g., you're trying to do statistics on an embedded processor or a web server. Simpler models are also easier to understand and interpret; by contrast, complex machine-learning models such as neural networks tend to end up as black boxes, more or less.
Even if linear regression someday becomes no longer practically useful (which seems extremely unlikely in the foreseeable future), it will still be theoretically important, because more complex models tend to build on linear regression as a foundation. For example, in order to understand a regularized mixed-effects logistic regression, you need to understand plain old linear regression first.
This isn't to say that more complex, newer, and shinier models aren't useful or important. Many of them are. But the simpler models are more widely applicable and hence more important, and clearly make sense to present first if you're going to present a variety of models. There are a lot of bad data analyses conducted these days by people who call themselves "data scientists" or something but don't even know the foundational stuff, like what a confidence interval really is. Don't be a statistic!
-
$\begingroup$ Can you clarify what you mean by a "complex model"? Does OP mean the same thing? $\endgroup$ Sep 26, 2017 at 23:50
-
1$\begingroup$ @Hatshepsut Practically anything that isn't just linear regression or a special case thereof. The OP gave SVMs and Gaussian-process models as examples. I mentioned mixed models, logistic regression, and penalized regression. Some other examples are decision trees, neural networks, MARS, Bayesian hierarchical models, and structural equation models. If you're asking how we decide whether one model is more complex than another, or what exactly counts as a model, those are Cross Validated questions unto themselves. $\endgroup$ Sep 27, 2017 at 0:10
-
$\begingroup$ "Overfitting"; like using a ninth-order polynomial to fit something that turned out to be a weighted sum of exponentials. It fit so good the plot reproduced the instrument errors just above the noise level. I still wonder if actually using that polynomial would have worked better. $\endgroup$– JoshuaSep 27, 2017 at 3:17
$\begingroup$
$\endgroup$
Linear regression in general is not obsolete. There are still people that are working on research around LASSO-related methods, and how they relate to multiple testing for example - you can google Emmanuel Candes and Malgorzata Bogdan.
If you're asking about OLS algorithm in particular, the answer why they teach this is that method is so simple that it has closed-form solution. Also it's just simpler than ridge regression or the version with lasso/elasticnet. You can build your intuition/proofs on the solution to simple linear regression and then enrich the model with additional constraints.
$\begingroup$
$\endgroup$
I don't think regression is old, it might be considered as trivial for some problems that are currently faced by data scientists, but still is the ABC of statistical analysis. How are you supposed to understand if SVM are working correctly if you don't know how the simplest model is working? Using such a simple tool teaches YOU how to look into the data before jumping into crazy complex models and understand deeply which tools can be used in further analysis and which cannot. Once having this conversation with a professor and colleague of mine she told me that her students where great in applying complex models but they could not understand what leverage is or read a simple qq-plot to understand what was wrong with the data. Often in the most simple and readable model stands the beauty.
$\begingroup$
$\endgroup$
3
The short answer is no. For example, if you try linear model with MNIST data, you will still get ~90% of the accuracy!
A long answer would be "depending on the domain", but linear model is widely used.
In certain fields, say, medical study, it is super expensive to get one data point. And the analysis work is still similar to many years ago: linear regression is still plays an very important role.
In morden machine learning, say, text classification, linear model is still very important, although there are other fancier models. This is because linear model is very "stable", it will have less like to over fit the data.
Finally, linear model is really the building blocks for most of the other models. Learning in well will benefit you in the future.
-
$\begingroup$ Are you sure you can get 90% accuracy on MNIST using linear regression? Is that test set or train set accuracy? $\endgroup$ Jan 24, 2021 at 20:09
-
-
$\begingroup$ That is not linear regression. It is classification with linear layer. $\endgroup$ Jan 26, 2021 at 1:44
$\begingroup$
$\endgroup$
In practical terms, linear regression is useful even if you are also using a more complex model for your work. The key is that linear regression is easy to understand and therefore easy to use to conceptually understand what is happening in more complex models.
I can offer you a practical application example from my real live job as a statistical analyst. If you find yourself out in the wild, unsupervised, with a large dataset, and your boss asks you to run some analysis on it, where do you start? Well, if you are unfamiliar with the dataset and don't have a good idea of how the various features are expected to relate to each other, then a complex model like the ones you suggested is a bad place to start investigating.
Instead, the best place to start is simple old linear regression. Perform a regression analysis, look at coefficients and graph the residuals. Once you start to see what is going on with the data, then you can make some decisions as to what advanced methods you are going to try to apply.
I assert that if you just plugged your data into some advanced model black box like sklearn.svm (if you are into Python), then you will have very low confidence that your results will be meaningful.