When should linear regression be called "machine learning"?

Question

In a recent colloquium, the speaker's abstract claimed they were using machine learning. During the talk, the only thing related to machine learning was that they perform linear regression on their data. After calculating the best-fit coefficients in 5D parameter space, they compared these coefficients in one system to the best-fit coefficients of other systems.

When is linear regression machine learning, as opposed to simply finding a best-fit line? (Was the researcher's abstract misleading?)

With all the attention machine learning has been garnering recently, it seems important to make such distinctions.

My question is like this one, except that that question asks for the definition of "linear regression", whereas mine asks when linear regression (which has a broad number of applications) may appropriately be called "machine learning".

Clarifications

I'm not asking when linear regression is the same as machine learning. As some have pointed out, a single algorithm does not constitute a field of study. I'm asking when it's correct to say that one is doing machine learning when the algorithm one is using is simply a linear regression.

All jokes aside (see comments), one of the reasons I ask this is because it is unethical to say that one is doing machine learning to add a few gold stars to your name if they aren't really doing machine learning. (Many scientists calculate some type of best-fit line for their work, but this does not mean that they are doing machine learning.) On the other hand, there are clearly situations when linear regression is being used as part of machine learning. I'm looking for experts to help me classify these situations. ;-)

Answer 1

Answering your question with a question: what exactly is machine learning? Trevor Hastie, Robert Tibshirani and Jerome Friedman in The Elements of Statistical Learning, Kevin P. Murphy in Machine Learning A Probabilistic Perspective, Christopher Bishop in Pattern Recognition and Machine Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville in Deep Learning and a number of other machine learning "bibles" mention linear regression as one of the machine learning "algorithms". Machine learning is partly a buzzword for applied statistics and the distinction between statistics and machine learning is often blurry.

Answer 2

Linear regression is definitely an algorithm that can be used in machine learning. But, reductio ad absurdum: Anyone with a copy of Excel can fit a linear model.

Even restricting ourselves to linear models, there are a few more things to consider when discussing machine learning:

• Machine learning on business problems may involve a lot more data. "Big data", if you want to use the buzzword. Cleaning and preparing the data may take more work than the actual modelling. And when the volume of data exceeds the capacity of a single machine to process it then the engineering challenges are as significant as the statistical challenges. (Rule of thumb: if it fits in main memory it's not big data).
• Machine learning often involves many more explanatory variables (features) than traditional statistical models. Perhaps dozens, sometimes even hundreds of them, some of which will be categorical variables with many levels. When these features can potentially interact (e.g. in a cross effects model) the number of potential models to be fit grows rapidly.
• The machine learning practitioner is usually less concerned with the significance of individual features, and more concerned with squeezing as much predictive power as possible out of a model, using whichever combination of features does that. (P-values are associated with explanation, not prediction.)
• With a large number of features, and various ways of engineering those features, model selection by hand becomes infeasible. In my opinion, the real challenge in machine learning is the automated selection of features (feature engineering) and other aspects of model specification. With a linear model there are various ways of doing this, usually variants of brute force; including step-wise regression, back elimination etc, all of which again require significant computing power. (Second rule of thumb: if you are selecting features by hand, you are doing statistics, not machine learning).
• When you automatically fit many models with many features, over-fitting is a serious potential issue. Dealing with this problem often involves some form of cross validation: i.e. yet more brute force computation!

The short answer, from my point of view, is that where machine learning deviates from traditional statistical modelling is in the application of brute force and numerical approaches to model selection, especially in domains with a large amount of data and a large number of explanatory variables, with a focus on predictive power, followed by more brute force for model validation.

Answer 3

I think Mitchell's definition provides a helpful way to ground the discussion of machine learning, a sort of first principle. As reproduced on Wikipedia:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.

This is helpful in a few ways. First, to your immediate question: Regression is machine learning when its task is to provide an estimated value from predictive features in some application. Its performance should improve, as measured by mean squared (or absolute, etc.) held out error, as it experiences more data.

Second, it helps delineate machine learning from related terms, and its use as a marketing buzzword. Contrast the task above with a standard, inferential regression, wherein an analyst interprets coefficients for significant relationships. Here the program returns a summary: coefficients, p-values, etc. The program cannot be said to improve this performance with experience; the task is elaborate calculation.

Finally, it helps unify machine learning sub fields, both those commonly used in introductory exposition (supervised, unsupervised) with others like reinforcement learning or density estimation. (Each has a task, performance measure and concept of experience, if you think on them enough.) It provides, I think, a richer definition that helps delineate the two fields without unnecessarily reducing either. As an example, "ML is for prediction, statistics for inference" ignores both machine learning techniques outside supervised learning, and statistical techniques that focus on prediction.

Answer 4

There's no law that says that a cabinet maker can't use a barrel maker's saw.

Machine learning and statistics are vague labels, but if well-defined there is a lot of overlap between statistics and machine learning. And this goes for methods of these two areas as well as (and separately) for people who label themselves with these two areas. But as far as math goes, machine learning is entirely within the field of statistics.

Linear regression is a very well defined mathematical procedure. I tend to associate it with the area of statistics and people who call themselves 'statisticians' and those who come out of academic programs with labels like 'statistics'. SVM (Support Vector Machines) is likewise a very well defined mathematical procedure that has some every similar inputs and outputs and solves similar problems. But I tend to associate it however with the area of machine learning and people who call themselves computer scientists or people who work in artificial intelligence or machine learning which tend to be considered part of computer science as a discipline.

But some statisticians might use SVM and some AI people use logistic regression. Just to be clear, it is more likely that a statistician or AI researcher would develop a method than actually put it to practical use.

I put all the methods of machine learning squarely inside the domain of statistics. Even such recent things like Deep Learning, RNNs, CNNs, LSTMs, CRFs. An applied statistician (biostatistician, agronomist) may well not be familiar with them. Those are all predictive modeling methods usually labeled with 'machine learning', and rarely associated with statistics. But they are predictive models, with the allowance that they can be judged using statistical methods.

In the end, logistic regression must be considered part of machine learning.

But, yes, I see and often share your distaste for the misapplication of these words. Linear regression is such a fundamental part of things called statistics that it feels very strange and misleading to call its use 'machine learning'.

To illustrate, Logistic regression is identical mathematically to a Deep Learning network with no hidden nodes and the logistic function as the activation function for the single output node. I wouldn't call logistic regression a machine learning method, but it is certainly used in machine learning contexts.

It's mostly an issue of expectation.

A:"I used machine learning to predict readmission to a hospital after heart surgery."

B:"Oh yeah? Deep Learning? Random Forests?!!?"

A:"Oh, no, nothing as fancy as that, just Logistic Regression."

B: extremely disappointed look .

It's like saying, when washing a window with water that you're using quantum chemistry. Well yeah sure that's not technically wrong but you're implying a lot more than what's needed.

But really, that is exactly a culture difference vs. a substance difference. The connotations of a word and associations with groups of people (LR is totally not ML!) vs the math and applications (LR is totally ML!).

Answer 5

Linear regression is a technique, while machine learning is a goal that can be achieved through different means and techniques.

So regression performance is measured by how close it fits an expected line/curve, while machine learning is measured by how good it can solve a certain problem, with whatever means necessary.

Answer 6

It can be useful to call linear regression machine learning because doing so generally implies a couple important things about how you went about solving your problem:

1. You decided it wasn't necessary to check causal assumptions and prior theory behind your explanatory variables. It signals that your model was not intended to explain but to predict. This is perfectly reasonable in a lot of settings, for example, predicting email spam based on keywords. There isn't really a lot of literature on which words predict spam, and there are so many words it doesn't make sense to think through the theoretical significance of each word
2. You didn't check for variable significance or use p-values but instead likely opted for a holdout set or cross validation to assess out-of-sample predictive performance. This can be perfectly valid if - back to the email spam example - if really all you care about is producing a model that effectively predicts spam, even if this comes at at the cost of including variables that might not pass traditional significance tests.

However, if your model is more intended to explain than predict, and you do rigorously check your model's theoretical causal assumptions, etc then yes, it is rather silly to call it machine learning.

Answer 7

Common view is that machine learning made up of 4 areas:

1) Dimensionality Reduction

2) Clustering

3) Classification

4) Regression

Linear regression is a regression. Once the model is trained it could be used for predictions, like any other, say, Random Forest Regression.

Answer 8

I'll argue that the distinction between machine learning and statistical inference is clear. In short, machine learning = prediction of future observations; statistics = explanation.

Here is an example from my field of interest (medicine): when developing a drug, we search for gene(s) which best explain a disease state, with the goal of targeting it/them with the drug. We use statistis for that. In contrast, when developing diagnostic tests, for example predicting whether the drug will help a patient, the goal is strictly finding the best predictor of the future outcome, even if it comprises many genes and is too complicated to understand. We use machine learning for this purpose. There are multiple published examples [1], [2], [3], [4] showing that presence of the drug target is not a good predictor of the treatment outcome, hence the distinction.

Based on this, it is fair to say that one is doing machine learning when the goal is strictly predicting outcome of future/previously unseen observations. If the goal is understanding a particular phenomenon, then that is statistical inference, not machine learning. As others have pointed out, this is true regardless of the method involved.

To answer your question: in the specific research that you describe, the scientists were comparing the factor roles (weights) in different linear regression models, not comparing the model accuracies. Therefore, it is not accurate to call their inference machine learning.

[1] Messersmith WA, Ahnen DJ. Targeting EGFR in Colorectal Cancer. The New England Journal of Medicine; 2008; 359; 17.

[2] Pogue-Geile KL et al. Predicting Degree of Benefit From Adjuvant Trastuzumab in NSABP Trial B-31. J Natl Cancer Inst; 2013; 105:1782-1788.

[3] Pazdur R. FDA Approval for Vemurafenib. https://www.cancer.gov/about-cancer/treatment/drugs/fda-vemurafenib. Updated July 3, 2013.

[4] Ray T. Two ASCO Studies Show Challenge of Using MET Signaling as Predictive Marker in NSCLC Drug Trials. GenomeWeb, June 11, 2014.

Answer 9

Admittedly, any answer to this question is more opinion than objective fact, but I will try to lay out my logic why I think the answer is never. Any so-called machine learning expert or instructor only reveals their ignorance by representing linear regression as such.

Delineation of academic disciplines is more about delineation of communities than methods. Scientific disciplines borrow methods across disciplines all the time. Also, in the 19th century (when linear regression was developed) and prior to that, scientific disciplines were not so clearly delineated as they are today. So particularly when methods were developed in the 19th century or prior, we should be careful to assign them to a particular discipline.

That being said, one can look at the history of a discipline and reasonable conclude that particular methods "belong" to one discipline or another. No one would say today that calculus belongs to the field of physics, even though Newton, who was one of the inventors of calculus, was definitely trying to apply this to physics. Calculus clearly belongs to the discipline of mathematics, not physics. This is because calculus is a general mathematical method that can be used completely outside of a physics contexts.

By the same reasoning, linear regression belongs to the discipline of statistics, even though it is commonly used as a simple example of fitting data to a model in the context of machine learning. Just as calculus can be used outside the context of physics, linear regression can (and is) used outside the context of machine learning.

Machine learning instructors would be wise to point out that linear regression has been in use since the late 19th century long before the modern notion of machine learning came into existence. They should also emphasize that machine learning utilizes many concepts from probability and statistics, as well as other disciplines (e.g. information theory). However, these concepts do not themselves represent machine learning or an "algorithm" of machine learning.

Answer 10

It's the Machine, Stupid!

I am neither a statistician nor a Big Data(TM) expert. However, I would say that the essential distinction is that "machine learning" requires "a machine". In particular, it implies agency. The result will not be consumed leisurely by a human. Rather, the result will be the input to a closed cycle whereby an automated system improves its performance.

Closed System

This is very much in line with Sean Easter's answer, but I just want to emphasize that in commercial applications, a machine is looking at the results and acting on them. A classic example is the CineMatch algorithm which was the target of the Netflix Prize. A human could look at the output of CineMatch and learn interesting features about movie viewers. But that is not why it exists. The purpose of CineMatch is to provide a mechanism whereby Netflix servers can suggest movies to customers that they will enjoy. The output of the statistical model goes into the recommender service, which ultimately produces more input as customers rate movies, some of which were selected on the advice of CineMatch.

Open System

On the other hand, if a researcher uses an algorithm to produce statistical results which are displayed in a presentation to other humans, then that researcher is most decidedly not engaging in machine learning. This is, quite obviously to me, human learning. The analysis is performed by a machine, but it is not a machine that is doing the learning, per se. Now, it is "machine learning" to the extent that a human brain did not experience all of the sample inputs and derive the statistical results "biologically". But I would call it "statistics" because this is exactly what statisticians have been doing since the field was invented.

Conclusion

Thus, I would answer this question by asking: "Who consumes the results?" If the answer is: "humans", then it's "statistics". If the answer is: "software", then it's "machine learning." And when we say that "software consumes the results", we don't mean that it stores it somewhere for later retrieval. We mean that it performs behavior which is determined by the results in a closed loop.

Answer 11

One thing that I need to add (other have made great comments on distinction between ML and Stat) is that on the technical side, many of the classic assumptions of LR does not need to hold for predicting the mean response! Like no body even cares about "homoscedasticity" in ML, or even whether the residuals are normal (at least as long as they are symmetric around zero),... . People do careless univariate transformation on the response, while these are all important problems in statistics! Mainly because you don't need to do any inference about the coefficients, etc in ML!

Another related point, when it comes to how we use it, is that you need to calculate design matrix for LR in statistics (which is the closed form result of MLE). For ML though, people find coefficients by gradient descent, stochastic gradient descent, etc. At the end, they only need the coefficients!

Answer 12

In my opinion, one can speak of machine learning when a machine is programmed to infer parameters of some model using some data.

If a linear regression is done by machine, it therefore qualifies.

If done by hand, then it does not.

Definitions that hinge on the prevalence of some agent (like Excel), or iterative improvement (like Sean Easter suggests above), somehow trying to separate it from statistics or depending on what to do with the results will prove inconsistent, in my opinion.