90
$\begingroup$

In a recent colloquium, the speaker's abstract claimed they were using machine learning. During the talk, the only thing related to machine learning was that they perform linear regression on their data. After calculating the best-fit coefficients in 5D parameter space, they compared these coefficients in one system to the best-fit coefficients of other systems.

When is linear regression machine learning, as opposed to simply finding a best-fit line? (Was the researcher's abstract misleading?)

With all the attention machine learning has been garnering recently, it seems important to make such distinctions.

My question is like this one, except that that question asks for the definition of "linear regression", whereas mine asks when linear regression (which has a broad number of applications) may appropriately be called "machine learning".

Clarifications

I'm not asking when linear regression is the same as machine learning. As some have pointed out, a single algorithm does not constitute a field of study. I'm asking when it's correct to say that one is doing machine learning when the algorithm one is using is simply a linear regression.

All jokes aside (see comments), one of the reasons I ask this is because it is unethical to say that one is doing machine learning to add a few gold stars to your name if they aren't really doing machine learning. (Many scientists calculate some type of best-fit line for their work, but this does not mean that they are doing machine learning.) On the other hand, there are clearly situations when linear regression is being used as part of machine learning. I'm looking for experts to help me classify these situations. ;-)

$\endgroup$
  • 13
    $\begingroup$ Maybe you want to see the thread: "The Two Cultures: statistics vs. machine learning?". $\endgroup$ – usεr11852 Mar 20 '17 at 22:29
  • 75
    $\begingroup$ You should rename your regression as 'machine learning' whenever you want to double the fees on your rate card. $\endgroup$ – Sycorax Mar 20 '17 at 22:35
  • 3
    $\begingroup$ There is a difference. Learning is a process. A best fit is an objective. See my answer below. Frankly, the words do not have the same meaning, although the can appear in the same context, like "birds fly", one can associate the two, but birds are not flight, and although flying is for the birds, it is for F-18 fighter jets as well. $\endgroup$ – Carl Mar 21 '17 at 0:53
  • 20
    $\begingroup$ @Sycorax and deep learning when you want to quadruple $\endgroup$ – Franck Dernoncourt Mar 21 '17 at 3:17
  • 11
    $\begingroup$ @FranckDernoncourt "I'm a data scientist using deep learning in big data environment to solve machine learning problems" sounds like a nice header for LinkedIn profile ;) $\endgroup$ – Tim Mar 21 '17 at 8:51

11 Answers 11

78
$\begingroup$

Answering your question with a question: what exactly is machine learning? Trevor Hastie, Robert Tibshirani and Jerome Friedman in The Elements of Statistical Learning, Kevin P. Murphy in Machine Learning A Probabilistic Perspective, Christopher Bishop in Pattern Recognition and Machine Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville in Deep Learning and a number of other machine learning "bibles" mention linear regression as one of the machine learning "algorithms". Machine learning is partly a buzzword for applied statistics and the distinction between statistics and machine learning is often blurry.

$\endgroup$
  • 4
    $\begingroup$ True but they are in large part siloed disciplines with large quantities of nonoverlapping literature, methods and algorithms. For instance, in today's world machine learning, data and computer science grads are way ahead of statistical applicants in terms of funding, grants and job opps, you name it. $\endgroup$ – Mike Hunter Mar 20 '17 at 22:26
  • 6
    $\begingroup$ @DJohnson so it's applied statistics with new package, sold at higher price..? I do not think that the fact that it's trendy does not make it a buzzword. Bayesian statistics also have their own methods, journals, conferences, handbooks and applications that are partly non-overlapping with classical statistics - does it make it a discipline that is distinct to statistics? $\endgroup$ – Tim Mar 20 '17 at 22:57
  • 3
    $\begingroup$ Yup. I neglected to caveat my observation about ML practitioners with the more general observation that siloed, narrowly focused practitioners are endemic to every field and profession, not just ML. It's a kind of occupational hazard -- read human failing -- that people grow blinders to information outside their immediate needs and interests. CV is no exception to this. $\endgroup$ – Mike Hunter Mar 20 '17 at 23:18
  • 23
    $\begingroup$ (+1) I agree there is no clear distinction. To the extent I think of differences, I would typically think of ML as more concerned with the predictions, and statistics as more concerned with the parameter inference (e.g. experimental design for response surface modeling would not be typical in ML?). So in that sense, the OP example -- where the regression coefficients seem to be of most concern -- would be more "statistics-like" (?) $\endgroup$ – GeoMatt22 Mar 21 '17 at 1:43
  • 3
    $\begingroup$ See also The two cultures by Leo Breiman which makes a point similar to that of @GeoMatt22: ML focuses on accurate prediction. Whether the model is true is not important. Classical statistics is looking for the "true" model, in some sense, or at least a model that gives some insight into the processes that produced the data. $\endgroup$ – Peter Mar 21 '17 at 8:12
41
$\begingroup$

Linear regression is definitely an algorithm that can be used in machine learning. But, reductio ad absurdum: Anyone with a copy of Excel can fit a linear model.

Even restricting ourselves to linear models, there are a few more things to consider when discussing machine learning:

  • Machine learning on business problems may involve a lot more data. "Big data", if you want to use the buzzword. Cleaning and preparing the data may take more work than the actual modelling. And when the volume of data exceeds the capacity of a single machine to process it then the engineering challenges are as significant as the statistical challenges. (Rule of thumb: if it fits in main memory it's not big data).
  • Machine learning often involves many more explanatory variables (features) than traditional statistical models. Perhaps dozens, sometimes even hundreds of them, some of which will be categorical variables with many levels. When these features can potentially interact (e.g. in a cross effects model) the number of potential models to be fit grows rapidly.
  • The machine learning practitioner is usually less concerned with the significance of individual features, and more concerned with squeezing as much predictive power as possible out of a model, using whichever combination of features does that. (P-values are associated with explanation, not prediction.)
  • With a large number of features, and various ways of engineering those features, model selection by hand becomes infeasible. In my opinion, the real challenge in machine learning is the automated selection of features (feature engineering) and other aspects of model specification. With a linear model there are various ways of doing this, usually variants of brute force; including step-wise regression, back elimination etc, all of which again require significant computing power. (Second rule of thumb: if you are selecting features by hand, you are doing statistics, not machine learning).
  • When you automatically fit many models with many features, over-fitting is a serious potential issue. Dealing with this problem often involves some form of cross validation: i.e. yet more brute force computation!

The short answer, from my point of view, is that where machine learning deviates from traditional statistical modelling is in the application of brute force and numerical approaches to model selection, especially in domains with a large amount of data and a large number of explanatory variables, with a focus on predictive power, followed by more brute force for model validation.

$\endgroup$
  • 2
    $\begingroup$ I do like this distinction in general. However, is cross-validation ever used in "statistical" models or is this rarely needed as they are normally done by hand? Is feature engineering considered statistics then as it is done by hand? $\endgroup$ – josh Mar 22 '17 at 9:13
  • 3
    $\begingroup$ @josh, Yes, it can be. But if you look at the cross validation tag, almost all the questions are about predictive modelling. $\endgroup$ – david25272 Mar 22 '17 at 22:26
  • $\begingroup$ @david25272 I'd be curious as to how you think of the bootstrap, .632+ bootstrap, and permutation tests -- I've always thought of them as more "applied statistics" than "machine learning" because of how they're motivated, but they're similarly "brute-force" to k-fold or leave-k-out cross-validation. I think L1 regularization can also be thought of as a type of feature selection within a statistical framework... $\endgroup$ – Patrick B. Mar 22 '17 at 23:58
  • $\begingroup$ @Patrick stats.stackexchange.com/questions/18348 is a better answer on the uses of bootstapping for model validation than I could give. $\endgroup$ – david25272 Mar 23 '17 at 1:37
  • $\begingroup$ @david25272 ah, sorry, my question was more whether you think of them as "machine learning" techniques or "applied statistics" techniques, since they're statistically motivated but also "brute force." I'm familiar with the use of bias-corrected bootstraps for model validation. $\endgroup$ – Patrick B. Mar 24 '17 at 1:19
14
$\begingroup$

I think Mitchell's definition provides a helpful way to ground the discussion of machine learning, a sort of first principle. As reproduced on Wikipedia:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.

This is helpful in a few ways. First, to your immediate question: Regression is machine learning when its task is to provide an estimated value from predictive features in some application. Its performance should improve, as measured by mean squared (or absolute, etc.) held out error, as it experiences more data.

Second, it helps delineate machine learning from related terms, and its use as a marketing buzzword. Contrast the task above with a standard, inferential regression, wherein an analyst interprets coefficients for significant relationships. Here the program returns a summary: coefficients, p-values, etc. The program cannot be said to improve this performance with experience; the task is elaborate calculation.

Finally, it helps unify machine learning sub fields, both those commonly used in introductory exposition (supervised, unsupervised) with others like reinforcement learning or density estimation. (Each has a task, performance measure and concept of experience, if you think on them enough.) It provides, I think, a richer definition that helps delineate the two fields without unnecessarily reducing either. As an example, "ML is for prediction, statistics for inference" ignores both machine learning techniques outside supervised learning, and statistical techniques that focus on prediction.

$\endgroup$
12
$\begingroup$

There's no law that says that a cabinet maker can't use a barrel maker's saw.

Machine learning and statistics are vague labels, but if well-defined there is a lot of overlap between statistics and machine learning. And this goes for methods of these two areas as well as (and separately) for people who label themselves with these two areas. But as far as math goes, machine learning is entirely within the field of statistics.

Linear regression is a very well defined mathematical procedure. I tend to associate it with the area of statistics and people who call themselves 'statisticians' and those who come out of academic programs with labels like 'statistics'. SVM (Support Vector Machines) is likewise a very well defined mathematical procedure that has some every similar inputs and outputs and solves similar problems. But I tend to associate it however with the area of machine learning and people who call themselves computer scientists or people who work in artificial intelligence or machine learning which tend to be considered part of computer science as a discipline.

But some statisticians might use SVM and some AI people use logistic regression. Just to be clear, it is more likely that a statistician or AI researcher would develop a method than actually put it to practical use.

I put all the methods of machine learning squarely inside the domain of statistics. Even such recent things like Deep Learning, RNNs, CNNs, LSTMs, CRFs. An applied statistician (biostatistician, agronomist) may well not be familiar with them. Those are all predictive modeling methods usually labeled with 'machine learning', and rarely associated with statistics. But they are predictive models, with the allowance that they can be judged using statistical methods.

In the end, logistic regression must be considered part of machine learning.

But, yes, I see and often share your distaste for the misapplication of these words. Linear regression is such a fundamental part of things called statistics that it feels very strange and misleading to call its use 'machine learning'.

To illustrate, Logistic regression is identical mathematically to a Deep Learning network with no hidden nodes and the logistic function as the activation function for the single output node. I wouldn't call logistic regression a machine learning method, but it is certainly used in machine learning contexts.

It's mostly an issue of expectation.

A:"I used machine learning to predict readmission to a hospital after heart surgery."

B:"Oh yeah? Deep Learning? Random Forests?!!?"

A:"Oh, no, nothing as fancy as that, just Logistic Regression."

B: extremely disappointed look .

It's like saying, when washing a window with water that you're using quantum chemistry. Well yeah sure that's not technically wrong but you're implying a lot more than what's needed.

But really, that is exactly a culture difference vs. a substance difference. The connotations of a word and associations with groups of people (LR is totally not ML!) vs the math and applications (LR is totally ML!).

$\endgroup$
3
$\begingroup$

Common view is that machine learning made up of 4 areas:

1) Dimensionality Reduction

2) Clustering

3) Classification

4) Regression

Linear regression is a regression. Once the model is trained it could be used for predictions, like any other, say, Random Forest Regression.

$\endgroup$
  • $\begingroup$ There is actually a difference, although linear regression can be solved using machine learning. A common regression target is ordinary least squares, which means, that our target loss function, sum squared residuals, is to be minimized. Now, machine learning would simply refer to that method by which we minimize a loss function. $\endgroup$ – Carl Mar 21 '17 at 2:04
  • $\begingroup$ Thus conceptually, linear regression via gradient descent (learning) chooses better and better summed square residuals (loss function). The basic concepts are the same as those for much more advanced learning algorithms, such as neural networks. These algorithms simply replace the linear model with a much more complex model - and, correspondingly, a much more complex cost function.. $\endgroup$ – Carl Mar 21 '17 at 2:05
  • 1
    $\begingroup$ So the answer to the OP question When is linear regression machine learning, as opposed to simply finding a best-fit line? When linear regression is performed using a definable element of machine learning, like gradient descent, it is then linear regression performed using machine learning. $\endgroup$ – Carl Mar 21 '17 at 2:06
  • 5
    $\begingroup$ @Carl, the problem here that "machine learning" defined. To me if we can use a statistical model, and that model would have ability to predict it is machine learning. And it does not matter what approach was used to find the coefficients of the model. $\endgroup$ – Akavall Mar 21 '17 at 3:59
  • 1
    $\begingroup$ I found Akavall's reply pretty clear. I believe Akavall's problem is that the definition you present is circular, because it appears to boil down to "Q: when does technique X count as 'machine learning'? A: when technique X is performed using a definable element of machine learning." (Unfortunately I don't understand the second point you're making so I can't respond to that.) $\endgroup$ – Patrick B. Mar 22 '17 at 23:20
2
$\begingroup$

Linear regression is a technique, while machine learning is a goal that can be achieved through different means and techniques.

So regression performance is measured by how close it fits an expected line/curve, while machine learning is measured by how good it can solve a certain problem, with whatever means necessary.

$\endgroup$
2
$\begingroup$

I'll argue that the distinction between machine learning and statistical inference is clear. In short, machine learning = prediction of future observations; statistics = explanation.

Here is an example from my field of interest (medicine): when developing a drug, we search for gene(s) which best explain a disease state, with the goal of targeting it/them with the drug. We use statistis for that. In contrast, when developing diagnostic tests, for example predicting whether the drug will help a patient, the goal is strictly finding the best predictor of the future outcome, even if it comprises many genes and is too complicated to understand. We use machine learning for this purpose. There are multiple published examples [1], [2], [3], [4] showing that presence of the drug target is not a good predictor of the treatment outcome, hence the distinction.

Based on this, it is fair to say that one is doing machine learning when the goal is strictly predicting outcome of future/previously unseen observations. If the goal is understanding a particular phenomenon, then that is statistical inference, not machine learning. As others have pointed out, this is true regardless of the method involved.

To answer your question: in the specific research that you describe, the scientists were comparing the factor roles (weights) in different linear regression models, not comparing the model accuracies. Therefore, it is not accurate to call their inference machine learning.

[1] Messersmith WA, Ahnen DJ. Targeting EGFR in Colorectal Cancer. The New England Journal of Medicine; 2008; 359; 17.

[2] Pogue-Geile KL et al. Predicting Degree of Benefit From Adjuvant Trastuzumab in NSABP Trial B-31. J Natl Cancer Inst; 2013; 105:1782-1788.

[3] Pazdur R. FDA Approval for Vemurafenib. https://www.cancer.gov/about-cancer/treatment/drugs/fda-vemurafenib. Updated July 3, 2013.

[4] Ray T. Two ASCO Studies Show Challenge of Using MET Signaling as Predictive Marker in NSCLC Drug Trials. GenomeWeb, June 11, 2014.

$\endgroup$
  • 7
    $\begingroup$ I agree that machine learning research has a much heavier emphasis on predictions over parameter estimation. But that's not a clear dividing line: statistics research is rich with predictive methods. $\endgroup$ – Cliff AB Mar 21 '17 at 20:44
  • 4
    $\begingroup$ So what about statisticians that made predictions before computers existed (or were widely available)? Were they applying paper-and-pencil machine learning?! $\endgroup$ – Tim Mar 22 '17 at 8:35
  • 1
    $\begingroup$ @Tim: very fine argument. I believe the answer is yes if they were focused on future observations, though I acknowledge in those (rare) cases the name statistical learning would be more appropriate. With the advent of computers, the term machine learning became more fashionable. The point is not the name, nor the use of computers; it is the clarity of purpose. In my view, it is almost impossible to successfully optimize both accurate prediction of previously unseen observations, and understanding of the phenomenon. Better to focus appropriately. $\endgroup$ – ljubomir Mar 22 '17 at 11:02
  • 4
    $\begingroup$ Time series forecasting (prediction of future observation) was long a popular problem in statistics (and econometrics), so I do not agree with a clear distinction based on that. $\endgroup$ – Richard Hardy Mar 24 '17 at 16:54
  • 1
    $\begingroup$ This answer is bogus. Prediction is just one small part of machine learning. Statisticians also do prediction. While it is hard to delineate between machine learning and statistics, but this is definitely not the correct way. $\endgroup$ – robguinness Jan 15 '18 at 9:19
2
$\begingroup$

It can be useful to call linear regression machine learning because doing so generally implies a couple important things about how you went about solving your problem:

  1. You decided it wasn't necessary to check causal assumptions and prior theory behind your explanatory variables. It signals that your model was not intended to explain but to predict. This is perfectly reasonable in a lot of settings, for example, predicting email spam based on keywords. There isn't really a lot of literature on which words predict spam, and there are so many words it doesn't make sense to think through the theoretical significance of each word
  2. You didn't check for variable significance or use p-values but instead likely opted for a holdout set or cross validation to assess out-of-sample predictive performance. This can be perfectly valid if - back to the email spam example - if really all you care about is producing a model that effectively predicts spam, even if this comes at at the cost of including variables that might not pass traditional significance tests.

However, if your model is more intended to explain than predict, and you do rigorously check your model's theoretical causal assumptions, etc then yes, it is rather silly to call it machine learning.

$\endgroup$
2
$\begingroup$

Admittedly, any answer to this question is more opinion than objective fact, but I will try to lay out my logic why I think the answer is never. Any so-called machine learning expert or instructor only reveals their ignorance by representing linear regression as such.

Delineation of academic disciplines is more about delineation of communities than methods. Scientific disciplines borrow methods across disciplines all the time. Also, in the 19th century (when linear regression was developed) and prior to that, scientific disciplines were not so clearly delineated as they are today. So particularly when methods were developed in the 19th century or prior, we should be careful to assign them to a particular discipline.

That being said, one can look at the history of a discipline and reasonable conclude that particular methods "belong" to one discipline or another. No one would say today that calculus belongs to the field of physics, even though Newton, who was one of the inventors of calculus, was definitely trying to apply this to physics. Calculus clearly belongs to the discipline of mathematics, not physics. This is because calculus is a general mathematical method that can be used completely outside of a physics contexts.

By the same reasoning, linear regression belongs to the discipline of statistics, even though it is commonly used as a simple example of fitting data to a model in the context of machine learning. Just as calculus can be used outside the context of physics, linear regression can (and is) used outside the context of machine learning.

Machine learning instructors would be wise to point out that linear regression has been in use since the late 19th century long before the modern notion of machine learning came into existence. They should also emphasize that machine learning utilizes many concepts from probability and statistics, as well as other disciplines (e.g. information theory). However, these concepts do not themselves represent machine learning or an "algorithm" of machine learning.

$\endgroup$
1
$\begingroup$

It's the Machine, Stupid!

I am neither a statistician nor a Big Data(TM) expert. However, I would say that the essential distinction is that "machine learning" requires "a machine". In particular, it implies agency. The result will not be consumed leisurely by a human. Rather, the result will be the input to a closed cycle whereby an automated system improves its performance.

Closed System

This is very much in line with Sean Easter's answer, but I just want to emphasize that in commercial applications, a machine is looking at the results and acting on them. A classic example is the CineMatch algorithm which was the target of the Netflix Prize. A human could look at the output of CineMatch and learn interesting features about movie viewers. But that is not why it exists. The purpose of CineMatch is to provide a mechanism whereby Netflix servers can suggest movies to customers that they will enjoy. The output of the statistical model goes into the recommender service, which ultimately produces more input as customers rate movies, some of which were selected on the advice of CineMatch.

Open System

On the other hand, if a researcher uses an algorithm to produce statistical results which are displayed in a presentation to other humans, then that researcher is most decidedly not engaging in machine learning. This is, quite obviously to me, human learning. The analysis is performed by a machine, but it is not a machine that is doing the learning, per se. Now, it is "machine learning" to the extent that a human brain did not experience all of the sample inputs and derive the statistical results "biologically". But I would call it "statistics" because this is exactly what statisticians have been doing since the field was invented.

Conclusion

Thus, I would answer this question by asking: "Who consumes the results?" If the answer is: "humans", then it's "statistics". If the answer is: "software", then it's "machine learning." And when we say that "software consumes the results", we don't mean that it stores it somewhere for later retrieval. We mean that it performs behavior which is determined by the results in a closed loop.

$\endgroup$
  • 8
    $\begingroup$ This is a reasonable point, but I think in practice ML models are often handed off to people to interpret & work with. $\endgroup$ – gung Mar 22 '17 at 20:21
  • 1
    $\begingroup$ I would say that's because ML as a field has spawned a variety of useful tools leveraged by statisticians, even if that's not what they want to call themselves, for marketing purposes. ;) $\endgroup$ – Lawnmower Man Mar 23 '17 at 18:31
  • $\begingroup$ I strongly agree with @gung; similar to other answers, I agree that this is more often the motivation for people who call themselves "ML researchers", its definitely not a defining line. Two counter examples: recommender systems are considered a ML research area, but the results are fed directly to a human. Kalman filters are very often used in navigation for auto-pilot, with no human in the loop, yet are typically considered to be a statistics methodology. $\endgroup$ – Cliff AB Apr 7 '17 at 22:51
-1
$\begingroup$

In my opinion, one can speak of machine learning when a machine is programmed to infer parameters of some model using some data.

If a linear regression is done by machine, it therefore qualifies.

If done by hand, then it does not.

Definitions that hinge on the prevalence of some agent (like Excel), or iterative improvement (like Sean Easter suggests above), somehow trying to separate it from statistics or depending on what to do with the results will prove inconsistent, in my opinion.

$\endgroup$
  • 3
    $\begingroup$ So if you calculate regression, or kNN, or decision tree, using paper and pencil and get the same results as calculated on computer, then in the first case it would be a machine learning and in second not..? On another hand, if you use a computer to randomly assign some values as "parameters" of your model, then you'd qualify it as a machine learning since it was done by a machine? This definition does not seem to have much sense... $\endgroup$ – Tim Nov 4 '17 at 20:00
  • $\begingroup$ You can hardly call it machine learning if you don't use a machine. It is the machine that learns, after all. And I have actually deployed models that "learned" their parameters by a random (Monte Carlo) process. However, I must admit that there was a validation step involved afterwards. $\endgroup$ – Ytsen de Boer Nov 5 '17 at 20:42
  • 2
    $\begingroup$ Algorithms like Support Vector Machines are called as "machines" for historical reasons, because in the early days people would have to build actual machines/computers to run them ( stats.stackexchange.com/questions/261041/… ), it has nothing to do with "algorithms that are run on machines". Moreover, time-series models like ARIMA are not in the scope of machine learning, but statistics, and they are run on computers. $\endgroup$ – Tim Nov 9 '17 at 12:33

protected by Tim Mar 22 '17 at 21:40

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.