18
$\begingroup$

If I understood correctly, in a machine learning algorithm, the model has to learn from its experience, i.e when the model gives the wrong prediction for the new cases, it must adapt to the new observations, and in time, the model becomes increasingly better. I don't see that the logistic regression has this characteristic. So why is it still regarded as a machine learning algorithm? What is the difference between logistic regression with the normal regression in term of "learning"?

I have the same question for random forests!

And what is the definition of "machine learning"?

$\endgroup$
  • 4
    $\begingroup$ I edited your question for grammatical clarity, but am not sure what you mean overall... Logistic Regression falls under ML because it is a classification algorithm. Machine Learning does not imply that the algorithm has to be adaptive (although there are algorithms that learn from new observations). Adapting is more an implementation choice, usually achieved by generative machine learning algorithms which model the joint probability. $\endgroup$ – Zhubarb Jun 25 '15 at 13:06
  • 12
    $\begingroup$ "Machine learning" is a rather loosely defined concept. Really, all statistical procedures that involve fitting a model can be thought of machine learning. (Assuming the model fitting can be done by a computer, to some extent!). This is why some statistician get frustrated with "big data", "machine learning", etc communities muddying the waters about what statistics is (and isn't!) $\endgroup$ – P.Windridge Jun 25 '15 at 13:11
  • 1
  • 1
    $\begingroup$ @P.Windridge: if "all statistical procedures that involve fitting a model can be thought of machine learning" so I don't see why should we distinguish machine learning and statistic $\endgroup$ – Metariat Jun 25 '15 at 15:37
  • 4
    $\begingroup$ @XuanQuangDO We probably shouldn't distinguish machine learning and statistics. $\endgroup$ – Sycorax Aug 17 '15 at 12:36

11 Answers 11

20
$\begingroup$

Machine Learning is not a well defined term.

In fact, if you Google "Machine Learning Definition" the first two things you get are quite different.

From WhatIs.com,

Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data.

From Wikipedia,

Machine learning explores the construction and study of algorithms that can learn from and make predictions on data.

Logistic regression undoubtedly fits the Wikipedia definition and you could argue whether or not it fits the WhatIs defintion.

I personally define Machine Learning just as Wikipedia does and consider it a subset of statistics.

$\endgroup$
  • 1
    $\begingroup$ I agree with most of what you said, except that it is a subset of statistics. It has a large overlap, but there are types of learning, such as reinforcement learning, which can't really be considered to be a subset of statistics. $\endgroup$ – George Oct 16 '15 at 14:04
  • 2
    $\begingroup$ These are not good sources. $\endgroup$ – Neil G Nov 6 '15 at 12:29
  • $\begingroup$ @George Right, but let's face it, if you had to apply a label all data collection, analysis, and modeling methodologies, whether it's machine learning, supervised or unsupervised, parameteric or nonparametric, it's all statistics. ML is a specialized field in statistics. $\endgroup$ – RobertF Jun 29 '16 at 15:24
  • $\begingroup$ @RobertF I disagree. Machine learning is the field that studies how machines can learn. I agree that most methods used in ML can be considered statistical methods, but the field is not inherently a subfield of statistics. For example, I do not think Markov decision processes are considered statistical methods. $\endgroup$ – George Jul 3 '16 at 9:24
  • 1
    $\begingroup$ @George Discrete time Markov models are probability models. Once you estimate unknown parameters of a probability model (e.g. Markov decision processes) that is the textbook definition of a statistical procedure. I think the main class of activities that can be called ML and not statistics are specific applications, like building a robot that plays chess. The underlying algorithms will undoubtedly involve probability and statistics, but the application isn't really "statistics". Kind of like how genomics research uses statistics heavily, but they are decidedly different fields. $\endgroup$ – ahfoss Sep 3 '16 at 1:27
19
$\begingroup$

Machine Learning is hot and is where the money is. People call things they're trying to sell whatever is hot at the moment and therefore "sells". That can be selling software. That can be selling themselves as current employees trying to get promoted, as prospective employees, as consultants, etc. That can be a manager trying to get budget approved from a company bigwig to hire people and buy stuff, or to convince investors to invest in his/her hot new startup which does Machine Learning as the key to making an improved sexting app. So software does Machine Learning and people are Machine Learning experts, because that's what's hot and therefore what sells ... at least for now.

I did all kinds of linear and nonlinear statistical model fitting more than 30 years ago. It wasn't called Machine Learning then. Now, most of it would be.

Just as everyone and their uncle is now a Data "Scientist". That's hot, that's supposedly sexy, so that's what people call themselves. And that's what hiring managers who have to get budget approved to hire someone list positions as. So someone who doesn't know the first thing about math, probability, statistics, optimization, or numerical/floating point computation, uses an R or Python package of dubious correctness and robustness of implementation, and which is labeled as a Machine Learning algorithm, to apply to data they don't understand, and call themselves a Data Scientist based on their experience in doing so.

This may sound flippant, but I believe it to be the essence of the situation.

Edit: The following was tweeted on September 26, 2019:

https://twitter.com/daniela_witten/status/1177294449702928384

Daniela Witten @daniela_witten "When we raise money it’s AI, when we hire it's machine learning, and when we do the work it's logistic regression."

(I'm not sure who came up with this but it's a gem 💎)

$\endgroup$
  • 16
    $\begingroup$ I won't hide that I share some of these opinions and am sympathetic to the rest. However, for them to be appropriate as an answer on an SE site they need to have some kind of support. Obviously that won't be through deductive reasoning: it has to come from adducing facts and/or citing authoritative sources. It would be cool if you could do that! $\endgroup$ – whuber Jun 25 '15 at 18:21
  • 10
    $\begingroup$ Easily the most entertaining post I've read today on this site, and I agree with much of it. But I have to agree with @whuber that it doesn't really answer the question in its current form. $\endgroup$ – Nick Cox Jun 25 '15 at 18:31
  • 5
    $\begingroup$ As a small clarification. I work in both software development and the maligned "Data Science". I interview a lot of people. The rate of people interviewing for software development positions and data science positions who don't have the skills to do the job are about the same. So what's special about the data science title? People are going to inflate their skills in all technical disciplines. I'm sure programming stack exchange has many of the same complaints. $\endgroup$ – Matthew Drury Jun 25 '15 at 19:08
  • 7
    $\begingroup$ This feels more like a rant than an answer. Sure, names change, branding is important and machine learning is hot (and hence has many self-proclaimed practitioners that don't know what they're doing). However, using that as an argument to downplay a field which has become established and highly relevant in both research and industry seems cheap to me. $\endgroup$ – Marc Claesen Jun 25 '15 at 21:12
  • 7
    $\begingroup$ @MarkL.Stone I understand your situation and I completely agree that there are many incompetent insert hot term here's out there. However, in my opinion the fact such people find (and keep!) jobs is the fault of management. If managers are unhappy with the results of analysts, and treat all analysts the same regardless of individual skills/results, then the management is equally incompetent as the bad analysts. Any job that has a scent of cash has quacks, take medicine for instance. Sweeping generalizations about data scientists/machine learning guys are as bad as mistrusting all analysts. $\endgroup$ – Marc Claesen Jun 26 '15 at 6:42
18
$\begingroup$

As others have mentioned already, there's no clear separation between statistics, machine learning, artificial intelligence and so on so take any definition with a grain of salt. Logistic regression is probably more often labeled as statistics rather than machine learning, while neural networks are typically labeled as machine learning (even though neural networks are often just a collection of logistic regression models).

In my opinion, machine learning studies methods that can somehow learn from data, typically by constructing a model in some shape or form. Logistic regression, like SVM, neural networks, random forests and many other techniques, does learn from data when constructing the model.

If I understood correctly, in a Machine Learning algorithm, the model has to learn from its experience

That is not really how machine learning is usually defined. Not all machine learning methods yield models which dynamically adapt to new data (this subfield is called online learning).

What is the difference between logistic regression with the normal regression in term of "learning"?

Many regression methods are also classified as machine learning (e.g. SVM).

$\endgroup$
  • 2
    $\begingroup$ Note that unsupervised learning is still called (machine) learning, so you don't necessarily need to have any feedback loop to classify something as "machine learning". $\endgroup$ – vsz Jun 26 '15 at 6:21
  • $\begingroup$ This isn't on topic for the question, but this answer mentions the separation between AI and ML as well. I always liked this definition of AI: en.wikipedia.org/wiki/… $\endgroup$ – Davis Yoshida Jul 17 '15 at 1:24
9
$\begingroup$

Logistic regression was invented by statistician DR Cox in 1958 and so predates the field of machine learning. Logistic regression is not a classification method, thank goodness. It is a direct probability model.

If you think that an algorithm has to have two phases (initial guess, then "correct" the prediction "errors") consider this: Logistic regression gets it right the first time. That is, in the space of additive (in the logit) models. Logistic regression is a direct competitor of many machine learning methods and outperforms many of them when predictors mainly act additively (or when subject matter knowledge correctly pre-specifies interactions). Some call logistic regression a type of machine learning but most would not. You could call some machine learning methods (neural networks are examples) statistical models.

$\endgroup$
  • 1
    $\begingroup$ Funnily Amazon's machine learning service uses only one algorithm (afaik) - logistic regression - for classification tasks :p aws.amazon.com/machine-learning/faqs $\endgroup$ – stmax Aug 27 '15 at 10:08
  • $\begingroup$ You could just present the data incrementally — as in an online learning problem. In that case, logistic regression doesn't "get it right the first time". I progressively learns. It has a standard loss, and its update is standard application of gradient descent. Logistic regression is in every machine learning text book that I've seen. $\endgroup$ – Neil G Nov 6 '15 at 12:34
  • 1
    $\begingroup$ The fact that you could sample data in an incremental fashion can apply to any estimator even a mean so keep that separate. In a method such as logistic models where the first and second derivatives of the log likelihood function are available analytically you just use the ultra-fast Newton-Raphson method with step-halving to estimate $\beta$ with initial estimates set to zero except for the intercept. $\endgroup$ – Frank Harrell Nov 6 '15 at 13:50
  • $\begingroup$ @FrankHarrell: Right, and that's how maximum likelihood estimation of the solution of a logistic regression problem proceeds. $\endgroup$ – Neil G Nov 6 '15 at 17:18
7
$\begingroup$

I'll have to disagree with most of the answers here and claim that Machine Learning has a very precise scope and a clear cut distinction from Statistics. ML is a sub-field of computer science with a long history, which only in recent years has found applications outside its domain. ML's paternal field and application domain lies within Artificial Intelligence (robotics, pattern recognition software, etc), therefore, it's not just a "hot term" like "Big Data" or "Data Science". Statistics, on the other hand, (which comes from the word "state") was developed within social and economical sciences as a tool for humans, not machines. ML evolved separately from statistics and, eventhough somewhere along the way it started relying heavily on statistical principles, it is by no means a subfield of statistics. ML and statistics are complementary, not overlapping fields.

Long answer:

As implied by its name ML methods were made for software/machines while statistical methods were made for humans. Both ML and statistics deal with predictions on data, however, ML methods follow a non-parametric automatised approach whereas statistical methods require a great deal of manual model-building work with an added explanatory factor. This makes perfect sense if you consider that ML algorithms were developed in AI research as a means of automatised prediction-making that was meant to be integrated in robotics software (e.g. for the purposes of voice and face recognition). When a "machine" makes a prediction, it doesn't care about the reasons behind it. A machine doesn't care to know the drivers/predictors behind a model which classifies email as spam or non-spam, it only cares to have the best accuracy of prediction. This is why virtually all ML methods are black boxes, it's not because they don't have a model, it's because the model is constructed algorithmically and not meant to be visible to neither human nor machine.

The concept of "training" in ML relies on computational power, whereas statistical model-building with OLS-type of methods for parameter estimation relies on the knowledge of a human expert. In a multiple regression scenario it's strictly up to the statistician to use his expert judgement in order to choose his model and verify all required statistical assumptions. A statistician's goal is not just to find patterns and use them for predictions but also to understand his data and his problem in a much greater depth than ML.

Of course in some occasions ML and statistics do overlap, as is the case with many disciplines. Logistic regression is one of these occasions; originally a statistical method, which bears so much resemblance to the simple Perceptron (one of the most fundamental ML techniques), that it is by some seen as a ML method.

$\endgroup$
  • 1
    $\begingroup$ Perhaps you've never heard of nonparametric statistics and nonparametric statistical models and model building? $\endgroup$ – Mark L. Stone Jul 25 '15 at 15:12
  • $\begingroup$ Yes, I use nonparametric stats on a daily basis. I didn't say that ML is the nonparametric answer to statistics, I just find that ML methods being nonparametric comes as a side-effect. Nonparametric statistics is an alternative option of the statistician when parametric statistics fails, but it's still the result of an expert's conscious choice. I'm probably not being clear enough in communicating my view and for that I apologise. $\endgroup$ – Digio Jul 25 '15 at 21:21
  • 3
    $\begingroup$ There are plenty of statisticians who do nonparametric models, statistics all the time. Have you heard of Empirical Likelihood - invented by a statistician, used by statisticians, and quite nonparametric, although it can also be used in a semi-parametric fashion. So I disagree with you, but I did not downvote you. $\endgroup$ – Mark L. Stone Jul 25 '15 at 22:39
  • $\begingroup$ Disagreeing is fine Mark but I still don't quite understand what your counter argument is about. Are you implying that nonparametric statistics has no need of machine learning (something I never denied)? Or are you claiming that machine learning is in fact just another name for nonparametric statistics (something I did deny)? $\endgroup$ – Digio Jul 26 '15 at 7:07
  • 3
    $\begingroup$ There is much to disagree with here. Multivariable regression models, when used in conjunction with modern statistical tools, can be flexible and highly competitive with ML. $\endgroup$ – Frank Harrell Aug 17 '15 at 12:55
3
$\begingroup$

Machine learning is pretty loosely defined and you're correct in thinking that regression models--and not just logistic regression ones--also "learn" from the data. I'm not really sure if this means machine learning is really statistics or statistics is really machine learning--or if any of this matters at all.

However, I don't think it's necessary for an algorithm to repeatedly learn from its mistakes. Most methods use a training set to calculate some parameters and then use these fixed parameters to make predictions on some additional test data. The training process may involve repeatedly updating the parameters (as in backpropagation), but it doesn't necessarily ($k$-nearest neighbours doesn't do anything at all during training!). In any case, at test-time, you may not even have access to ground-truth data.

That said, some algorithms do learn from prediction errors--this is particularly common in reinforcement learning, where an agent takes some action, observes its result, and then uses the outcome to plan future actions. For example, a robotic vacuum might start with a model of the world where it cleans all locations equally often, and then learn to vacuum dirty places (where it is "rewarded" by finding dirt) more and clean places less.

Online or incremental algorithms can be repeatedly updated with new training data. This doesn't necessarily depend on the model's prediction accuracy, but I could imagine an algorithm where the weights are updated more aggressively if, for example, the new data seems very unlikely given the current model. There are online versions for logistic regression: e.g., McMahan and Streeeter (2012).

$\endgroup$
3
$\begingroup$

I finally figured it out. I now know the difference between statistical model fitting and machine learning.

  • If you fit a model (regression), that's statistical model fitting
  • If you learn a model (regression), that's machine learning

So if you learn a logistic regression, that is a machine learning algorithm.

Comment: Pardon me for being an old geezer, but whenever I hear people talking about learning a model, or learning a regression, it makes me think of Jethro "I done learned me an education".

END OF THREAD

$\endgroup$
  • $\begingroup$ ??? I can also learn a logistics model, what're you talking about? $\endgroup$ – SmallChess Dec 23 '15 at 8:44
  • 1
    $\begingroup$ @Student T , if you fit a logistics model, that is statistical model fitting. If you learn a logistics model, that is machine learning.I.e., it's really a matter of the terminology used by the different fields. The same thing can be called different things by different fields (Statistics and Machine Learning). $\endgroup$ – Mark L. Stone Dec 23 '15 at 12:57
-1
$\begingroup$

I think the other answers do a good job at identifying more or less what Machine Learning is (as they indicate, it can be a fuzzy thing). I will add that Logistic Regression (and its more general multinomial version) is very commonly used as a means of performing classification in artificial neural networks (which I think are unambiguously covered by whatever sensible machine learning definition you choose), and so if you mention Logistic Regression to a neural net person, they are likely to immediately think of it in this context. Getting tied up with a heavy hitter in machine learning is a good way to become a machine learning technique yourself, and I think to some extent that is what happened with various regression techniques, though I wouldn't discount them from being proper machine learning techniques in and of themselves.

$\endgroup$
  • $\begingroup$ Note that logistic regression is not a classifier but a direct probability estimation method. $\endgroup$ – Frank Harrell Aug 18 '15 at 12:20
  • $\begingroup$ For further information on Dr. Harrell's point, please see my post here. stats.stackexchange.com/questions/127042/… $\endgroup$ – Sycorax Aug 26 '15 at 13:21
  • $\begingroup$ @FrankHarrell We can also use the probability for classification, so it's really a classifier. $\endgroup$ – SmallChess Dec 23 '15 at 8:45
  • $\begingroup$ @StudentT4 That could not be more incorrect. If is a direct probability estimator. How you use the final result of the logistic model is up to you. By your logic the sample mean is a classifier. $\endgroup$ – Frank Harrell Dec 23 '15 at 13:39
-1
$\begingroup$

Logistic regression (and more generally, GLM) does NOT belong to Machine Learning! Rather, these methods belongs to parametric modeling.

Both parametric and algorithmic (ML) models use the data, but in different ways. Algorithmic models learn from the data how predictors map to the predictand, but they do not make any assumption about the process that has generated the observations (nor any other assumption, actually). They consider that the underlying relationships between input and output variables are complex and unknown, and thus, adopt a data driven approach to understand what's going on, rather than imposing a formal equation.

On the other hand, parametric models are prescribed a priori based on some knowledge of the process studied, use the data to estimate their parameters, and make a lot of unrealistic assumptions that rarely hold in practice (such as the independence, equal variance, and Normal distribution of the errors).

Also, parametric models (like logistic regression) are global models. They cannot capture local patterns in the data (unlike ML methods that use trees as their base models, for instance RF or Boosted Trees). See this paper page 5. As a remediation strategy, local (i.e., nonparametric) GLM can be used (see for instance the locfit R package).

Often, when little knowledge about the underlying phenomenon is available, it is better to adopt a data-driven approach and to use algorithmic modeling. For instance, if you use logistic regression in a case where the interplay between input and output variables is not linear, your model will be clearly inadequate and a lot of signal will not be captured. However, when the process is well understood, parametric models have the advantage of providing a formal equation to summarize everything, which is powerful from a theoretical standpoint.

For a more detailed discussion, read this excellent paper by Leo Breiman.

$\endgroup$
  • 3
    $\begingroup$ Please take the time to understand logistic regression. It makes no distributional assumptions whatsoever. It makes exactly the same kind of independence assumption made by ML. ML requires much larger sample sizes than logistic regression. For example, random forests and SVM can require 200 events per candidate feature to be stable whereas logistic regression typically requires 200 events per candidate variable. $\endgroup$ – Frank Harrell Aug 17 '15 at 12:52
  • 4
    $\begingroup$ There are plenty of good books on the subject and I recommend you consult them before proceeding. Logistic regression does not assume identical distributions and in effect assumes no distribution at all. Unless you can demonstrate how you factor in correlation structure in ML, both approaches assume independence. Regression splines have been used since 1982 to relax linearity assumptions in logistic regression. For this discussion feature=variable unless expanded in a spline. $\endgroup$ – Frank Harrell Aug 17 '15 at 14:42
  • 4
    $\begingroup$ Breiman understood things quite well. He just didn't deal with post 1982 developments in logistic regression, e.g. penalized maximum likelihood estimation, regression splines, and combinations with data reduction methods. The only serious limitation to logistic regression is that like other methods it is not good at finding the right interactions if one searches for interactions and they are not pre-specified. Most methods that purport to be able to do this do not result in replicable findings. Also, Breiman used an improper accuracy score that can be optimzed by a bogus model. $\endgroup$ – Frank Harrell Aug 18 '15 at 12:19
  • 3
    $\begingroup$ @Antoine: "why logistic regression radically differs from ML". Notice that some methods in ML (most noticeably, SVM) are very much related to logistic regression. With the exception of multiple interactions -as Frank wrote- logistic reg with non-linearities and penalization give very similar results to SVM and other ML methods. It continues to amaze me how some papers cite performance improvements based of an ML method vs. a stat101 logistic model to negatively frame logistic regression. $\endgroup$ – Thomas Speidel Aug 18 '15 at 14:25
  • 2
    $\begingroup$ You are confusing distributional assumptions (left hand side of a regression model) with link function and regression assumptions (right hand side of the model). You can use a huge variety of link functions providing the underlying heterogeneous probabilities for a Bernoulli sequence (see the original paper on the logistic model - Cox, 1958). That in no way changes that it is a Bernoulli sequence. And your disagreement over 'outcomes' must stem from my lack of specificity of 'outcomes' vs. 'potential outcomes'. Using new terms for old things, e.g. "surprise" vs. likelihood is suboptimal. $\endgroup$ – Frank Harrell Nov 10 '15 at 12:22
-1
$\begingroup$

I think any procedure which is "iterative" can be considered a case of machine learning. Regression can be considered machine learning. We could do it by hand, but it would take a long time, if at all possible. So now we have these programs, machines, which do the iterations for us. It gets closer and closer to a solution, or to the best solution or best fit. Thus, "machine learning". Of course things like neural networks get most of the attention in regard to machine learning, so we usually associate machine learning to these sexy procedures. Also, the difference between "supervised" and "unsupervised" machine learning is relevant here

$\endgroup$
-2
$\begingroup$

It is a very common mistake that most people do and i can see it here also (done by almost everyone). Let me explain it in detail... Logistic Regression and linear Regression model, both are parametric model as well as Machine Learning Technique. It just depends on the method you are using to estimate the model parameters(theta's). There are 2 ways of finding model parameters in Linear Regression and Logistic reg.

  1. Gradient Descent Technique: Here we starts by assigning random values to the parameters and find cost function(error). In each iteration we update our parameters and minimize cost function. After certain number of iterations, cost function reduced to desired values and corresponding parameters values are our final values. This is what a machine learning techniques supposed to do. So, if You are using Gradient Descent technique, Logistic regression can call as a machine learning technique.

  2. By using Least Square Method: Here we have direct formula to find our parameters (some matrix algebra is required to understand the derivation of this formula) which is known as normal equation. Least Square Method

Here b represents parameters X is design Matrix. Both Methods have their own advantages and limitations. To get more details: follow coursera Machine Learning course still running.

I hope this post might be helpful .. :-)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.