Why is logistic regression called a machine learning algorithm?

Question

If I understood correctly, in a machine learning algorithm, the model has to learn from its experience, i.e when the model gives the wrong prediction for the new cases, it must adapt to the new observations, and in time, the model becomes increasingly better. I don't see that the logistic regression has this characteristic. So why is it still regarded as a machine learning algorithm? What is the difference between logistic regression with the normal regression in term of "learning"?

I have the same question for random forests!

And what is the definition of "machine learning"?

Answer 1

Machine Learning is not a well defined term.

In fact, if you Google "Machine Learning Definition" the first two things you get are quite different.

From WhatIs.com,

Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data.

From Wikipedia,

Machine learning explores the construction and study of algorithms that can learn from and make predictions on data.

Logistic regression undoubtedly fits the Wikipedia definition and you could argue whether or not it fits the WhatIs defintion.

I personally define Machine Learning just as Wikipedia does and consider it a subset of statistics.

Answer 2

Machine Learning is hot and is where the money is. People call things they're trying to sell whatever is hot at the moment and therefore "sells". That can be selling software. That can be selling themselves as current employees trying to get promoted, as prospective employees, as consultants, etc. That can be a manager trying to get budget approved from a company bigwig to hire people and buy stuff, or to convince investors to invest in his/her hot new startup which does Machine Learning as the key to making an improved sexting app. So software does Machine Learning and people are Machine Learning experts, because that's what's hot and therefore what sells ... at least for now.

I did all kinds of linear and nonlinear statistical model fitting more than 30 years ago. It wasn't called Machine Learning then. Now, most of it would be.

Just as everyone and their uncle is now a Data "Scientist". That's hot, that's supposedly sexy, so that's what people call themselves. And that's what hiring managers who have to get budget approved to hire someone list positions as. So someone who doesn't know the first thing about math, probability, statistics, optimization, or numerical/floating point computation, uses an R or Python package of dubious correctness and robustness of implementation, and which is labeled as a Machine Learning algorithm, to apply to data they don't understand, and call themselves a Data Scientist based on their experience in doing so.

This may sound flippant, but I believe it to be the essence of the situation.

Edit: The following was tweeted on September 26, 2019:

https://twitter.com/daniela_witten/status/1177294449702928384

Daniela Witten @daniela_witten "When we raise money it’s AI, when we hire it's machine learning, and when we do the work it's logistic regression."

(I'm not sure who came up with this but it's a gem 💎)

Answer 3

As others have mentioned already, there's no clear separation between statistics, machine learning, artificial intelligence and so on so take any definition with a grain of salt. Logistic regression is probably more often labeled as statistics rather than machine learning, while neural networks are typically labeled as machine learning (even though neural networks are often just a collection of logistic regression models).

In my opinion, machine learning studies methods that can somehow learn from data, typically by constructing a model in some shape or form. Logistic regression, like SVM, neural networks, random forests and many other techniques, does learn from data when constructing the model.

If I understood correctly, in a Machine Learning algorithm, the model has to learn from its experience

That is not really how machine learning is usually defined. Not all machine learning methods yield models which dynamically adapt to new data (this subfield is called online learning).

What is the difference between logistic regression with the normal regression in term of "learning"?

Many regression methods are also classified as machine learning (e.g. SVM).

Answer 4

Logistic regression was invented by statistician DR Cox in 1958 and so predates the field of machine learning. Logistic regression is not a classification method, thank goodness. It is a direct probability model.

If you think that an algorithm has to have two phases (initial guess, then "correct" the prediction "errors") consider this: Logistic regression gets it right the first time. That is, in the space of additive (in the logit) models. Logistic regression is a direct competitor of many machine learning methods and outperforms many of them when predictors mainly act additively (or when subject matter knowledge correctly pre-specifies interactions). Some call logistic regression a type of machine learning but most would not. You could call some machine learning methods (neural networks are examples) statistical models.

Answer 5

I'll have to disagree with most of the answers here and claim that Machine Learning has a very precise scope and a clear cut distinction from Statistics. ML is a sub-field of computer science with a long history, which only in recent years has found applications outside its domain. ML's paternal field and application domain lies within Artificial Intelligence (robotics, pattern recognition software, etc), therefore, it's not just a "hot term" like "Big Data" or "Data Science". Statistics, on the other hand, (which comes from the word "state") was developed within social and economical sciences as a tool for humans, not machines. ML evolved separately from statistics and, eventhough somewhere along the way it started relying heavily on statistical principles, it is by no means a subfield of statistics. ML and statistics are complementary, not overlapping fields.

Long answer:

As implied by its name ML methods were made for software/machines while statistical methods were made for humans. Both ML and statistics deal with predictions on data, however, ML methods follow a non-parametric automatised approach whereas statistical methods require a great deal of manual model-building work with an added explanatory factor. This makes perfect sense if you consider that ML algorithms were developed in AI research as a means of automatised prediction-making that was meant to be integrated in robotics software (e.g. for the purposes of voice and face recognition). When a "machine" makes a prediction, it doesn't care about the reasons behind it. A machine doesn't care to know the drivers/predictors behind a model which classifies email as spam or non-spam, it only cares to have the best accuracy of prediction. This is why virtually all ML methods are black boxes, it's not because they don't have a model, it's because the model is constructed algorithmically and not meant to be visible to neither human nor machine.

The concept of "training" in ML relies on computational power, whereas statistical model-building with OLS-type of methods for parameter estimation relies on the knowledge of a human expert. In a multiple regression scenario it's strictly up to the statistician to use his expert judgement in order to choose his model and verify all required statistical assumptions. A statistician's goal is not just to find patterns and use them for predictions but also to understand his data and his problem in a much greater depth than ML.

Of course in some occasions ML and statistics do overlap, as is the case with many disciplines. Logistic regression is one of these occasions; originally a statistical method, which bears so much resemblance to the simple Perceptron (one of the most fundamental ML techniques), that it is by some seen as a ML method.

Answer 6

I finally figured it out. I now know the difference between statistical model fitting and machine learning.

• If you fit a model (regression), that's statistical model fitting
• If you learn a model (regression), that's machine learning

So if you learn a logistic regression, that is a machine learning algorithm.

Comment: Pardon me for being an old geezer, but whenever I hear people talking about learning a model, or learning a regression, it makes me think of Jethro "I done learned me an education".

END OF THREAD

Answer 7

Machine learning is pretty loosely defined and you're correct in thinking that regression models--and not just logistic regression ones--also "learn" from the data. I'm not really sure if this means machine learning is really statistics or statistics is really machine learning--or if any of this matters at all.

However, I don't think it's necessary for an algorithm to repeatedly learn from its mistakes. Most methods use a training set to calculate some parameters and then use these fixed parameters to make predictions on some additional test data. The training process may involve repeatedly updating the parameters (as in backpropagation), but it doesn't necessarily ($k$-nearest neighbours doesn't do anything at all during training!). In any case, at test-time, you may not even have access to ground-truth data.

That said, some algorithms do learn from prediction errors--this is particularly common in reinforcement learning, where an agent takes some action, observes its result, and then uses the outcome to plan future actions. For example, a robotic vacuum might start with a model of the world where it cleans all locations equally often, and then learn to vacuum dirty places (where it is "rewarded" by finding dirt) more and clean places less.

Online or incremental algorithms can be repeatedly updated with new training data. This doesn't necessarily depend on the model's prediction accuracy, but I could imagine an algorithm where the weights are updated more aggressively if, for example, the new data seems very unlikely given the current model. There are online versions for logistic regression: e.g., McMahan and Streeeter (2012).

Answer 8

Logistic regression (and more generally, GLM) does NOT belong to Machine Learning! Rather, these methods belongs to parametric modeling.

Both parametric and algorithmic (ML) models use the data, but in different ways. Algorithmic models learn from the data how predictors map to the predictand, but they do not make any assumption about the process that has generated the observations (nor any other assumption, actually). They consider that the underlying relationships between input and output variables are complex and unknown, and thus, adopt a data driven approach to understand what's going on, rather than imposing a formal equation.

On the other hand, parametric models are prescribed a priori based on some knowledge of the process studied, use the data to estimate their parameters, and make a lot of unrealistic assumptions that rarely hold in practice (such as the independence, equal variance, and Normal distribution of the errors).

Also, parametric models (like logistic regression) are global models. They cannot capture local patterns in the data (unlike ML methods that use trees as their base models, for instance RF or Boosted Trees). See this paper page 5. As a remediation strategy, local (i.e., nonparametric) GLM can be used (see for instance the locfit R package).

Often, when little knowledge about the underlying phenomenon is available, it is better to adopt a data-driven approach and to use algorithmic modeling. For instance, if you use logistic regression in a case where the interplay between input and output variables is not linear, your model will be clearly inadequate and a lot of signal will not be captured. However, when the process is well understood, parametric models have the advantage of providing a formal equation to summarize everything, which is powerful from a theoretical standpoint.

For a more detailed discussion, read this excellent paper by Leo Breiman.

Answer 9

I think the other answers do a good job at identifying more or less what Machine Learning is (as they indicate, it can be a fuzzy thing). I will add that Logistic Regression (and its more general multinomial version) is very commonly used as a means of performing classification in artificial neural networks (which I think are unambiguously covered by whatever sensible machine learning definition you choose), and so if you mention Logistic Regression to a neural net person, they are likely to immediately think of it in this context. Getting tied up with a heavy hitter in machine learning is a good way to become a machine learning technique yourself, and I think to some extent that is what happened with various regression techniques, though I wouldn't discount them from being proper machine learning techniques in and of themselves.

Answer 10

I think any procedure which is "iterative" can be considered a case of machine learning. Regression can be considered machine learning. We could do it by hand, but it would take a long time, if at all possible. So now we have these programs, machines, which do the iterations for us. It gets closer and closer to a solution, or to the best solution or best fit. Thus, "machine learning". Of course things like neural networks get most of the attention in regard to machine learning, so we usually associate machine learning to these sexy procedures. Also, the difference between "supervised" and "unsupervised" machine learning is relevant here

Answer 11

It is a very common mistake that most people do and i can see it here also (done by almost everyone). Let me explain it in detail... Logistic Regression and linear Regression model, both are parametric model as well as Machine Learning Technique. It just depends on the method you are using to estimate the model parameters(theta's). There are 2 ways of finding model parameters in Linear Regression and Logistic reg.

1. Gradient Descent Technique: Here we starts by assigning random values to the parameters and find cost function(error). In each iteration we update our parameters and minimize cost function. After certain number of iterations, cost function reduced to desired values and corresponding parameters values are our final values. This is what a machine learning techniques supposed to do. So, if You are using Gradient Descent technique, Logistic regression can call as a machine learning technique.

2. By using Least Square Method: Here we have direct formula to find our parameters (some matrix algebra is required to understand the derivation of this formula) which is known as normal equation.

Here b represents parameters X is design Matrix. Both Methods have their own advantages and limitations. To get more details: follow coursera Machine Learning course still running.

I hope this post might be helpful .. :-)