47
$\begingroup$

Some material I've seen on machine learning said that it's a bad idea to approach a classification problem through regression. But I think it's always possible to do a continuous regression to fit the data and truncate the continuous prediction to yield discrete classifications. So why is it a bad idea?

$\endgroup$

migrated from math.stackexchange.com Feb 7 '12 at 8:47

This question came from our site for people studying math at any level and professionals in related fields.

  • $\begingroup$ i think regression is always more complicated than classification in production environment $\endgroup$ – user78382 May 29 '15 at 8:04
53
$\begingroup$

"..approach classification problem through regression.." by "regression" I will assume you mean linear regression, and I will compare this approach to the "classification" approach of fitting a logistic regression model.

Before we do this, it is important to clarify the distinction between regression and classification models. Regression models predict a continuous variable, such as rainfall amount or sunlight intensity. They can also predict probabilities, such as the probability that an image contains a cat. A probability-predicting regression model can be used as part of a classifier by imposing a decision rule - for example, if the probability is 50% or more, decide it's a cat.

Logistic regression predicts probabilities, and is therefore a regression algorithm. However, it is commonly described as a classification method in the machine learning literature, because it can be (and is often) used to make classifiers. There are also "true" classification algorithms, such as SVM, which only predict an outcome and do not provide a probability. We won't discuss this kind of algorithm here.

Linear vs. Logistic Regression on Classification Problems

As Andrew Ng explains it, with linear regression you fit a polynomial through the data - say, like on the example below we're fitting a straight line through {tumor size, tumor type} sample set:

enter image description here

Above, malignant tumors get $1$ and non-malignant ones get $0$, and the green line is our hypothesis $h(x)$. To make predictions we may say that for any given tumor size $x$, if $h(x)$ gets bigger than $0.5$ we predict malignant tumor, otherwise we predict benign.

Looks like this way we could correctly predict every single training set sample, but now let's change the task a bit.

Intuitively it's clear that all tumors larger certain threshold are malignant. So let's add another sample with a huge tumor size, and run linear regression again:

enter image description here

Now our $h(x) > 0.5 \rightarrow malignant$ doesn't work anymore. To keep making correct predictions we need to change it to $h(x) > 0.2$ or something - but that not how the algorithm should work.

We cannot change the hypothesis each time a new sample arrives. Instead, we should learn it off the training set data, and then (using the hypothesis we've learned) make correct predictions for the data we haven't seen before.

Hope this explains why linear regression is not the best fit for classification problems! Also, you might want to watch VI. Logistic Regression. Classification video on ml-class.org which explains the idea in more detail.


EDIT

probabilityislogic asked what a good classifier would do. In this particular example you would probably use logistic regression which might learn a hypothesis like this (I'm just making this up):

enter image description here

Note that both linear regression and logistic regression give you a straight line (or a higher order polynomial) but those lines have different meaning:

  • $h(x)$ for linear regression interpolates, or extrapolates, the output and predicts the value for $x$ we haven't seen. It's simply like plugging a new $x$ and getting a raw number, and is more suitable for tasks like predicting, say car price based on {car size, car age} etc.
  • $h(x)$ for logistic regression tells you the probability that $x$ belongs to the "positive" class. This is why it is called a regression algorithm - it estimates a continuous quantity, the probability. However, if you set a threshold on the probability, such as $h(x) > 0.5$, you obtain a classifier, and in many cases this is what is done with the output from a logistic regression model. This is equivalent to putting a line on the plot: all points sitting above the classifier line belong to one class while the points below belong to the other class.

So, the bottom line is that in classification scenario we use a completely different reasoning and a completely different algorithm than in regression scenario.

$\endgroup$
  • $\begingroup$ @andreister: But what would be if all outliers have been removed or truncated, is linear regression still a bad idea? $\endgroup$ – Tomek Tarczynski Feb 7 '12 at 9:26
  • $\begingroup$ Your example is good, however it doesn't show what a "good classifier" would do. would you be able to add this? note that adding data points should change the line for just about any method. You haven't explain why this is a bad change. $\endgroup$ – probabilityislogic Feb 7 '12 at 9:40
  • 1
    $\begingroup$ @andreister: Your example showed some bad data might spoil linear regression. But can we use quadric regression or even more complicated hypothesis to make "regression" a good classifier? $\endgroup$ – Strin Feb 7 '12 at 11:53
  • 1
    $\begingroup$ @probabilityislogic - good point, I updated the answer. $\endgroup$ – andreister Feb 7 '12 at 13:17
  • 2
    $\begingroup$ @Strin: More-complicated hypotheses are more likely to overfit the data. (That is, to fit the quirks of the data you have in hand, resulting in poor fitting on future data.) I remember a class I took where a guy in the front row was just sure that the professor was holding back on us and not giving us the sophisticated algorithms that would let us make a killing in the electricity markets... He never really comprehended overfitting. $\endgroup$ – Wayne Feb 7 '12 at 18:23
14
$\begingroup$

I can't think of an example in which classification is actually the ultimate goal. Almost always the real goal is to make accurate predictions, e.g., of probabilities. In that spirit, (logistic) regression is your friend.

$\endgroup$
  • 5
    $\begingroup$ It seems to me what is effectively classification is ultimately the goal in most any automated process in which it is impractical or impossible to have human intervention or judgment. When receiving, say, a noisy transmitted digital signal, the receiver cannot decide that a particular bit should be 0.97 instead of 0 or 1. $\endgroup$ – cardinal Feb 7 '12 at 17:55
  • 6
    $\begingroup$ Except for the fact that the cost of a false positive or the cost of a false negative are seldom under the control of the analyst who made the classification, hence the original analyst cannot reliably choose the "right" cutpoint for classification. In addition, it is wise to have a "gray zone" of intermediate risk in which no classification is made and the recommendation is "get more data". $\endgroup$ – Frank Harrell Feb 7 '12 at 18:53
  • 1
    $\begingroup$ I think I believe exactly the opposite of the claim in this answer, and never encountered this perspective in my entire university education in machine learning. It's very surprising to me that someone would say this. In practice, I've almost always faced problems where people think they want to predict a continuous quantity, but really they want to predict membership in different categorical buckets of that quantity. I struggle to find instances where actually predicting a continuous quantity is useful in terms of the substantive inference underlying the problem. $\endgroup$ – ely May 29 '13 at 17:17
  • 9
    $\begingroup$ I think you have taken a good deal of machine learning dogma for granted. You are making a large number of unwarranted assumptions. One of them is that people actually need a forced choice into a categorical bucket. They may claim to want this but they really don't need this in most situations. Choices don't have to be forced. A great choice is "no decision, get more data". Prediction of an underlying continuous quantity is usually what is needed. It would be worth your while to study optimum (Bayes) decision theory. If you can provide a concrete example I'll comment further. $\endgroup$ – Frank Harrell May 29 '13 at 20:53
  • 11
    $\begingroup$ +1 on @Frank Harrell's comment. For example, predicting temperatures, rainfalls, river levels is immensely more helpful than predictions that it will be hot or wet or will flood. Even if the problem is sheep or goat? an estimate of pr(sheep) is more informative than binary "sheep" or "goat". $\endgroup$ – Nick Cox Jun 4 '13 at 13:29
1
$\begingroup$

Why not look at some evidence? Although many would argue that linear regression is not right for classification, it may still work. To gain some intuition, I included linear regression (used as classifier) into scikit-learn's classifier comparison. Here is what happens:

enter image description here

The decision boundary is narrower than with the other classifiers, but the accuracy is the same. Much like the linear support vector classifier, the regression model gives you a hyperplane that separates the classes in feature space.

As we see, using linear regression as classifier can work, but as always, I would cross validate the predictions.

For the record, this is how my classifier code looks like:

class LinearRegressionClassifier():

def __init__(self):
    self.reg = LinearRegression()

def fit(self, X, y):
    self.reg.fit(X, y)

def predict(self, X):
    return np.clip(self.reg.predict(X),0,1)

def decision_function(self, X):
    return np.clip(self.reg.predict(X),0,1)

def score(self, X, y):
    return accuracy_score(y,np.round(self.predict(X)))
$\endgroup$
0
$\begingroup$

Further, to expand on already good answers, for any classification task beyond a bivariate one, using the regression would require us to impose a distance and ordering between the classes. In other words, we might get different results just by shuffling the labels of the classes or changing the scale of assigned numeric values (say classes labeled as $1, 10, 100, ...$ vs $1, 2, 3, ...$), which defeats the purpose of the classification problem.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.