Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Some material I've seen on machine learning said that it's a bad idea to approach a classification problem through regression. But I think it's always possible to do a continuous regression to fit the data and truncate the continuous prediction to yield discrete classifications. So why is it a bad idea?

share|improve this question
    
I voted to migrate to CV; I think it stands a better chance of getting answered there than here. –  J. M. Feb 5 '12 at 6:16
add comment

migrated from math.stackexchange.com Feb 7 '12 at 8:47

This question came from our site for people studying math at any level and professionals in related fields.

2 Answers

"..approach classification problem through regression.." I assume you mean "linear regression" because, say, "logistic regression" algorithm also has the word "regression" in the name but is a pure classification one.

As Andrew Ng explains it, with linear regression you fit a polynomial through the data - say, like on the example below we're fitting a straight line through {tumor size, tumor type} sample set:

enter image description here

Above, malignant tumors get $1$ and non-malignant ones get $0$, and the green line is our hypothesis $h(x)$. To make predictions we may say that for any given tumor size $x$, if $h(x)$ gets bigger than $0.5$ we predict malignant tumor, otherwise we predict benign.

Looks like this way we could correctly predict every single training set sample, but now let's change the task a bit.

Intuitively it's clear that all tumors larger certain threshold are malignant. So let's add another sample with a huge tumor size, and run linear regression again:

enter image description here

Now our $h(x) > 0.5 \rightarrow malignant$ doesn't work anymore. To keep making correct predictions we need to change it to $h(x) > 0.2$ or something - but that not how the algorithm should work.

We cannot change the hypothesis each time a new sample arrives. Instead, we should learn it off the training set data, and then (using the hypothesis we've learned) make correct predictions for the data we haven't seen before.

Hope this explains why linear regression is not the best fit for classification problems! Also, you might want to watch VI. Logistic Regression. Classification video on ml-class.org which explains the idea in more detail.


EDIT

probabilityislogic asked what a good classifier would do. In this particular example you would probably use logistic regression which might learn a hypothesis like this (I'm just making this up):

enter image description here

Note that both linear regression and logistic regression give you a straight line (or a higher order polynomial) but those lines have different meaning:

  • $h(x)$ for linear regression interpolates, or extrapolates, the output and predicts the value for $x$ we haven't seen. It's simply like plugging a new $x$ and getting a raw number, and is more suitable for tasks like predicting, say car price based on {car size, car age} etc.
  • $h(x)$ for logistic regression tells you that all points sitting at the right from the classifier line belong to one class while the points on the left side belongs to the other class. In this case, our $h(x)$ is a raw number which is probability that $x$ belongs to the "positive" class.

So, the bottom line is that in classification scenario we use a completely different reasoning and a completely different algorithm than in regression scenario.

share|improve this answer
    
@andreister: But what would be if all outliers have been removed or truncated, is linear regression still a bad idea? –  Tomek Tarczynski Feb 7 '12 at 9:26
    
Your example is good, however it doesn't show what a "good classifier" would do. would you be able to add this? note that adding data points should change the line for just about any method. You haven't explain why this is a bad change. –  probabilityislogic Feb 7 '12 at 9:40
1  
@andreister: Your example showed some bad data might spoil linear regression. But can we use quadric regression or even more complicated hypothesis to make "regression" a good classifier? –  Strin Feb 7 '12 at 11:53
1  
@probabilityislogic - good point, I updated the answer. –  andreister Feb 7 '12 at 13:17
1  
@Strin: More-complicated hypotheses are more likely to overfit the data. (That is, to fit the quirks of the data you have in hand, resulting in poor fitting on future data.) I remember a class I took where a guy in the front row was just sure that the professor was holding back on us and not giving us the sophisticated algorithms that would let us make a killing in the electricity markets... He never really comprehended overfitting. –  Wayne Feb 7 '12 at 18:23
show 4 more comments

I can't think of an example in which classification is actually the ultimate goal. Almost always the real goal is to make accurate predictions, e.g., of probabilities. In that spirit, (logistic) regression is your friend.

share|improve this answer
1  
It seems to me what is effectively classification is ultimately the goal in most any automated process in which it is impractical or impossible to have human intervention or judgment. When receiving, say, a noisy transmitted digital signal, the receiver cannot decide that a particular bit should be 0.97 instead of 0 or 1. –  cardinal Feb 7 '12 at 17:55
2  
Except for the fact that the cost of a false positive or the cost of a false negative are seldom under the control of the analyst who made the classification, hence the original analyst cannot reliably choose the "right" cutpoint for classification. In addition, it is wise to have a "gray zone" of intermediate risk in which no classification is made and the recommendation is "get more data". –  Frank Harrell Feb 7 '12 at 18:53
    
I think I believe exactly the opposite of the claim in this answer, and never encountered this perspective in my entire university education in machine learning. It's very surprising to me that someone would say this. In practice, I've almost always faced problems where people think they want to predict a continuous quantity, but really they want to predict membership in different categorical buckets of that quantity. I struggle to find instances where actually predicting a continuous quantity is useful in terms of the substantive inference underlying the problem. –  EMS May 29 '13 at 17:17
3  
I think you have taken a good deal of machine learning dogma for granted. You are making a large number of unwarranted assumptions. One of them is that people actually need a forced choice into a categorical bucket. They may claim to want this but they really don't need this in most situations. Choices don't have to be forced. A great choice is "no decision, get more data". Prediction of an underlying continuous quantity is usually what is needed. It would be worth your while to study optimum (Bayes) decision theory. If you can provide a concrete example I'll comment further. –  Frank Harrell May 29 '13 at 20:53
2  
+1 on @Frank Harrell's comment. For example, predicting temperatures, rainfalls, river levels is immensely more helpful than predictions that it will be hot or wet or will flood. Even if the problem is sheep or goat? an estimate of pr(sheep) is more informative than binary "sheep" or "goat". –  Nick Cox Jun 4 '13 at 13:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.