I've been trying to gather intuition on the relationship between methods that seems to be escaping me.

Can someone explain how regression and classification can be performed by the same method, such as SVM (Support Vector Machine) and what this implies about the interchangeability of these two tasks?

Background: I understand how an SVM performs discrimination but I cannot really understand how it could regress an arbitrary non-linear function, though I am told this can be done.

  • $\begingroup$ Please spell out SVM - acronyms can be mysterious, especially to people whose native language is not English. $\endgroup$ – Peter Flom Nov 19 '14 at 11:48
  • $\begingroup$ Support Vector Machines, my mistake, I'll edit it. $\endgroup$ – mrdevlar Nov 19 '14 at 11:54

So as I hate to leave something unanswered that I asked but later found an answer for myself.

Here is a quote from Christopher M. Bishop's "Neural Networks for Pattern Recognition" that seems to perfectly summarize the interchangability of regression and classification:

In the case of regression problems it is the regression function we wish to approximate, while for classification problems the functions which we seek to approximate are the probabilities of membership of the different classes expressed as functions of the input variables.

So effectively, the problems can be seen as functional mappings of one another.

| cite | improve this answer | |
  • 3
    $\begingroup$ Thank you for completing this thread! I get how classification can be seen as a very special form of regression--in effect, a generalized (possibly non-) linear model for discrete (categorical) responses. I don't see how it goes the other way, though. How is regression the same as classification? The prototypical regression example of fitting a linear function to ordered pairs $(x_i,y_i)$ using OLS would not seem to "approximate" any "probabilities of membership" of anything at all. In what way do you understand OLS to solve a classification problem? $\endgroup$ – whuber Jan 21 '15 at 17:33
  • $\begingroup$ Humm, I don't know to be honest. My immediate reaction is that regular regression can be scaled by another function to demonstrate the probability of membership in a group. This is done in GLM applications, where you wrap your regression function with something that then expresses the output as a probability. That said, I'm not sure if that answers your question. Edit: Didn't know enter sends automatically. $\endgroup$ – mrdevlar Jan 23 '15 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.