13
$\begingroup$

Say I want to predict whether or not a project will be profitable. In my sample data, the response variable is actually a continuous variable: the $ profit/loss of the project.

Because my ultimate goal is just a binary classification (profitable project or unprofitable project), should I use a classification technique? Or should I use regression so that I'm not throwing away the additional information a continuous response variable provides?

$\endgroup$

5 Answers 5

11
$\begingroup$

I can't think of an example where I would recommend a classification technique when the variable is continuous or ordinal. After efficiently fitting a continuous model you can use that model to estimate the probability that $Y$ exceeds any level of interest. If the model is Gaussian this probability is a function of the predicted mean and the residual standard deviation.

$\endgroup$
1
  • 4
    $\begingroup$ One example (as noted below) is if the cost of incorrect prediction is the same for all levels of profitability. Ie. when you have a continuous variable, but you are truly only interested in the discrete values. A spline with n knots regressed to the continuous variable may put many knots on the extreme values to accurately mode the shape of the data there, whereas a spline optimized for classification can put all its knots around 0. $\endgroup$ Feb 19, 2013 at 13:25
11
$\begingroup$

Vladimir Vapnik (co-inventor of the Support Vector Machine and leading computational learning theorist) advocates always trying to solve the problem directly, rather than solving some more general problem and then discarding some of the information provided by the solution. I am generally in agreement with this, so I would suggest a classification approach for the problem as currently posed. The reason for this is that if we are only interested in classifying a project as profitable or non-profitable, then we are really only interested in the region where profitability is around zero. If we form a classification model, that is where we will be concentrating our modelling resources. If we take a regression approach, we may be wasting modelling resources to make small improvements in performance for projects that will either be very profitable or unprofitable, potentially at the expense of improving performance of borderline projects.

Now the reason that I said "as currently posed", is that very few problems actually do involve simple, hard binary classification (optical character recognition would probably be one). Generally different kinds of misclassification have different costs, or operational class frequencies may be unknown, or variable etc. In such cases it is better to have a probabilistic classifier, such as logistic regression, rather than an SVM. If seems to me that for a financial application, we will do better if we know the probability of whether the project will be profitable, and how profitable or otherwise it is likely to be. We may well be willing to fund a project that has a small chance of being profitable, but massively profitable should it succeed, but not a project that is almost guaranteed to be successful, but which will have such a small profit margin that we would be better off just sticking the money in a savings account.

So Frank and Omri374 are both right! (+1 ;o)

EDIT: To clarify why regression might not always be a good approach for solving a classification problem, here is an example. Say we have three projects, with profitability $\vec{y} = (-\$1000,+\$1, +\$1000)$, and for each project, we have an explanatory variable that we hope is indicative of profitability, $\vec{x} = (1, 2, 10)$. If we take a regression approach (with offset), we get regression coefficients $\beta_0 = -800.8288$ and $\beta_1 = 184.8836$ (provided I have done the sums correctly!). The model then predicts the projects as yielding profits $\hat{y}_1 \approx -\$616$, $\hat{y}_2 \approx -\$431$ and $\hat{y}_3 \approx \$1048$. Note that the second project is incorrectly predicted as being unprofitable. If on the other hand, we take a classification approach, and regress instead on $\vec{t} = 2*(y >= 0) - 1$, we get regression coefficients $\beta_0 = -0.2603$ and $\beta_1 = 0.1370$, which scores the three projects as follows: $\hat{t}_1 = -0.1233$, $\hat{t}_2 = 0.0137$ and $\hat{t}_3 = 1.1096$. So a classification approach correctly classifies project 1 as unprofitable and the other two as being profitable.

The reason why this happens is that a regression approach tries equally hard to minimise the sum of squared errors for each of the data points. In this case, a lower SSE is obtained by allowing project two to fall on the incorrect side of the decision boundary, in order to achieve lower errors on the other two points.

So Frank is correct in saying that a regression approach is likely to be a good approach in practice, but if classification actually is the ultimate aim, there are situations where it can perform poorly and a classification approach will perform better.

$\endgroup$
7
  • $\begingroup$ I can't follow that logic. A classification model weakens the relationships in the data, and a continuous model does not require more modeling resources. A continuous model takes into account that a profit of \$1 (though "profitable") is much different than a profit of \$1B. A classification model is a forced choice where "no choice" (gray zone) is not an option. And the statistical inefficiency of binary models over continuous ones is striking. You can always use a continuous model to estimate Prob$[Y > 0 | X]$ when finished. $\endgroup$ Feb 19, 2013 at 13:52
  • $\begingroup$ Whether a continuous model requires more resources than a classification model depends on what type of model it is (e.g. a neural network model could use its hidden units to model features near the borderline or it could use them to improve the fit away from the borderline). The same is true to a lesser extent of the weights of a linear model, where the fitted values may be dominated by high leverage points that are nowhere near the borderline, which might be a bad thing if simple classification actually was what is important. $\endgroup$ Feb 19, 2013 at 14:08
  • $\begingroup$ Your second point about profitability appears to be essentially why I am describing in my second paragraph (the real problem probably isn't actually a simple hard classification), which is why I said both you and omri374 were correct. $\endgroup$ Feb 19, 2013 at 14:09
  • $\begingroup$ "Near the borderline" is unknown to a classifier that is not provided the continuous $Y$ values. $\endgroup$ Feb 19, 2013 at 19:26
  • $\begingroup$ Classifier systems have been used to locate the decision boundary using discrete labels for a long time. You are missing the point, I am actually mostly in agreement with what you have written, with the caveat that the model can be biased by high leverage points that are not near the decision boundary, which can reduce performance if classification actually is the aim (which is relatively rare in practice). I have seen this phenomenon in my applied work over the years, but I still often use regression models to solve classification problems myself. Ask Prof. Vapnik. $\endgroup$ Feb 20, 2013 at 9:00
3
$\begingroup$

After reading the comments, I think the following distinction is what's missing from the discussion:

How to model the problem

This has nothing to do with what technique to use. It's the question of what the inputs and outputs are and how to evaluate the result.

If you truly only care about whether or not our projects are profitable, and the amount by which they are so is absolutely irrelevant, then you should model this as a classification problem. That means you are ultimately optimizing for the expected rate of correct classification (accuracy) or AUC. What this optimization translates to depends on what technique you use.

All questions of model choice and search algorithms can be approached heuristically (using the arguments that have been put forth in the other answers and comments) but the ultimate proof of the pudding is in the eating. Whatever model you have, you will evaluate by cross validated tests for accuracy, so accuracy is what you optimize.

How to solve the problem

You can use any method you like that fits the classification paradigm. Since you have a continuous y variable, you can do regression on that, and translate to a binary classification. This will likely work well. However, there is no guarantee that the optimal regression model (by sum of squared errors or maximum likelihood or whatever) will also give you the optimal classification model (by accuracy or AUC).

$\endgroup$
6
  • $\begingroup$ +1 as a general maxim, I would suggest that the first task is to have a clear idea of the problem to be solved, and the second is to approach the problem with the method that gives the most direct answer to the question that is actually being asked. This is a slight generalisation of Vapnik's maxim, but only very slight! $\endgroup$ Feb 20, 2013 at 14:42
  • $\begingroup$ Phrasing the problem like that still does not imply that dichotomizing $Y$ is the right approach. As I said elsewhere you can compute the probability that $Y>0$ given optimum regression coefficient estimates from a continuous model. $\endgroup$ Feb 20, 2013 at 19:12
  • $\begingroup$ Note that I'm not saying that you should necessarily throw away or ignore the continuous y values. But there's a difference between using them in a classifier and optimizing for regression accuracy (you model the problem as classification, but you solve it with regression). It may well be that your best solution is a regression method, but you should prove this by evaluating it as a classifier. And there are situations where throwing the continuous values away and only using the discretized values, will give you better performance. $\endgroup$ Feb 20, 2013 at 22:31
  • $\begingroup$ Evaluating it as a classifier implies that your utility function is discontinuous which does not seem realistic to me. It also implies that binary decisions are forced, i.e., there are no category of "no decisions, get more data". I've created examples where classification accuracy goes down after adding a highly important variable to the model. The problem is not with the variable; it is with the accuracy measure. $\endgroup$ Feb 20, 2013 at 23:37
  • $\begingroup$ While it is true that you can determine the decision boundary if you have the probability that $Y >0$, the problem is that estimating this probability is a more difficult estimation problem than simply estimating the decision boundary. As we generally have a finite amount of data, the additional difficulty of estimation means that the dichotomizing approach works better in practice. This is the idea underpinning the SVM, which has proved its worth in a wide variety of classification problems. $\endgroup$ Feb 21, 2013 at 9:50
1
$\begingroup$

A classification model generally attempts to minimize the sign (error in terms of class) and not the bias. In case of many outliers, for example, I would prefer using a classification model and not a regression model.

$\endgroup$
1
  • $\begingroup$ That doesn't follow, and will be terribly inefficient. You can use a robust continuous model including a semiparametric model such as the proportional odds model. $\endgroup$ Feb 19, 2013 at 19:26
0
$\begingroup$

I would frame the problem as that of minimizing loss. The question is what is your true loss function? Does an incorrect prediction of profitable when the project lost \$1 cost as much as a prediction of profitable when the project lost \$1000? In that case your loss function is truly binary, and you're better of casting the whole thing as a classification problem. The regression function may still be one of your candidate classifiers, but but you should optimize it with the discrete loss function rather than the continuous one. If you have a more complicated definition of loss, then you should try to formalize it, and see what you get if you take the derivative.

Interestingly, many machine learning methods actually optimize a discrete loss function by approximating with a continuous one, since a discrete loss function provides poor gradients for optimization. So you may end up casting it as a classification problem, since that's your loss function, but then approximating that loss function with the original continuous one.

$\endgroup$
5
  • $\begingroup$ That doesn't tell the whole story. With Gaussian regression the maximum likelihood estimator of Prob$[Y > 0 | X]$ is a function of the predicted mean and residual variance. $\endgroup$ Feb 19, 2013 at 19:27
  • $\begingroup$ That's interesting. But ML is by no means the ultimate goal, that's either accuracy or AUC. If you are optimizing the likelihood (or SSE), you may end up "spending model complexity" on modeling data artifacts that don't matter. An equivalent model can actually reduce the accuracy of its modeling to focus on improving classification accuracy. $\endgroup$ Feb 20, 2013 at 14:15
  • $\begingroup$ It depends on what you mean by "accuracy", and AUC is seldom an appropriate quantity to optimize due to its implied loss function. You needn't spend model complexity on artifacts if doing continuous modeling correctly. Proportion classified correct is an improper scoring rule that is optimized by a bogus model. If 0.99 of companies are profitable in a good year, you would be 0.99 correct by ignoring all $X$ data and just classifying all companies as profitable. Using valuable predictors (in any sense other than classification) may make the classification accuracy actually decrease. $\endgroup$ Feb 20, 2013 at 19:17
  • $\begingroup$ I agree with your misgivings about AUC. By accuracy, I mean the proportion classified correctly. I agree that it is unlikely that the poster is truly only interested in the binary variables, and I suspect that actually the amount of profit made plays some part. But if the discrete classification is really the only concern, then I don't see anything else to optimize but a classification measure. And if your classes are that strongly biased towards the profitable class then ignoring the data and always classifying as profitable will indeed be a hard baseline to beat. $\endgroup$ Feb 20, 2013 at 22:37
  • $\begingroup$ Proportion classified correctly performs even worse than AUC. It was shown in the German decision making literature in the 1970s that classification accuracy is an improper scoring rule. If discrete classification is your concern, that can be obtained at the last second. Bayes optimum decisions use full conditioning on all available information. $\endgroup$ Feb 20, 2013 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.