Regression vs. Classification: Is there a clear, generally accepted definition?

Question

As a mathematician/economist, I am not trained to think in classification and regression tasks. This is why I wonder: is there a clear, widely accepted definition of regression and classification problems?

E.g., this paper says that When $Y$ has a finite number of states we refer to the task as classification. Otherwise we refer to the task as regression.

Does this mean that count data with many counts are a classification problem (e.g., Poisson regression). Or if we model life expectancy, is this generally supposed to be a classification task?

Is this definition generally accepted?

Answer 1

Classification denotes an action. It's what you do with the result of an analysis in which there is one or more outcome variables and one or more input (predictor; covariate) variables. If there is a single outcome variable, the discreteness of the variable does not matter. For example, binary logistic regression is for binary Y and is a direct (continuous) probability model that was not intended to be used for classification. The action of classification involves making choices and use of decision rules. In most cases it represents a premature decision made by an analyst who is not blessed with knowledge about the consequence of the decision (i.e., does not possess the utility/loss/cost function needed to make a good decision).

One can use any predictive method to do classification even if that was not the intent of the method. For example one can use arbitrary thresholds on predicted values to do classification from ordinary regression for continuous Y, or ordinal or binary regression for ordered or binary Y.

Many in machine learning think of classification as a good default mode; it is not, as detailed in my blog post. Among other things, classification hides close calls and lulls users into making decisions at the boundaries (e.g., when a predicted probability is 0.5001) when a better approach would be "get more data first".

Most of the time when you see classifier used in a sentence the correct term is prediction when the output is considered to be continuous.

Answer 2

No, I don't think that definition is generally accepted. I would not regard Poisson regression as classification as the thing you are generally interested is the conditional values of a Poisson distribution that describes the distribution of the target variable for those values of the attributes. Those parameters are generally continuous. You might then use that to work out the most likely count, but that would be discretising the predictive distribution given by the model.

Likewise some here (e.g. Frank Harrell - see his answer to this question +1) view logistic regression purely as a probabilistic model, used to estimate a conditional probability, and not as a classification model (which is what you get by applying a threshold and discretising the continuous output of the model). I have a lot of sympathy with this view, except that in practical applications where you need to perform that discretisation, that still impacts on the design and evaluation of the model and shouldn't be ignored. The optimal classification is not always obtained by estimating the probability of class membership and thresholding, sometimes it is better to classify the data directly. If that were not the case, [kernel] logistic regression would not perform worse than the Support Vector Machine, but on some applications it clearly does.

I'd probably say that a classifier is a problem where the target distribution is categorical, and the aim to to place each object into a category.

Answer 3

A reasonable standard definition of classification would be that the $Y$ value is of nominal scale level, i.e., that order and numerical differences are not meaningful (or at least not of interest). Models with count or ordinal responses are widely referred to as regression, e.g., Poisson regression, ordinal regression.

Note that the comment by Dikran Marsupial is correct that some models with nominal scaled outcome are also referred to as regression ((multinomial) logistic regression). My interpretation of this would be that the term regression is also used to refer to a class of models that grew out of historically quantitative regression models. I think it makes still sense (and would probably be standard handling of the term) to also refer to these as classification methods, and certainly to the problems solved by them as classification problems. This handling of terms would have regression and classification as not necessarily mutually exclusive. Probably regression is more ambiguous than classification, as it can be used for methods for quantitative or at least ordinal responses (and insofar be distinct from classification), but also for certain model classes that may also have versions for classification problems.

Answer 4

To muddy the waters further, classification can mean

1. trying to find distinct classes in a dataset from scratch, which has attracted many different names, including mathematical or numerical taxonomy, but cluster analysis seems the most durable and popular

2. assigning observations to classes already defined, which has other names too including identification and discrimination.

Answer 5

Definitions are overrated. They are just words which someone claims are synonymous to other words. They may be useful in special situations, e.g. to help a novice get a grasp of a concept, or to ensure that experts, when communicating with each other, know precisely what they are talking about. But, standing alone, who cares?

That said, I believe all machine learners and a good deal of statisticians wouldn't object and would understand what you mean if you say "classification" for a task where your dependent variable is nominal (synonym: categorical) in Stevens' typology. Most machine learners will associate regression with numerical dependent variables, but some statisticians would say that "everything is regression", and classification a step---obviously outside of "everything"---you apply after performing some kind of regression (e.g. logistic regression) for the purpose of decision-making.

For dependent variables which don't fit these two categories, the things get murky. Ordinal regression lies somewhere between classical (numerical) regression and classification. Predicting counts is, in my opinion, clearly a regression (Poisson, binomial etc.), and not classification.

Answer 6

You need to search more about these terms. Resources may contain 'Linear Algebra', 'Signal & Control', 'Data Mining', 'Predictive Models',... vice versa. I explain some of them briefly here:

Interpolation: Some times we need to predict value of some data points based on another ones, that is generalizing the order we found on some observations (sampled data points) , to another one. Observed data points might be sampled points. We describe this order as mathematical functions (named basis function). This function may be a linear function; in this case we call it Linear Regression. If interpolation is based on current observed values, and aims to predict upcoming observations (assumes that current trend of data points continues on future) we call the method 'Extrapolation'.

Regression: Is the process of finding the best values for parameters of a regression function (for example parameters of a line) in-order to fir into data points. Regression about doesn't have any scenes about trend, it just tries to optimize the parameters of a linear function and find their best values, which fit best to data points. A polynomial regression tries to fit the best polynomial function to data points.

Classification: Some times you want to assign a category to data points. Classes are finite and are pre-known. It is like a process of mapping data points from a infinite space, into some points in a finite space. In Regression, you was trying to search for best values for a basis functions parameters, but in classification you are trying to find best values for a Mapping function. A mapping function could also be a line, a polynomial, or each function which was applicable in interpolation (and regression). The only difference is the way we use it. We use mapping function for separating data points from each other, while use basis functions in regression for finding out the order which describes them the best. Neural Networks (which are used vastly in classifications) contain several mapping matrices which maps input data described in finite properties to output finite classes.

Prediction: Could be prediction value of future data points or their class. That is a general term which can contain both Regression, and Classification. Predication can employ any mathematical approach with the assumption that the current order of events, will continue in future, or any changes will follow an order which could be known or estimated.

Modeling: Is another general term (more general one) which is the way we describe a system by a mathematical (or any thing) abstract model. The output of this process is a tool for predicting, interpolation, describing, and analyzing a system. Regression is a kind of modeling which describes the system as a linear function and could be used for prediction and describing.

Answer 7

I think people are overcomplicating this. Simply put, in classification problems the target variable is nominal (eg "dog" vs "cat"), whereas in regression problems the target is numeric.