# Regression vs. Classification: Is there a clear, generally accepted definition?

As a mathematician/economist, I am not trained to think in classification and regression tasks. This is why I wonder: is there a clear, widely accepted definition of regression and classification problems?

E.g., this paper says that When $$Y$$ has a finite number of states we refer to the task as classification. Otherwise we refer to the task as regression.

Does this mean that count data with many counts are a classification problem (e.g., Poisson regression). Or if we model life expectancy, is this generally supposed to be a classification task?

Is this definition generally accepted?

• classification is really an ML name, with the historical view of assigning labels to a data point (ignoring class probabilities - unlike this paper). for a statistician everything is really regression estimating E(y|x). Commented Oct 29, 2022 at 10:11
• @seanv507 I don't think that is really true, there are plenty of statisticians that seem happy to talk of classification (or equivalently pattern recognition), e.g. Brian Ripley or Geoff McLachlan. Also Vladimir Vapnik is a statistician. Commented Oct 29, 2022 at 10:19
• The question "Is there a generally accepted definition of concept X in science?" always seems a bit of fruitless to me. (Usually the answer is "no, so what?") And for distinction between classification and regression, variations of this question have been asked on CV before, incl. 1, 2, 3. I find this thread most interesting: Why not approach classification through regression?. Commented Oct 30, 2022 at 10:58
• Classification is (also) what people do with language (that's a cat; that' policy is reprehensible) and was a really big deal in much of science (e.g. biology) long before machine learning was ever a thing. Commented Oct 30, 2022 at 12:47

Classification denotes an action. It's what you do with the result of an analysis in which there is one or more outcome variables and one or more input (predictor; covariate) variables. If there is a single outcome variable, the discreteness of the variable does not matter. For example, binary logistic regression is for binary Y and is a direct (continuous) probability model that was not intended to be used for classification. The action of classification involves making choices and use of decision rules. In most cases it represents a premature decision made by an analyst who is not blessed with knowledge about the consequence of the decision (i.e., does not possess the utility/loss/cost function needed to make a good decision).

One can use any predictive method to do classification even if that was not the intent of the method. For example one can use arbitrary thresholds on predicted values to do classification from ordinary regression for continuous Y, or ordinal or binary regression for ordered or binary Y.

Many in machine learning think of classification as a good default mode; it is not, as detailed in my blog post. Among other things, classification hides close calls and lulls users into making decisions at the boundaries (e.g., when a predicted probability is 0.5001) when a better approach would be "get more data first".

Most of the time when you see classifier used in a sentence the correct term is prediction when the output is considered to be continuous.

• Sometimes decisions must be made under constraints which prevent deep analysis. Take tumour segmentation on a 1 Mpixel image: Do you want to perform 1 million statistical analyses? While knowing that the independence assumption is extremely violated? And the linearity of the odds, at best, a pious hope? Or, think about stock trading: By the time you have made your analytical decision, the conditions for it will have become obsolete a thousand times. The choice is not between classification and statistical analysis; it is between classification and doing nothing. Commented Oct 31, 2022 at 7:20
• @IgorF. It’s certainly possible to code the nuances of probabilistic predictions into whatever algorithm you put into production to operate autonomously.
– Dave
Commented Oct 31, 2022 at 12:46
• @IgorF. there are many statistical modeling approaches that do not assume independence or linearity. Commented Oct 31, 2022 at 14:14
• Please consider making it clearer that the provided reference is not a usual one, but something written solely by you. Commented Oct 31, 2022 at 15:25
• That would make sense were practitioners to do what you mentioned (create a "we don't know" category). But they almost always forget to do this. Commented Nov 1, 2022 at 16:53

No, I don't think that definition is generally accepted. I would not regard Poisson regression as classification as the thing you are generally interested is the conditional values of a Poisson distribution that describes the distribution of the target variable for those values of the attributes. Those parameters are generally continuous. You might then use that to work out the most likely count, but that would be discretising the predictive distribution given by the model.

Likewise some here (e.g. Frank Harrell - see his answer to this question +1) view logistic regression purely as a probabilistic model, used to estimate a conditional probability, and not as a classification model (which is what you get by applying a threshold and discretising the continuous output of the model). I have a lot of sympathy with this view, except that in practical applications where you need to perform that discretisation, that still impacts on the design and evaluation of the model and shouldn't be ignored. The optimal classification is not always obtained by estimating the probability of class membership and thresholding, sometimes it is better to classify the data directly. If that were not the case, [kernel] logistic regression would not perform worse than the Support Vector Machine, but on some applications it clearly does.

I'd probably say that a classifier is a problem where the target distribution is categorical, and the aim to to place each object into a category.

A reasonable standard definition of classification would be that the $$Y$$ value is of nominal scale level, i.e., that order and numerical differences are not meaningful (or at least not of interest). Models with count or ordinal responses are widely referred to as regression, e.g., Poisson regression, ordinal regression.

Note that the comment by Dikran Marsupial is correct that some models with nominal scaled outcome are also referred to as regression ((multinomial) logistic regression). My interpretation of this would be that the term regression is also used to refer to a class of models that grew out of historically quantitative regression models. I think it makes still sense (and would probably be standard handling of the term) to also refer to these as classification methods, and certainly to the problems solved by them as classification problems. This handling of terms would have regression and classification as not necessarily mutually exclusive. Probably regression is more ambiguous than classification, as it can be used for methods for quantitative or at least ordinal responses (and insofar be distinct from classification), but also for certain model classes that may also have versions for classification problems.

• +1, but a [multinomial] logistic regression model would have a Y value that is nominal, but some would still reasonably view it as a regression problem. Commented Oct 29, 2022 at 10:55
• @DikranMarsupial Fair enough. To some extent regression and classification can be seen as overlapping. I add something to the answer. Commented Oct 29, 2022 at 10:57

To muddy the waters further, classification can mean

1. trying to find distinct classes in a dataset from scratch, which has attracted many different names, including mathematical or numerical taxonomy, but cluster analysis seems the most durable and popular

2. assigning observations to classes already defined, which has other names too including identification and discrimination.

• These are rather routinely referred to as unsupervised vs. supervised classification, although of course the other terms exist as well. Commented Oct 29, 2022 at 12:58
• Sure, but only routinely over the last decade or so, I think. The terminology I mention goes back much further. Commented Oct 29, 2022 at 15:34
• Trivially, but crucially in practice, terminology that is familiar comes to seem obvious, even though you know that wasn't true when you first learned it and isn't true for many other people still. Thus I have no difficulty in distinguishing independent and dependent variables, but think that poor terminology given that the terms are often confused. Conversely, which is unsupervised and which is supervised I find quite hard to remember, that could mean stupidity, or arise because I don't mix with machine learners or read much in their literature. Commented Oct 30, 2022 at 12:37

Definitions are overrated. They are just words which someone claims are synonymous to other words. They may be useful in special situations, e.g. to help a novice get a grasp of a concept, or to ensure that experts, when communicating with each other, know precisely what they are talking about. But, standing alone, who cares?

That said, I believe all machine learners and a good deal of statisticians wouldn't object and would understand what you mean if you say "classification" for a task where your dependent variable is nominal (synonym: categorical) in Stevens' typology. Most machine learners will associate regression with numerical dependent variables, but some statisticians would say that "everything is regression", and classification a step---obviously outside of "everything"---you apply after performing some kind of regression (e.g. logistic regression) for the purpose of decision-making.

For dependent variables which don't fit these two categories, the things get murky. Ordinal regression lies somewhere between classical (numerical) regression and classification. Predicting counts is, in my opinion, clearly a regression (Poisson, binomial etc.), and not classification.

• Nominal isn't a synonym of categorical to just about any statistical person looking at texts or courses on categorical data analysis, as there usually categorical includes ordinal. Now you can agree or disagree with that, like or dislike that, but the point is that even if you don't use or like a definition it may be what other people are using. I agree that fussing about terminology or word use can seem trivial or pointless, but it is needed whenever people can't communicate easily or effectively because they are using different definitions. Commented Oct 30, 2022 at 12:44
• Your take on the usefulness of definitions seems very strange to me. We use words so that we can communicate efficiently but clearly it only works so long as we mostly agree about definitions. Definitions only “stand alone” for people who don’t have to communicate in that field. Commented Oct 30, 2022 at 14:07
• @einar I don't think natural languages work that way (which is one reason maths is so useful). We often have to use reasoning to disambiguate meanings and also the meanings tend to evolve differently in different communities, c.f. "two great [sic] nations divided by a common language" - attributed to George Bernard Shaw. Definitions are useful, but they don't define what words mean so much as tell you generally accepted usages (but there may also be other usages that don't fit the definition). ML is a good example of this, due to the diversity in educational backgrounds of practitioners. Commented Oct 30, 2022 at 14:27
• @DikranMarsupial no you’re probably right and I’m not even sure I expressed myself clearly in that comment just to underline your point. Commented Oct 30, 2022 at 18:11

You need to search more about these terms. Resources may contain 'Linear Algebra', 'Signal & Control', 'Data Mining', 'Predictive Models',... vice versa. I explain some of them briefly here:

Interpolation: Some times we need to predict value of some data points based on another ones, that is generalizing the order we found on some observations (sampled data points) , to another one. Observed data points might be sampled points. We describe this order as mathematical functions (named basis function). This function may be a linear function; in this case we call it Linear Regression. If interpolation is based on current observed values, and aims to predict upcoming observations (assumes that current trend of data points continues on future) we call the method 'Extrapolation'.

Regression: Is the process of finding the best values for parameters of a regression function (for example parameters of a line) in-order to fir into data points. Regression about doesn't have any scenes about trend, it just tries to optimize the parameters of a linear function and find their best values, which fit best to data points. A polynomial regression tries to fit the best polynomial function to data points.

Classification: Some times you want to assign a category to data points. Classes are finite and are pre-known. It is like a process of mapping data points from a infinite space, into some points in a finite space. In Regression, you was trying to search for best values for a basis functions parameters, but in classification you are trying to find best values for a Mapping function. A mapping function could also be a line, a polynomial, or each function which was applicable in interpolation (and regression). The only difference is the way we use it. We use mapping function for separating data points from each other, while use basis functions in regression for finding out the order which describes them the best. Neural Networks (which are used vastly in classifications) contain several mapping matrices which maps input data described in finite properties to output finite classes.

Prediction: Could be prediction value of future data points or their class. That is a general term which can contain both Regression, and Classification. Predication can employ any mathematical approach with the assumption that the current order of events, will continue in future, or any changes will follow an order which could be known or estimated.

Modeling: Is another general term (more general one) which is the way we describe a system by a mathematical (or any thing) abstract model. The output of this process is a tool for predicting, interpolation, describing, and analyzing a system. Regression is a kind of modeling which describes the system as a linear function and could be used for prediction and describing.

• I can't see that this really complements exists fine answers helpfully, and some of its details are disputable. The insistence that regression is about fitting a linear function doesn't capture many important variants such as logit regression. Commented Nov 8, 2022 at 9:14
• I wanted to show questioner that there are many related or similar terms. I tried to clarify the purpose of different methods, to make him understand why we can use methods in any problem (using classification instead of regression). And Yes! these are disputable because can't claim that i am an expert. @NickCox Commented Nov 10, 2022 at 8:46

I think people are overcomplicating this. Simply put, in classification problems the target variable is nominal (eg "dog" vs "cat"), whereas in regression problems the target is numeric.

• What to do with ordinal target variables? Commented Nov 1, 2022 at 19:44
• This misses the major difficulty, which is more than that you and people close by have a terminology you think is good, clear and consistent. The problem is twofold at least (a) all the meanings other people have (b) where to draw the line. For example, it's not (in my view) perverse to regard classification as a limiting case of regression. Commented Nov 2, 2022 at 12:20
• @RichardHardy I think you need to interpret the answer as an exercise in classification, while the problem in the question is actually one of regression 😉 Commented Nov 2, 2022 at 17:46
• @RichardHardy Well "ordinal" means the order matters, so this is a regression problem to me. Commented Nov 7, 2022 at 18:16
• Your comment like your post is all of a circle with your terminology without recognizing fully that terminology is not consistent across statistical science. Naturally if you define the two as disjoint any other usage is at odds with that. I too prefer the terminology I use. But here it goes. Regression can in essence forecast probabilities and a decision to classify can follow. But it's not my point; it's quite standard and already very well developed by @Frank Harrell in his answer. Commented Nov 7, 2022 at 18:25