6
$\begingroup$

The distinction between classification and regression accounts for a model output. I know that classification models have discrete and regression continuous outputs.

I want to focus on a taxonomy subtlety between these classifications and regression tasks, though that has been bothering me and I have encountered in many places, for example:

In some cases, classification algorithms will output continuous values in the form of probabilities. Likewise, regression algorithms can sometimes output discrete values in the form of integers.

Maybe the source is not the best example, but this is not the first article I see such a statement in.

Classification with probabilities output seems odd because: probabilities is any number in the continuous interval [0,1]. Can someone elaborate please and mention a specific model?

IMHO, many models can be reformed for either, but a task and model are different things.

An example that might fit the case, I think, but I am not sure is valid are neural networks with softmax outer nodes which have continuous output in [0,1], but we select the maximum among the nodes in a classification task. Otherwise, we speak of a generative model (for example, the decoder part of VAE).

Is there some formal definition that incorporates that or should I just stop reading random articles?

$\endgroup$

2 Answers 2

10
$\begingroup$

To start from the end:

should I just stop reading random articles?

Maybe you should first take a course on statistics, data science, or machine learning, before you return to reading 'random articles'. There are many gems on the internet, but even more garbage, and without a solid foundation it may be hard to distinguish between them.

Classification with probabilities output seems odd

This is subjective. Maybe probabilities aren't always the best criterion for classification, but sometimes they are, and even more often, another continuous value (perhaps expected gain/loss, which is, in part, derived from the probability) is. As an (artificial) example:

Say, you are an umbrella shop owner and the weather forecast says that there is a 70% chance of raining today. Do you open your shop, or do you give your employees a day off? If you open your shop, but it doesn't rain, you have effectively lost money. If you don't open the shop, but it rains, you have forgone profit. Depending on the cost of keeping the shop open and the profit you make on a rainy day and on the probability of raining you can make the optimal decision.

IMHO many models can be reformed for either

For example: Generalised linear models. Depending on the 'link function' they can be used for linear regression, Poisson regression, logistic regression (which would give you probabilities and allow for classification), and many more.

The choice of the model depends (or should depend) on the assumed process that generates the data. In statistics, people usually assume a law that relates the input and the output variables and superimpose a random noise ('error') over it. Depending on the form (probability distribution) of the noise you get different models.

For example, if you assume that the noise is additive and Gaussian, this leads to ordinary linear regression. On the other hand, if you assume that the 'noise' is something like a coin flip (a 'Bernoulli process'), you get logistic regression.

Hope that helps.

$\endgroup$
8
  • $\begingroup$ Thank you for the answer, in the infamous Andrew Ng's course it starts with this taxonomy between classification and regresssion models but I didn't understand from a definition that could interpret classification as a continuous output task, or regression with discret output. I seek for is a mathematical definition incorporating this. $\endgroup$
    – partizanos
    Jan 8, 2021 at 6:58
  • $\begingroup$ Regarding the umbrealla example it is a classification task if you want to open or not the shop. A regression task if you seek the probability of rain, they are related but distinct questions despite being able to answer them with the same model. $\endgroup$
    – partizanos
    Jan 8, 2021 at 6:59
  • 1
    $\begingroup$ Regarding mathematical definition, I suggest you look into generalised linear models (en.wikipedia.org/wiki/Generalized_linear_model, rss.onlinelibrary.wiley.com/doi/abs/10.2307/2344614). As for regression with discrete output, you can take a look at proportional odds model (en.wikipedia.org/wiki/Ordered_logit) $\endgroup$
    – Igor F.
    Jan 8, 2021 at 7:10
  • 2
    $\begingroup$ Re: Umbrella: First there is a (logistic?) regression, performed by the meteorologists. They give you a continuous value (0.7 or 70% in the example). Based on that, and your cost/benefit deliberations, you make a decision (you can call it 'classification') whether to open the shop or not. So you can regard regression as a step in the classification process. $\endgroup$
    – Igor F.
    Jan 8, 2021 at 7:14
  • 2
    $\begingroup$ The opposite is closer to truth: Classification is mapping a real-valued random variable to a discrete space. Logistic regression, for example, models the log-odds of the outcome as a linear combination of the predictors. The 'classification' step is then to map these log-odds to discrete classes. $\endgroup$
    – Igor F.
    Jan 8, 2021 at 7:28
1
$\begingroup$

Any neural network trained on a crossentropy loss function performs categorical prediction, but the raw (trained) model output is a probability distribution (after normalization, possibly softmax).

Outputting a distribution is a hallmark of probabilistic methods. The model doesn't make a prediction, per se, but you can think of the model return as prediction distribution. This is the theoretical basis for using crossentropy functions for training models. The prediction comes from taking the choice that maximizes the probability of being correct according to the distribution. To obtain a single prediction from the model for use in an application, we guess the most-likely prediction according to the distribution we got from the neural network model.

Any distinction is a result of the method used to define the model and the mathematical optimization method used to solve it.

$\endgroup$
1
  • $\begingroup$ Isn't prediction distribution the same as probability? In classification you need to perform on this distribution you can perform hypothesis testing (eg Neuman Pearson criterion) and derive an exact threshold to output in the discrete space. Aren't both the derivatio nof the probability and the and threshold selection part of the predictive model? $\endgroup$
    – partizanos
    Jan 8, 2021 at 7:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.