Confusion in classification and regression task exception

Question

The distinction between classification and regression accounts for a model output. I know that classification models have discrete and regression continuous outputs.

I want to focus on a taxonomy subtlety between these classifications and regression tasks, though that has been bothering me and I have encountered in many places, for example:

In some cases, classification algorithms will output continuous values in the form of probabilities. Likewise, regression algorithms can sometimes output discrete values in the form of integers.

Maybe the source is not the best example, but this is not the first article I see such a statement in.

Classification with probabilities output seems odd because: probabilities is any number in the continuous interval [0,1]. Can someone elaborate please and mention a specific model?

IMHO, many models can be reformed for either, but a task and model are different things.

An example that might fit the case, I think, but I am not sure is valid are neural networks with softmax outer nodes which have continuous output in [0,1], but we select the maximum among the nodes in a classification task. Otherwise, we speak of a generative model (for example, the decoder part of VAE).

Is there some formal definition that incorporates that or should I just stop reading random articles?

Answer 1

To start from the end:

should I just stop reading random articles?

Maybe you should first take a course on statistics, data science, or machine learning, before you return to reading 'random articles'. There are many gems on the internet, but even more garbage, and without a solid foundation it may be hard to distinguish between them.

Classification with probabilities output seems odd

This is subjective. Maybe probabilities aren't always the best criterion for classification, but sometimes they are, and even more often, another continuous value (perhaps expected gain/loss, which is, in part, derived from the probability) is. As an (artificial) example:

Say, you are an umbrella shop owner and the weather forecast says that there is a 70% chance of raining today. Do you open your shop, or do you give your employees a day off? If you open your shop, but it doesn't rain, you have effectively lost money. If you don't open the shop, but it rains, you have forgone profit. Depending on the cost of keeping the shop open and the profit you make on a rainy day and on the probability of raining you can make the optimal decision.

IMHO many models can be reformed for either

For example: Generalised linear models. Depending on the 'link function' they can be used for linear regression, Poisson regression, logistic regression (which would give you probabilities and allow for classification), and many more.

The choice of the model depends (or should depend) on the assumed process that generates the data. In statistics, people usually assume a law that relates the input and the output variables and superimpose a random noise ('error') over it. Depending on the form (probability distribution) of the noise you get different models.

For example, if you assume that the noise is additive and Gaussian, this leads to ordinary linear regression. On the other hand, if you assume that the 'noise' is something like a coin flip (a 'Bernoulli process'), you get logistic regression.

Hope that helps.

Answer 2

Any neural network trained on a crossentropy loss function performs categorical prediction, but the raw (trained) model output is a probability distribution (after normalization, possibly softmax).

Outputting a distribution is a hallmark of probabilistic methods. The model doesn't make a prediction, per se, but you can think of the model return as prediction distribution. This is the theoretical basis for using crossentropy functions for training models. The prediction comes from taking the choice that maximizes the probability of being correct according to the distribution. To obtain a single prediction from the model for use in an application, we guess the most-likely prediction according to the distribution we got from the neural network model.

Any distinction is a result of the method used to define the model and the mathematical optimization method used to solve it.