Is Regression and Classification "Inherently" Based on Probability?

Question

From a classical perspective, I have outlined some examples of models in which Probability seems to play an "inherent role" in Regression and Classification:

• As a simple example, suppose we have some data (e.g. heights of students: 175 cm, 181 cm, 162 cm, etc.) . If we assume that this data comes for a Normal Probability Distribution, we could use the Likelihood Function of a Normal Distribution to estimate the specific Normal Distribution (i.e. the mean and standard deviation) that would have been "most likely" (i.e. optimal) to produce this data.

• Regression: As a slightly more involved example, when we are working with Regression Models, we are interested in optimizing the "conditional expectation of the response given the covariates", provided we assume some underlying Probability Distribution. In many aspects, this is just like the first example - in most cases, we assume that the difference between the Actual Response and the Predicted Response (i.e. "error") is Normally Distributed - effectively, this Normal Distribution has a mean of "b0 + b1x1..." . We are now trying to find out the optimal values of these beta regression coefficients provided some underlying Probability Distribution.

• Classification: In Classification problems, a regression model tries to learn a "optimal hyperplane" that separates the data such that the resulting error is minimized. When a new data point appears, the model will calculate the probability that this new data point belongs to any of the classes - and the the class corresponding to the highest of these probabilities will be assigned to this new point.

The above models are generally more closely related to Probability Distributions compared to models such as Neural Networks. However, I am not sure if Neural Networks for Regression and Classification tasks still operate under the following premise: Given a specific input (i.e. combination of covariates), we would like to identify the response for this input that has the highest probability "conditional" on this observed input.

For models such as Neural Networks:

• Do Neural Networks have an inherent Probability component? Can we say that the predictions being made by some specific Neural Network model is "probabilistically the most likely response" given some observed inputs - Or does the notion of Probability have no real relevance here?

• In a very general case for a classical MLP Network, suppose that for some specific input, a Neural Network predicts that the response is 17.6 : It would be wrong to imagine some probability distribution associated with this response and the expectation of this probability distribution being 17.6?

Thanks!

Note: I am aware of some general ideas in Machine Learning that involve Probability such as "PAC Theory" and "Empirical Risk Minimization" - but I am interested in learning whether models such as Neural Networks have an inherent probabilistic interpretation when making predictions (as do Regression Models). I am thinking that perhaps all Statistical Decision Theory (e.g. optimal classification label for a new observation, optimal prediction for a new observation) might have an inherent probabilistic interpretation, regardless of the Machine Learning model being used?

Answer 1

Christopher Bishop called his classic book Pattern Recognition and Machine Learning because what machine learning algorithms do is find patterns in the data and identify those patterns at prediction time. It proved to work very well as an automated solution for solving many different practical problems.

However, if you think about it, why if you collected a bunch of samples, and trained an algorithm on them, does it allow you to generalize the results to the samples you didn't see? To do this, you need to assume that your training data is a sample that comes from a broader population, and shares common characteristics with the population, so you can extrapolate your results to the rest of the population. Probability theory and statistics provide formal theoretical frameworks for doing exactly this.

As stated in the other answers, not every model predicts probabilities or expected values, not every loss directly translates to a likelihood function, but on theoretical grounds, we think of all those models using probability theory language because it allows us to connect the samples with the population. Most statistical models were designed in probabilistic terms from the beginning. For other algorithms that were not, on theoretical grounds we will still use probabilistic language to reason about them.

Answer 2

How you interpret the model's predictions depend on the model and the data. Some losses take the form of a conditional expectation, but not all losses.

• Quantile regression estimates a conditional quantile (e.g. conditional median) instead of the conditional mean.

• Not all neural networks use loss functions that are based in probability. An example that I've worked with is the triplet-loss network, which seeks to map data in the same class to a vector representation that is far away from the vector representations of all other classes.

• Neural networks are not unique in being unrelated to probability. SVMs are not directly related to a probability model. Instead, the model training is motivated by optimizing a separating hyperplane. (But sometimes researchers will transform the signed distance from the hyperplane to a probability -- but this is not innately connected to training an SVM.)

Answer 3

In a certain sense a lot of models are allowing some sort of probabilistic interpretation even when they do not directly optimize a probability.

Many models are, for instance, optimised via some learning method that involves a probabilistic cross validation step. They optimize an estimate of parameters via trial-and-error and select the one that most often leads to a low cost. This estimate is probably close to the true value that is estimated.

Anything for which a standard error can be computed or estimated can be interpreted as probabilistic. Although sometimes the probability is not exactly computed, for instance just a variance is estimated.