From a classical perspective, I have outlined some examples of models in which Probability seems to play an "inherent role" in Regression and Classification:
As a simple example, suppose we have some data (e.g. heights of students: 175 cm, 181 cm, 162 cm, etc.) . If we assume that this data comes for a Normal Probability Distribution, we could use the Likelihood Function of a Normal Distribution to estimate the specific Normal Distribution (i.e. the mean and standard deviation) that would have been "most likely" (i.e. optimal) to produce this data.
Regression: As a slightly more involved example, when we are working with Regression Models, we are interested in optimizing the "conditional expectation of the response given the covariates", provided we assume some underlying Probability Distribution. In many aspects, this is just like the first example - in most cases, we assume that the difference between the Actual Response and the Predicted Response (i.e. "error") is Normally Distributed - effectively, this Normal Distribution has a mean of "b0 + b1x1..." . We are now trying to find out the optimal values of these beta regression coefficients provided some underlying Probability Distribution.
Classification: In Classification problems, a regression model tries to learn a "optimal hyperplane" that separates the data such that the resulting error is minimized. When a new data point appears, the model will calculate the probability that this new data point belongs to any of the classes - and the the class corresponding to the highest of these probabilities will be assigned to this new point.
The above models are generally more closely related to Probability Distributions compared to models such as Neural Networks. However, I am not sure if Neural Networks for Regression and Classification tasks still operate under the following premise: Given a specific input (i.e. combination of covariates), we would like to identify the response for this input that has the highest probability "conditional" on this observed input.
For models such as Neural Networks:
Do Neural Networks have an inherent Probability component? Can we say that the predictions being made by some specific Neural Network model is "probabilistically the most likely response" given some observed inputs - Or does the notion of Probability have no real relevance here?
In a very general case for a classical MLP Network, suppose that for some specific input, a Neural Network predicts that the response is 17.6 : It would be wrong to imagine some probability distribution associated with this response and the expectation of this probability distribution being 17.6?
Note: I am aware of some general ideas in Machine Learning that involve Probability such as "PAC Theory" and "Empirical Risk Minimization" - but I am interested in learning whether models such as Neural Networks have an inherent probabilistic interpretation when making predictions (as do Regression Models). I am thinking that perhaps all Statistical Decision Theory (e.g. optimal classification label for a new observation, optimal prediction for a new observation) might have an inherent probabilistic interpretation, regardless of the Machine Learning model being used?