0
$\begingroup$

In terms of Machine Learning models, as far as I know, there are two kinds of models in Machine Learning. One kind are discriminative models and the other kind are generative models. But I wonder if those two kinds of Machine Learning models are probabilistic models.

$\endgroup$
1
  • $\begingroup$ It seems like the answer to this question must depend on which machine learning model in particular we are considering and how you define a probabilistic model. This question reminds me of the discussion around stats.stackexchange.com/questions/127042/… $\endgroup$
    – Sycorax
    Jan 3 '20 at 16:25
5
$\begingroup$

Some, but not all, of the machine learning models, are probabilistic models. There are machine learning models that are probabilistic by design, such as Naive Bayes. There are also ones that are not probabilistic, like SVM, random forest, or $k$-NN, because they were not designed in terms of thinking of random variables and probability distributions.

On another hand, many models that are not defined in probabilistic terms per se, can be interpreted as probabilistic models. Minimizing a loss function is equivalent to maximizing some likelihood, for example minimizing squared loss is equivalent to maximizing Gaussian likelihood, and logistic loss to Bernoulli likelihood. Also regularization can be thought as setting priors for the parameters, so it is relatively easy to translate something like neural network, to Bayesian neural network. However, as you can learn from this thread, in some cases you can run into problems with it.

Being a probabilistic model is a "nice to have" feature for machine learning model, because it makes easier for us to evaluate the uncertainties related to the predictions returned by the model. It is not "obligatory", because in machine learning we usually worry only about making accurate predictions (rather then inference), and in many cases the well established tools for assessing their performance, like cross-validation, are enough for this task. For example, $k$-NN classifies cases based on their similarities to other samples in the training data, and this both works and is intuitively appealing way of making classifications, there is no need to defining it in probabilistic terms.

$\endgroup$
2
  • $\begingroup$ Generative models must be probabilistic models by nature. But I am really confused about what discriminative models mean. All I know is that they use conditional probabilities to build boundaries and they directly build posterior distributions. How do they build direct posterior from conditional probabilities? and I don't think I really understand the phrase. $\endgroup$
    – StoryMay
    Jan 3 '20 at 10:20
  • 1
    $\begingroup$ @ChangheeKang this seems to be a separate question. Have you checked stats.stackexchange.com/questions/21434/… or stats.stackexchange.com/questions/12421/… threads? If they don't help, you can ask it as a separate question. $\endgroup$
    – Tim
    Jan 3 '20 at 11:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.