What is the difference between the probabilistic and non-probabilistic learning methods for different situations? When is one considered more appropriate than the other? Can you give me examples?
The task of classification enables a simple comparison:
A probabilistic approach (such as Random Forest) would yield a probability distribution over a set of classes for each input sample.
A deterministic approach (such as SVM) does not model the distribution of classes but rather seperates the feature space and return the class associated with the space where a sample originates from.
Yet it is possible for every probabilistic method to simply return the class with the highest probability and therefore seem deterministic. Further, the other way around, based on the distance to the seperating hyperplane in SVMs a probability can be computed and returned for each class.
Generally, the probabilistic approaches are better suited to incorporate uncertainty regarding the answer and can provide information about how safe a prediction is. Further, probabilitic methods might be able to incorporate prior information about the class distribution.
If you would for example try to predict gender based on body heigth, a deterministic appraoch would simply chose a seperating point and say that:
(completely made up numbers)
height >= 175 --> man
height < 175 --> woman
so for a sample with the height of 175 it would yield woman as well as it would for a heigth of 155.
A probabilistic approach in this case could yield a probability such as for a height of
155 ---> man(0.1), woman(0.9) and for
175 --> man(0.49), woman(0.51)