Say I have the titanic kaggle competition, but I'm not interested in the competition for predicting survival for each individual. Instead I want the most accurate estimate of total survivors on the titanic. Would this be achieved by using a probabilistic model, then adding the probabilities for each individual? For example, if I have 3 people and 1 survived, but my model produced 0.4, 0.4, and 0.4 probabilities for each person to survive, I calculate 0 survived. But if I add 0.4 for each person, I get 1.2, which is closer to the actual. Does this make sense?
5
2 Answers
$\begingroup$
$\endgroup$
5
A classificator is only trained to be good with the tendency, not the number. It does not return a probability, but a certainty (not necessarily on an obvious scale). There may be cases where the surviving cance is e. g. 75% and the classifier is 99% certain of this (and thus return 0.99 for surviving being the best guess). If you want to use the number, learn regression or carefully calibrate your classifier.
answered Jan 15, 2015 at 7:52
-
$\begingroup$ Could you explain the distinction between a "probability" and a "certainty"? On what scale is the certainty expressed? $\endgroup$– whuber ♦Commented Mar 4, 2015 at 21:38
-
$\begingroup$ I do not bother to be 100% precise, sorry. $\endgroup$ Commented Mar 4, 2015 at 21:47
-
$\begingroup$ Certainty would need to be given with an error interval I guess to conform to statistical definitions. But for most practical data, this requires making lots of unrealistic assumptions so that the benefit is rather limited except for theory. $\endgroup$ Commented Mar 4, 2015 at 22:13
-
1$\begingroup$ I didn't mean for my question to be interpreted in any sophisticated way. It was meant in the most basic sense: just what is a "certainty" and how is it related to the ordinary meaning of probability? Or is your "certainty" no more than an ad hoc term for an arbitrary value returned by a classifier? But if so, how would that be related to the "probabilistic model" at the basis of the original question? $\endgroup$– whuber ♦Commented Mar 4, 2015 at 22:25
-
$\begingroup$ Well, the point is that you have a predicted value - which can be a probability - and you can also have a probability that this prediction is correct (or not). A prediction of 100% is useless if it is wrong more often than right. In the case of uncertain data, predicting the extremes (or using the extremes for learning) is often not the best strategy. This situation (people that may have an illness, but with a lot of errors in the data) requires good calibration IMHO. $\endgroup$ Commented Mar 4, 2015 at 23:19
$\begingroup$
$\endgroup$
The approach makes sense to me. I would be careful though with the interpretation of the "probability" that some machine learning algorithms produce. The logistic regression directly models the probabilities. If correctly specified, then the output can be interpreted as "probability". Neural nets are similar; they can be seen as logistic regression with highly complex feature extraction.
I am not sure about the tree-based methods. As far as I know, there is no consensus if we can interpret the "scores" of methods like the random forest as "probabilities". As you said, the scores indicate the fraction of trees that judged the positive. The trees are samples from possible models, whereas what we want is samples from trials. I find no clear link between them.
As a result, I personally see the scores from the random forest just as scores. The higher the score, the larger the probability. Perhaps they are a monotonic transformation of probabilities, but not necessarily equal.
That said, I would be happy if someone could correct me if my interpretation is wrong.
E(X + Y) = E(X) + E(Y). $\endgroup$