# What are some classification approaches for real-valued features that produce actual probabilities?

I am looking for a classification algorithm that emits probabilities for each label and supports real-valued features.

1) From what I gather, logistic regression may output a variable in $(0, 1)$, however, it does not describe the true classification probability.

2) Naive Bayes supports probabilities, however, it uses discrete features.

I need this to compute expected utilities. There is no way to couple utilities with probabilities in the algorithm, because the features are not related to the utilities (they are external).

• Logistic regression should output a probability. It used the logistic function. – Zach Jul 11 '13 at 22:45
• It outputs a variable between $(0,1)$ but that doesn't make it a probability? – Peteris Jul 11 '13 at 22:50

## 1 Answer

Get out of the mindset of 'classification'. Use an older, direct probability model such as the logistic regression model. If you have a nominal (polytomous; multinomial) $Y$ to predict, use the multinomial logistic model. If $Y$ can be ordered, then use ordinal logistic regression or other cumulative probability ordinal models. If you have too small a sample size for the number of candidate features, consider using the lasso.

The output of logistic regression is a true probability of class membership.

• Do you have a proof or a source for the fact that logistic regression is the true probability? Does it assume some prior distribution on the feature space (such as normal)? – Peteris Jul 12 '13 at 1:23
• If the model is correctly specified and if $Prob[Y=1 | X]$ is "transportable", i.e., $X$ is inclusive of all variables that might affect the incidence of $Y=1$ in new samples (i.e., you don't have a shift in the incidence of $Y=1$ in a new sample due to an uncontrolled/unspecified variable; this would also mess up any classifier), then the binary logistic model (and the other logistic variants) correctly models $Prob[Y=1 | X]$ in the long run. An example where these restrictions apply is a cohort study whose subjects are not selected on unspecified X. No prior is needed. – Frank Harrell Jul 12 '13 at 8:34
• For what loss function is this true? Is it true when we minimize "regression error" $||x \cdot w - y||^2$ or when maximizing likelihood under binomial model? – Peteris Jul 12 '13 at 18:29
• If the model is correctly stated and the observations independent, we maximize the likelihood which you could call an optimality criterion. Think of the simplest case with no predictors, in which you want to estimate the overall probability of being a member of class 2. The proportion of observations in class 2 is the MLE of the true probability of class membership. This is a minimum assumption method in this Bernoulli distribution case. The main assumption is representative sampling. – Frank Harrell Jul 12 '13 at 18:47
• This is so basic to the method that I don't know what else to say other than refer you to one of the original papers such as DR Cox (1958): The regression analysis of binary sequences. J Roy Statist Soc B 20:215-242. – Frank Harrell Jul 13 '13 at 11:55