# What are some classification approaches for real-valued features that produce actual probabilities?

I am looking for a classification algorithm that emits probabilities for each label and supports real-valued features.

1) From what I gather, logistic regression may output a variable in $(0, 1)$, however, it does not describe the true classification probability.

2) Naive Bayes supports probabilities, however, it uses discrete features.

I need this to compute expected utilities. There is no way to couple utilities with probabilities in the algorithm, because the features are not related to the utilities (they are external).

• Logistic regression should output a probability. It used the logistic function.
– Zach
Commented Jul 11, 2013 at 22:45
• It outputs a variable between $(0,1)$ but that doesn't make it a probability? Commented Jul 11, 2013 at 22:50

Get out of the mindset of 'classification'. Use an older, direct probability model such as the logistic regression model. If you have a nominal (polytomous; multinomial) $Y$ to predict, use the multinomial logistic model. If $Y$ can be ordered, then use ordinal logistic regression or other cumulative probability ordinal models. If you have too small a sample size for the number of candidate features, consider using the lasso.
• If the model is correctly specified and if $Prob[Y=1 | X]$ is "transportable", i.e., $X$ is inclusive of all variables that might affect the incidence of $Y=1$ in new samples (i.e., you don't have a shift in the incidence of $Y=1$ in a new sample due to an uncontrolled/unspecified variable; this would also mess up any classifier), then the binary logistic model (and the other logistic variants) correctly models $Prob[Y=1 | X]$ in the long run. An example where these restrictions apply is a cohort study whose subjects are not selected on unspecified X. No prior is needed. Commented Jul 12, 2013 at 8:34
• For what loss function is this true? Is it true when we minimize "regression error" $||x \cdot w - y||^2$ or when maximizing likelihood under binomial model? Commented Jul 12, 2013 at 18:29