1
$\begingroup$

I am looking for a classification algorithm that emits probabilities for each label and supports real-valued features.

1) From what I gather, logistic regression may output a variable in $(0, 1)$, however, it does not describe the true classification probability.

2) Naive Bayes supports probabilities, however, it uses discrete features.

I need this to compute expected utilities. There is no way to couple utilities with probabilities in the algorithm, because the features are not related to the utilities (they are external).

Improve this question
$\endgroup$
2
  • $\begingroup$ Logistic regression should output a probability. It used the logistic function. $\endgroup$
    – Zach
    Commented Jul 11, 2013 at 22:45
  • $\begingroup$ It outputs a variable between $(0,1)$ but that doesn't make it a probability? $\endgroup$
    – Peteris
    Commented Jul 11, 2013 at 22:50

1 Answer 1

Reset to default
3
$\begingroup$

Get out of the mindset of 'classification'. Use an older, direct probability model such as the logistic regression model. If you have a nominal (polytomous; multinomial) $Y$ to predict, use the multinomial logistic model. If $Y$ can be ordered, then use ordinal logistic regression or other cumulative probability ordinal models. If you have too small a sample size for the number of candidate features, consider using the lasso.

The output of logistic regression is a true probability of class membership.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Do you have a proof or a source for the fact that logistic regression is the true probability? Does it assume some prior distribution on the feature space (such as normal)? $\endgroup$
    – Peteris
    Commented Jul 12, 2013 at 1:23
  • 2
    $\begingroup$ If the model is correctly specified and if $Prob[Y=1 | X]$ is "transportable", i.e., $X$ is inclusive of all variables that might affect the incidence of $Y=1$ in new samples (i.e., you don't have a shift in the incidence of $Y=1$ in a new sample due to an uncontrolled/unspecified variable; this would also mess up any classifier), then the binary logistic model (and the other logistic variants) correctly models $Prob[Y=1 | X]$ in the long run. An example where these restrictions apply is a cohort study whose subjects are not selected on unspecified X. No prior is needed. $\endgroup$
    – Frank Harrell
    Commented Jul 12, 2013 at 8:34
  • $\begingroup$ For what loss function is this true? Is it true when we minimize "regression error" $||x \cdot w - y||^2$ or when maximizing likelihood under binomial model? $\endgroup$
    – Peteris
    Commented Jul 12, 2013 at 18:29
  • 1
    $\begingroup$ If the model is correctly stated and the observations independent, we maximize the likelihood which you could call an optimality criterion. Think of the simplest case with no predictors, in which you want to estimate the overall probability of being a member of class 2. The proportion of observations in class 2 is the MLE of the true probability of class membership. This is a minimum assumption method in this Bernoulli distribution case. The main assumption is representative sampling. $\endgroup$
    – Frank Harrell
    Commented Jul 12, 2013 at 18:47
  • 1
    $\begingroup$ This is so basic to the method that I don't know what else to say other than refer you to one of the original papers such as DR Cox (1958): The regression analysis of binary sequences. J Roy Statist Soc B 20:215-242. $\endgroup$
    – Frank Harrell
    Commented Jul 13, 2013 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.