3
$\begingroup$

I've looked at a couple other questions here and here and here. So far, I understand that class imbalance is often a cause of this and when you have such a case, you can upsample/downsample or integrate a cost function to weight your preferences for different types of classification error. I also understand that you can change the threshold for a regression problem if you prefer to have more of one kind of error versus another, though it seems that some posters suggest that changing this threshold will just change how many false positives/false negatives you have.

My question, I believe, is different from these in that I think I don't understand something fundamental about logistic regression and binary prediction in general. Say I have a training set with the overall probability of success p = .2. This may not be an exact estimate of the true parameter for the entire population, but assume I do be believe the true parameter to be well below .5. I have a few explanatory variables and my initial exploratory data analysis suggests that in some sub populations, this p is closer to .4. With large enough groups, a boost from p = .2 to p = .4 seems like it could be considered quite significant, suggesting that taking that variable into account should be able to help me improve my prediction of the response variable. When I fit a logistic regression model, the class probabilities I predict do reflect these differences (p in the subpopulation is higher than in the overall population). However, all my probabilities are still below .5, meaning with the default threshold in most models, I will predict that all of the samples, regardless of this higher probability in the subpopulation, will be assigned to the negative class. Something about this doesn't seem quite right. It seems like the difference between .4 and .2 (or .4 and .01, for that matter) should be accounted for somewhere. I'm also wondering whether some of this has to do with the fact that logistic regression predictions are deterministic (i.e. you don't predict a class probability then generate a value of random variable with that probability of success to assign that sample to a group). Am I correctly understanding how classification models behave for p <.5? Any thoughts on this would be greatly appreciated.

$\endgroup$
2
$\begingroup$

This is where it becomes important to distinguish classification per se from probabilistic classification. Classification is the problem of predicting classes; you can assess predictions with metrics like percent agreement, sensitivity, and specificity. Probabilistic classification is the problem of predicting the probabilities of class membership; predictions are best assessed with proper scoring rules, such as the Brier score. In that context, a model will indeed be rewarded for distinguishing probabilities of .2 and .4.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Suppose 80 percent of your data was in group A and 20 percent in group B, and they are almost completely overlapping (according to the variables in your model). Then logistic regression will try to pick a rule (based on linear combinations of your variables, of course) that assigns items in group A the highest (group A probability) and items in group B the lowest probability. The best compromise might be to assign all items an 0.8 probability of being in group A. Then your classification rule (p>0.5) will assign everything to group A, which is probably the best you can do in these circumstances. A similar result might hold even if the classification does have some discriminating power, leaning to classify almost everything into one group could basically give the best fit. You can overcome this with the incorporation of misclassification cost, or else priors.

Logistic regression has no idea which miss-classifications you care about. That is why you need to follow up with (using cross-validation or test/training sets, perhaps) ROC curves and lots of classification tables so that you can decide what misclassification rates you can tolerate.

It is not necessarily a bad move for a classifier to "throw up its hands" and say, "Yo, just put them all in group A" if that works. For instance, if you have a diagnostic test and decide that false positives are no problem (there is an easy follow up) and false negatives are a disaster, the logical conclusion is to not use the test at all and assign everyone into the positive group.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.