I've looked at a couple other questions here and here and here. So far, I understand that class imbalance is often a cause of this and when you have such a case, you can upsample/downsample or integrate a cost function to weight your preferences for different types of classification error. I also understand that you can change the threshold for a regression problem if you prefer to have more of one kind of error versus another, though it seems that some posters suggest that changing this threshold will just change how many false positives/false negatives you have.
My question, I believe, is different from these in that I think I don't understand something fundamental about logistic regression and binary prediction in general. Say I have a training set with the overall probability of success p = .2. This may not be an exact estimate of the true parameter for the entire population, but assume I do be believe the true parameter to be well below .5. I have a few explanatory variables and my initial exploratory data analysis suggests that in some sub populations, this p is closer to .4. With large enough groups, a boost from p = .2 to p = .4 seems like it could be considered quite significant, suggesting that taking that variable into account should be able to help me improve my prediction of the response variable. When I fit a logistic regression model, the class probabilities I predict do reflect these differences (p in the subpopulation is higher than in the overall population). However, all my probabilities are still below .5, meaning with the default threshold in most models, I will predict that all of the samples, regardless of this higher probability in the subpopulation, will be assigned to the negative class. Something about this doesn't seem quite right. It seems like the difference between .4 and .2 (or .4 and .01, for that matter) should be accounted for somewhere. I'm also wondering whether some of this has to do with the fact that logistic regression predictions are deterministic (i.e. you don't predict a class probability then generate a value of random variable with that probability of success to assign that sample to a group). Am I correctly understanding how classification models behave for p <.5? Any thoughts on this would be greatly appreciated.