How to increase the accuracy of my logistic regression model?

Question

I am dealing with a tricky, unbalanced data set and trying to run a logistic regression model. One class is present with a 10:1 ratio.

My objective here is to boost my predictive accuracy - minimize the incorrect predictions and maximize the correct ones.

I've tried undersampling (which doesn't work very well) and I have tried logistic regression,case-weighted logistic, and Firth logistic regression.

None of them are very successful.

While I can sometimes yield good true negative rates depending on the dataset, in the end the prediction is merely representative of its underlying class distribution. The case-weighted logistic does about as well as the Firth when I test it - which is to say, it does horribly.

So...

1. Is there anything else I can do to meet my objective? From what I can tell, exact logistic regression is an option but only for very small datasets, which is not the case here.

2. Do I need to go back and explore variable selection?

3. Why is it that the penalized model (Firth) is not a substantial improvement over the case-weighted logistic?

4. Are there any options I am not considering?

5. Should I change tactics and look to something like an anomaly detection model instead?

-- Please excuse me if this question is lacking in details -this is my first time asking a question here. I would be happy to add in anything.

Answer 1

Just think about the fact that you have a 10:1 class ratio and yet your interest is in zero-one loss, that is, maximizing the proportion of cases correctly identified. The accuracy of a trivial model that just guesses the modal class for every case would be 10/(10 + 1) = 91%, which is pretty high. In order to substantially beat 91%, as with 95% accuracy, you need one or more highly predictive features. If you don't have any, as is often the case in real problems, the best you can hope for is quite small improvements on 91%. In a nutshell, this problem is too easy for statistics or machine learning to be able to get you much more predictive power.

Possibly I can give you more specific advice if you add details.

Answer 2

It might make sense for you to try different models. You might have some success with something like decision trees (which can also help you find highly predictive values as Kodiologist suggested), KNN, SVMs, or some other model that can handle non-linear relationships.

Answer 3

Generally, in case your model is biased you need to come up with a more complex model -- one way to achieve this is to add more (hopefully good) features as been pointed out previously. But this can be a no easy matter and probably depends on your domain expertise. What you, however, could try out of the box is features interaction. Essentially, you are not limited in choosing the order of interactions/number of interacting variables and then could compare different models produced in such a way. Hope this makes sence

Answer 4

As some others have suggested, moving to a more complex algorithm may provide an increase in accuracy. If you were to stick with logistic regression, perhaps oversampling your smaller cohort may help to enrich your predictive performance. This could be achieved using a weighted logistic regression weighted logistic regression. Thankfully social scientists have implemented many of these algorithms in R and SAS.

Another strategy could be to use either forward or backward feature selection.

Ultimately, my money would be to move to more complex models, and would give particular attention to random forests and boosting.