# Can LogisticRegression be optimised?

I am using a LR model and its got 80% prediction accuracy on test data. For the 20% where it has predicted wrongly, I know the right answer of course. I wonder if there is some optimisation method where I can take the trained LR model and iterate the weights inside the model until the 20% failures are 10% for example. I could use an Evolutionary Strategy maybe. Has anybody done that with success or would this be a bad idea because it would lead to an overfit model?

I tried what I suggested and got a small improvement in accuracy but not much, less than 0.5 percent. This was with using hillclimbing to improve the weights from the model. I am trying with genetic algorithms next. I am using binary classification with a balanced dataset so threshhold 0.5 should be good, as far as i know.

My data is balanced, so accuracy is a good measure of model quality, as far as i know.

I am using Python sklearn implementation of LR , by the way. I already optimised the two parameters "C" and "penalty".

• Logistic regression is optimized, with respect to likelihood, but not with respect to accuracy. Accuracy is not the best way to measure model quality. The proposed procedure will result in a bogus model. It sounds like OP has two unspoken questions: (1) how to measure model quality and (2) how to improve a logistic regression model.
– Sycorax
Jan 12 at 13:48
• Having selected that model (inputs etc), you could retrain the model with train and test data combined. (You would no longer have any data to assess how accurate the model was on unseen data..) Jan 12 at 13:52
• What is being suggested is not based on good statistical principles. The gold standard optimality criterion is the likelihood; no two-step procedure required. But a lot of confusion has been caused by applying classification error measures to non-classification problems. Probabilities are not "correct" or "incorrect". Jan 12 at 14:23
• What you're suggesting is evocative of, if not equivalent to, boosting, where you train again with weights applied to give more weight to bad misses; I have added the boosting tag. While I do not have expertise in boosting methods to expand on this comment, it seems that you can apply this to the proper scoring rules discussed in the other comments, such a the likelihood (equivalent to crossentropy loss and negative log likelihood loss), not just with hard classifications.
– Dave
Jan 12 at 14:37
• You might want to try a non-linear version of LR, such as Kernel Logistic Regression if it is a non-linear problem. I very much like Frank Harrell's 'Probabilities are not "correct" or "incorrect".' - nicely put! There are circumstances where you may be interested in accuracy (see many applications of SVMs), but if you are using LR you probably want probabilities, so you should use a criterion that measures the quality of the estimates of probability, not accuracy. Jan 12 at 16:07

• Logistic regression predicts probabilities, so to calculate accuracy you must have used some threshold for making hard classifications. If you used the "default" $$p>0.5$$, it is not necessarily an optimal choice. There are many methods for picking the threshold. This is something you could tune.