3
$\begingroup$

I am using a LR model and its got 80% prediction accuracy on test data. For the 20% where it has predicted wrongly, I know the right answer of course. I wonder if there is some optimisation method where I can take the trained LR model and iterate the weights inside the model until the 20% failures are 10% for example. I could use an Evolutionary Strategy maybe. Has anybody done that with success or would this be a bad idea because it would lead to an overfit model?

I tried what I suggested and got a small improvement in accuracy but not much, less than 0.5 percent. This was with using hillclimbing to improve the weights from the model. I am trying with genetic algorithms next. I am using binary classification with a balanced dataset so threshhold 0.5 should be good, as far as i know.

My data is balanced, so accuracy is a good measure of model quality, as far as i know.

I am using Python sklearn implementation of LR , by the way. I already optimised the two parameters "C" and "penalty".

Improve this question
$\endgroup$
13
  • 8
    $\begingroup$ Logistic regression is optimized, with respect to likelihood, but not with respect to accuracy. Accuracy is not the best way to measure model quality. The proposed procedure will result in a bogus model. It sounds like OP has two unspoken questions: (1) how to measure model quality and (2) how to improve a logistic regression model. $\endgroup$
    – Sycorax ♦
    Commented Jan 12, 2022 at 13:48
  • $\begingroup$ Having selected that model (inputs etc), you could retrain the model with train and test data combined. (You would no longer have any data to assess how accurate the model was on unseen data..) $\endgroup$
    – seanv507
    Commented Jan 12, 2022 at 13:52
  • 6
    $\begingroup$ What is being suggested is not based on good statistical principles. The gold standard optimality criterion is the likelihood; no two-step procedure required. But a lot of confusion has been caused by applying classification error measures to non-classification problems. Probabilities are not "correct" or "incorrect". $\endgroup$
    – Frank Harrell
    Commented Jan 12, 2022 at 14:23
  • 1
    $\begingroup$ What you're suggesting is evocative of, if not equivalent to, boosting, where you train again with weights applied to give more weight to bad misses; I have added the boosting tag. While I do not have expertise in boosting methods to expand on this comment, it seems that you can apply this to the proper scoring rules discussed in the other comments, such a the likelihood (equivalent to crossentropy loss and negative log likelihood loss), not just with hard classifications. $\endgroup$
    – Dave
    Commented Jan 12, 2022 at 14:37
  • 1
    $\begingroup$ You might want to try a non-linear version of LR, such as Kernel Logistic Regression if it is a non-linear problem. I very much like Frank Harrell's 'Probabilities are not "correct" or "incorrect".' - nicely put! There are circumstances where you may be interested in accuracy (see many applications of SVMs), but if you are using LR you probably want probabilities, so you should use a criterion that measures the quality of the estimates of probability, not accuracy. $\endgroup$
    – Dikran Marsupial
    Commented Jan 12, 2022 at 16:07

1 Answer 1

Reset to default
7
$\begingroup$
  • As noticed by others, accuracy is not the best metric for judging quality of the model.
  • Logistic regression predicts probabilities, so to calculate accuracy you must have used some threshold for making hard classifications. If you used the "default" $p>0.5$, it is not necessarily an optimal choice. There are many methods for picking the threshold. This is something you could tune.
  • As about the parameters of the model, you already found the optimal parameters given the data, model, and the hyperparameters, so there is nothing to improve further. If the results are not satisfactory, you can tune the hyperparameters, or try different model (either using another model type altogether, such as , or attempting further refinement of logistic regression, perhaps using a richer set of features).
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.