I'm using gradient descent to train my logistic regression model for a classification task. However, I notice that the accuracy of my model (using a boundary threshold of 0.5 to classify each sample) peaks at some intermediate iteration, after which it decreases until the point of convergence.

My understanding of this phenomenon is that logistic regression's primary aim is to maximize the joint likelihood of the data (by maximizing the log-likelihood). However, this does not guarantee that the accuracy is maximized.

For example, let's say with have 2 data points, whose ground-truth responses are both 1:

  • At some intermediate iteration, their predicted probabilities are 0.51 and 0.52, which gives a joint probability of 0.51 * 0.52 = 0.2652. The model accuracy at this point is 1, since both data points are classified correctly from their higher-than-0.5 probabilities.

  • At convergence, their predicted probabilities are 0.49 and 0.99, which gives a joint probability of 0.49 * 0.99 = 0.4851. Therefore, the logistic regression does its job of maximizing the joint probability. However, the model accuracy is now only 0.5, since only one data point (0.99) is correctly classified.

In short, it seems to me that logistic regression will generally increases the accuracy of the model, but this is not guaranteed.

  1. Please let me know if my understanding of this is correct.

  2. If so, can the logistic regression be modified to optimize for accuracy directly? Are there other statistical methods that directly optimize the accuracy of the model?

  • 2
    $\begingroup$ You're right that it doesn't. That is not a problem however. The problem is that accuracy is a poor measure of performance for a binary classification problem. $\endgroup$ – Frans Rodenburg Nov 17 '19 at 7:49
  • $\begingroup$ Similar questions have been asked a lot. Going through some of the posts in this list will help a lot. $\endgroup$ – kjetil b halvorsen Nov 17 '19 at 13:34
  • Logistic regression is trained by minimizing logistic loss (or maximizing likelihood, what is equivalent), so it will be guaranteed to minimize this loss, rather then something else.
  • It predicts probabilities, so you cannot calculate accuracy from the outputs, unless you decide on some decision rule to make clarifications. Using $\hat y > 0.5$ is just one such rule.
  • When you "round" the predicted probabilities to turn them to classifications, you loose precision, so you can optimize loss, while accuracy would keep the same.
  • Accuracy is not the best metric to care about.
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.