# Logistic regression does not seem to maximize model accuracy

I'm using gradient descent to train my logistic regression model for a classification task. However, I notice that the accuracy of my model (using a boundary threshold of 0.5 to classify each sample) peaks at some intermediate iteration, after which it decreases until the point of convergence.

My understanding of this phenomenon is that logistic regression's primary aim is to maximize the joint likelihood of the data (by maximizing the log-likelihood). However, this does not guarantee that the accuracy is maximized.

For example, let's say with have 2 data points, whose ground-truth responses are both 1:

• At some intermediate iteration, their predicted probabilities are 0.51 and 0.52, which gives a joint probability of 0.51 * 0.52 = 0.2652. The model accuracy at this point is 1, since both data points are classified correctly from their higher-than-0.5 probabilities.

• At convergence, their predicted probabilities are 0.49 and 0.99, which gives a joint probability of 0.49 * 0.99 = 0.4851. Therefore, the logistic regression does its job of maximizing the joint probability. However, the model accuracy is now only 0.5, since only one data point (0.99) is correctly classified.

In short, it seems to me that logistic regression will generally increases the accuracy of the model, but this is not guaranteed.

1. Please let me know if my understanding of this is correct.

2. If so, can the logistic regression be modified to optimize for accuracy directly? Are there other statistical methods that directly optimize the accuracy of the model?

• You're right that it doesn't. That is not a problem however. The problem is that accuracy is a poor measure of performance for a binary classification problem. Nov 17, 2019 at 7:49
• Similar questions have been asked a lot. Going through some of the posts in this list will help a lot. Nov 17, 2019 at 13:34

• It predicts probabilities, so you cannot calculate accuracy from the outputs, unless you decide on some decision rule to make clarifications. Using $$\hat y > 0.5$$ is just one such rule.