Logistic regression does not seem to maximize model accuracy

Question

I'm using gradient descent to train my logistic regression model for a classification task. However, I notice that the accuracy of my model (using a boundary threshold of 0.5 to classify each sample) peaks at some intermediate iteration, after which it decreases until the point of convergence.

My understanding of this phenomenon is that logistic regression's primary aim is to maximize the joint likelihood of the data (by maximizing the log-likelihood). However, this does not guarantee that the accuracy is maximized.

For example, let's say with have 2 data points, whose ground-truth responses are both 1:

• At some intermediate iteration, their predicted probabilities are 0.51 and 0.52, which gives a joint probability of 0.51 * 0.52 = 0.2652. The model accuracy at this point is 1, since both data points are classified correctly from their higher-than-0.5 probabilities.

• At convergence, their predicted probabilities are 0.49 and 0.99, which gives a joint probability of 0.49 * 0.99 = 0.4851. Therefore, the logistic regression does its job of maximizing the joint probability. However, the model accuracy is now only 0.5, since only one data point (0.99) is correctly classified.

In short, it seems to me that logistic regression will generally increases the accuracy of the model, but this is not guaranteed.

1. Please let me know if my understanding of this is correct.

2. If so, can the logistic regression be modified to optimize for accuracy directly? Are there other statistical methods that directly optimize the accuracy of the model?

Answer 1

• Logistic regression is trained by minimizing logistic loss (or maximizing likelihood, what is equivalent), so it will be guaranteed to minimize this loss, rather then something else.
• It predicts probabilities, so you cannot calculate accuracy from the outputs, unless you decide on some decision rule to make clarifications. Using $\hat y > 0.5$ is just one such rule.
• When you "round" the predicted probabilities to turn them to classifications, you loose precision, so you can optimize loss, while accuracy would keep the same.
• Accuracy is not the best metric to care about.

Answer 2

Following your points:

1. Yes, it is correct that the logistic regression does not generally optimises the accuracy. I would only say that the logistic regression estimates the conditional likelihood rather than the joint likelihood (this follows from the fact that the logistic regression ignores the marginal distribution of each class). If you want to maximise the whole joint likelihood, you should look into Linear Discrimination Analysis (LDA).
2. In some cases in theory one could construct a classifier that maximises the accuracy (at least asymptotically). One way to do this would be to define the risk of the classifier as the expected 0/1 loss. Then, one could try to find a Bayes classifier, i.e., the classifier that allows to attain the minimal risk (presumably, this classifier would be the most accurate). For example, one could show that in the domain with two balanced normally distributed classes with the same covariance matrix logistic regression is asymptotically Bayes optimal (see here).