I'm using gradient descent to train my logistic regression model for a classification task. However, I notice that the accuracy of my model (using a boundary threshold of 0.5 to classify each sample) peaks at some intermediate iteration, after which it decreases until the point of convergence.
My understanding of this phenomenon is that logistic regression's primary aim is to maximize the joint likelihood of the data (by maximizing the log-likelihood). However, this does not guarantee that the accuracy is maximized.
For example, let's say with have 2 data points, whose ground-truth responses are both 1:
At some intermediate iteration, their predicted probabilities are 0.51 and 0.52, which gives a joint probability of 0.51 * 0.52 = 0.2652. The model accuracy at this point is 1, since both data points are classified correctly from their higher-than-0.5 probabilities.
At convergence, their predicted probabilities are 0.49 and 0.99, which gives a joint probability of 0.49 * 0.99 = 0.4851. Therefore, the logistic regression does its job of maximizing the joint probability. However, the model accuracy is now only 0.5, since only one data point (0.99) is correctly classified.
In short, it seems to me that logistic regression will generally increases the accuracy of the model, but this is not guaranteed.
Please let me know if my understanding of this is correct.
If so, can the logistic regression be modified to optimize for accuracy directly? Are there other statistical methods that directly optimize the accuracy of the model?