For linear regression and many machine learning models, we use the same performance metric during the training and testing stage. For example, during the training stage, our machine learning algorithm is minimizing the mean-square-error and this is our performance metric. When we have got our model parameters, we apply our model to an independent test data set, on which, we look at the same performance metric, the mean-square-error. Almost always, the training performance is better than the test performance on that same performance metric.
However, I found that when we run logistic regressions, like in
sklearn, the objective function that is used to get those parameters is maximizing log likelihood, which is no problem, as it is just one specific performance metric. However, the problem is: after you get the parameters, then you apply the model to a test data set, people are often looking at other metrics. For example, AUC, accuracy. Then here comes my question: as the metric during the training and testing stage are different, how this should help us to tune the model?
To be specific, you train the logistic regression using loglikelihood. After you finished the training, you can still compute AUC of this model on the training data set and we denote this score as $M_1$. Then you apply your model to a test data set and get AUC performance on the test data set, denoted by $M_2$. Then should I expected $M_1$ is better than $M_2$? I don't think so, as $M_1$ is not the objective to optimize during the training stage. I've also observed this phenomenon in some data set. Why not just use AUC or accuracy rate during the training stage as the objective function to be optimized by the algorithm, if in the end, we will be using AUC or accuracy rate as our performance metric?