The performance metric used in prediction is different from the objective function to train the model

For linear regression and many machine learning models, we use the same performance metric during the training and testing stage. For example, during the training stage, our machine learning algorithm is minimizing the mean-square-error and this is our performance metric. When we have got our model parameters, we apply our model to an independent test data set, on which, we look at the same performance metric, the mean-square-error. Almost always, the training performance is better than the test performance on that same performance metric.

However, I found that when we run logistic regressions, like in sklearn, the objective function that is used to get those parameters is maximizing log likelihood, which is no problem, as it is just one specific performance metric. However, the problem is: after you get the parameters, then you apply the model to a test data set, people are often looking at other metrics. For example, AUC, accuracy. Then here comes my question: as the metric during the training and testing stage are different, how this should help us to tune the model?

To be specific, you train the logistic regression using loglikelihood. After you finished the training, you can still compute AUC of this model on the training data set and we denote this score as $$M_1$$. Then you apply your model to a test data set and get AUC performance on the test data set, denoted by $$M_2$$. Then should I expected $$M_1$$ is better than $$M_2$$? I don't think so, as $$M_1$$ is not the objective to optimize during the training stage. I've also observed this phenomenon in some data set. Why not just use AUC or accuracy rate during the training stage as the objective function to be optimized by the algorithm, if in the end, we will be using AUC or accuracy rate as our performance metric?

A classifier's actual output isn't a sample's class but the probability of a sample belonging to each class.

However, we usually care about the classes, so we check which class has the highest probability and consider that to be the model's prediction. Then let's say we want to see how many predictions the classifier got right, so we'll measure the accuracy of the predicted classes with the actual ones.

The probability that the classifier assigns to each class can be viewed as how confident it is for a prediction. For example a class assigned a $$99\%$$ probability means that the classifier is very sure the sample belongs to that class.

During training, though, we want our classifier make correct predictions with high confidence and wrong predictions with low confidence. We want to penalize our model more if it is very confident about a wrong prediction. For this reason we need a continuous scale for measuring the model's performance during training.

Example

Suppose you want to train a classifier to predict a person's sex from his height. Intuitively, tall people will be men and shorter people will be women.

Because the classifier will many more samples of males with high heights, it will learn to produce a high probability for men in these heights (i.e. high confidence that tall people are males).

Likewise, it will see more samples of females with low heights and will produce a high probability for them (i.e. high confidence that short people are females).

Finally, it will see a balanced number of males and females in intermediate heights. This will cause it to be unsure (i.e. low confidence - probabilities around $$50\%$$) for people with an average height.

• Theoretically speaking, can we also add another parameter, which is the threshold on the probabilities when we train the model? So when we train the model, we can use, say, accuracy as the objective function we are trying to optimize. If in the end, accuracy rate is THE metric for a specific problem (put aside those imbalance data issue), then will the above method be better in principle? – ftxx Jan 16 at 3:00
• The objective function should be continuous and in some cases differentiable. It needs to show you how close you are to your target, rather than if you got it right or not. Otherwise you wouldn't know in which way to update the model parameters to reduce the loss and make your model better. You can also read this article which elaborates a bit more. – Djib2011 Jan 16 at 8:49