Should the underlying logic of the model evaluation criterion match with the estimation procedure?

Question

Assume we have a parametric model with parameter $\beta$. We want to use this model to make predictions, e.g., credit card firm using some attributes $X$ of the customer to predict the default rate $Y$.

First, we need to estimate $\beta$ in our model. There are many ways to do the estimation. Let's say we are looking at extreme estimators, which are the solution of some optimization problem, e.g., Maximum Likelihood, Minimize SSE, etc.

After you estimate $\beta$, you want to apply the model (with $\beta$ being estimated from a specific estimation procedure you have used) to a test data set. Now we need to come up with an evaluation criterion to judge how well your model is. In the credit card example, a commonly used criterion is the misclassification rate, i.e., the average rate that your model prediction against the reality. So under this criterion, the model with smaller misclassification rate is considered as better.

Now, for the same model, but with two different estimation procedure (and hence two different values of $\beta$, which could be very different), then it is possible that the misclassification rates are different.

My question is: given the goal (i.e., the judging criterion of the model, in this example, misclassification rate), should we always use this or similar criterion as the objective function to derive our estimator for $\beta$? However, in practice, I see in many situations, people use MLE to run the estimation (I guess, because it is unbiased and consistent), but for model evaluation, they use a different criterion, e.g., misclassification rate.

I would feel more comfortable if the evaluation criterion is something like deviance if the estimation procedure is MLE, since deviance is equivalent to the log likelihood function.

Answer 1

I agree with your thinking. If you want to maximize predictive accuracy, then in general, when trying to minimize training error in order to select model parameters, you should use the same metric for training error that you'll use for test error. (You might want to do something other than just minimize training error, as in regularization, but deliberately mistmatching training-error and test-error metrics is probably not a good way to do this.)

So why do we see mismatches in practice? Partly it's because, as you indicate, estimating parameters according to the test-error metric may just be more difficult than using another metric. It may also be tradition and inertia. For example, in the case of logistic regression, people are used to fitting with MLE but evaluating predictions by discretizing the model outputs and using zero–one loss rather than using a proper scoring rule. Scoring rules are an obscure topic although logistic regression is a very popular technique.