Assume we have a parametric model with parameter $\beta$. We want to use this model to make predictions, e.g., credit card firm using some attributes $X$ of the customer to predict the default rate $Y$.
First, we need to estimate $\beta$ in our model. There are many ways to do the estimation. Let's say we are looking at extreme estimators, which are the solution of some optimization problem, e.g., Maximum Likelihood, Minimize SSE, etc.
After you estimate $\beta$, you want to apply the model (with $\beta$ being estimated from a specific estimation procedure you have used) to a test data set. Now we need to come up with an evaluation criterion to judge how well your model is. In the credit card example, a commonly used criterion is the misclassification rate, i.e., the average rate that your model prediction against the reality. So under this criterion, the model with smaller misclassification rate is considered as better.
Now, for the same model, but with two different estimation procedure (and hence two different values of $\beta$, which could be very different), then it is possible that the misclassification rates are different.
My question is: given the goal (i.e., the judging criterion of the model, in this example, misclassification rate), should we always use this or similar criterion as the objective function to derive our estimator for $\beta$? However, in practice, I see in many situations, people use MLE to run the estimation (I guess, because it is unbiased and consistent), but for model evaluation, they use a different criterion, e.g., misclassification rate.
I would feel more comfortable if the evaluation criterion is something like deviance if the estimation procedure is MLE, since deviance is equivalent to the log likelihood function.