Suitable metric for consistency of parametric models When fitting a parametric model to a data set assuming that our selected model class contains the truth, what performance metric should be used so that parameters converge to the truth as sample size increase?
I am reading that sometimes when going after out of sample prediction accuracy we end up with inconsistent estimation. This means some metrics will be better suited for model estimation and some are for prediction. My understanding is that AIC versus BIC dilemma is about this issue. AIC navigates towards better prediction but not convergent, and BIC navigates towards convergent but suboptimal prediction.


*

*Given all the above, how do you make sure that you have the right metric so that you don't get inconsistency?

*Does it matter to be inconsistent if out of sample prediction accuracy is OK?

*When the truth is not in our search space is there an ideal metric so that our model is as close to the truth as possible? KL divergence is a good candidate (it is not a metric but comparing against the true model a good relative measurement), but what about prediction in that case?

*When model is not probabilistic KL is out of scope. How do you compare the models in such cases?
Thank you, and sorry if I haven't phrased some technicalities as accurate as possible.
And I know it is a handful of questions, feel free to answer any.
 A: It looks like you are referring to information criteria AIC and BIC in this example according to the tags you have provided. I would like to note that for fixed dimension parameters, these metrics "metrize the same topology" as maximum likelihood. KL divergence, total variance, and Hellinger distance are all examples of metrics that measure the probablistic distance between any two probability mass functions. 


*

*For plain vanilla parametric estimation, nearly all of these give identical topologies.

*"Out of sample prediction accuracy": do you mean forecasting or external validation? None of these metrics are functions of the data per se, and you might not care about the consistency of parametric estimation, but how well the parameters (however poorly estimated) perform in prediction using external data. It might be useful to focus your attention on performance characteristics in a model validation setting.

*No, there is neither existence of uniqueness guaranteed for such parameters. For example, binary prediction with logistic regression and perfectly ordered outcomes. The odds ratio lives on the boundary of the parameter space. Hence the likelihood is singular. However, you have generated a valid prediction model. You predict risk 1 when risk falls beyond a certain threshold. In a lot of ways, this means that a variety of binary prediction models (decision trees, nearest neighbor, etc) are giving you the same answer.

*Refer to answer 2.
