When fitting a parametric model to a data set assuming that our selected model class contains the truth, what performance metric should be used so that parameters converge to the truth as sample size increase?
I am reading that sometimes when going after out of sample prediction accuracy we end up with inconsistent estimation. This means some metrics will be better suited for model estimation and some are for prediction. My understanding is that AIC versus BIC dilemma is about this issue. AIC navigates towards better prediction but not convergent, and BIC navigates towards convergent but suboptimal prediction.
Given all the above, how do you make sure that you have the right metric so that you don't get inconsistency?
Does it matter to be inconsistent if out of sample prediction accuracy is OK?
When the truth is not in our search space is there an ideal metric so that our model is as close to the truth as possible? KL divergence is a good candidate (it is not a metric but comparing against the true model a good relative measurement), but what about prediction in that case?
When model is not probabilistic KL is out of scope. How do you compare the models in such cases?
Thank you, and sorry if I haven't phrased some technicalities as accurate as possible. And I know it is a handful of questions, feel free to answer any.