When fitting a parametric model to a data set assuming that our selected model class contains the truth, what performance metric should be used so that parameters converge to the truth as sample size increase?

I am reading that sometimes when going after out of sample prediction accuracy we end up with inconsistent estimation. This means some metrics will be better suited for model estimation and some are for prediction. My understanding is that AIC versus BIC dilemma is about this issue. AIC navigates towards better prediction but not convergent, and BIC navigates towards convergent but suboptimal prediction.

  1. Given all the above, how do you make sure that you have the right metric so that you don't get inconsistency?

  2. Does it matter to be inconsistent if out of sample prediction accuracy is OK?

  3. When the truth is not in our search space is there an ideal metric so that our model is as close to the truth as possible? KL divergence is a good candidate (it is not a metric but comparing against the true model a good relative measurement), but what about prediction in that case?

  4. When model is not probabilistic KL is out of scope. How do you compare the models in such cases?

Thank you, and sorry if I haven't phrased some technicalities as accurate as possible. And I know it is a handful of questions, feel free to answer any.

  • $\begingroup$ This question seems to be mixing model selection procedures (comparing a set of candidate models) with estimation procedures (finding the best possible fit to data given a particular model). $\endgroup$ – Andrew M Oct 10 '14 at 1:40
  • $\begingroup$ @AndrewM - It is because your understanding of model selection is limited to number of paramaters involved or the functional form. In reality model selection also involves the information theoretic content of each paramater, which is determined during the estimation procedure. $\endgroup$ – Cagdas Ozgenc Oct 10 '14 at 6:06

It looks like you are referring to information criteria AIC and BIC in this example according to the tags you have provided. I would like to note that for fixed dimension parameters, these metrics "metrize the same topology" as maximum likelihood. KL divergence, total variance, and Hellinger distance are all examples of metrics that measure the probablistic distance between any two probability mass functions.

  1. For plain vanilla parametric estimation, nearly all of these give identical topologies.

  2. "Out of sample prediction accuracy": do you mean forecasting or external validation? None of these metrics are functions of the data per se, and you might not care about the consistency of parametric estimation, but how well the parameters (however poorly estimated) perform in prediction using external data. It might be useful to focus your attention on performance characteristics in a model validation setting.

  3. No, there is neither existence of uniqueness guaranteed for such parameters. For example, binary prediction with logistic regression and perfectly ordered outcomes. The odds ratio lives on the boundary of the parameter space. Hence the likelihood is singular. However, you have generated a valid prediction model. You predict risk 1 when risk falls beyond a certain threshold. In a lot of ways, this means that a variety of binary prediction models (decision trees, nearest neighbor, etc) are giving you the same answer.

  4. Refer to answer 2.

| cite | improve this answer | |
  • 1
    $\begingroup$ I cannot attest to whether the info here is cutting edge or even correct... nothing is explained quite rigorously, which may explain the sort of confusing nature of this question. If you cannot get away from a model selection framework (for presummably a supervised learning problem), BIC and AIC have some applicability, but your referring to parameters suggest to me that likelihood based inference is valid. Prob metrics and the ICs are more useful when you begin thinking in the more general space of functionals. Cleaning up your question may get a more precise response. $\endgroup$ – AdamO Oct 8 '14 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.