Parameter Estimation vs Inference Error

Question

I am having trouble reconciling (or maybe even understanding properly) the following issues:

We have a dataset. We hypothesize a functional form for probability density. Then we estimate the parameters of our proposed model with some procedure, such as MLE, MAP, OLS, or other more esoteric ones.

At this point the estimation procedure is in the cross-hairs. What is the objective? To lower the KL divergence of the whole model to the true model (MLE), to lower the KL divergence of the whole model to the true model by taking into consideration some subjective view (MAP), or are we actually trying to estimate each parameter of the model in an unbiased fashion with the lowest possible variance or maybe even minimum mean square error on one of the parameters? How can one goes on to do inference when the objective of estimation doesn't coincide with the errors/losses one will have when applying this model?

Moreover it seems we ignore the model error completely in this picture. In machine learning literature, there are at least some derived bounds, but in statistics literature this issue seems to be ignored (or at least at my level of reading).

In addition to all above, the Information Criteria tools available to us, such as AIC, BIC, TIC, MIC, etc. are targeted towards KL divergence, rather than our own objective. This leaves us with cross-validation to estimate the expected loss.

Thanks in advance.

Answer 1

Your goals of the analysis ought to coincide with how you are selecting a model. If you are trying to understand how some set of variables affect your response variable or if you are interested in how $X_1$ effects $Y$ while controlling for the effects of $X_2,...X_p$, than you are interested in minimizing estimation error of your parameters (sidenote: in GLM the MLE estimates of your parameters will minimize RSS). If you have very specific hypothesis in mind, than you may entirely skip a model selection here.

If the goal is to accurately predict $Y$, without caring about what or how many variables are in your model, than your model fitting procedures (as well as model selection procedures) ought to consider this. Methods such as ridge regression, LASSO, or elastic-net regression introduce bias in the estimation of your parameters while having smaller variance. These methods are widely used when the goal is to accurately predict $Y$.

Even when you do cross-validation you should consider your goals! Do you want to accurately classify and how close your classification is to the true value doesn't matter? Then minimize misclassification rate. Do you want close predictions but getting it on the nose isn't that relevant? Then minimize mean absolute error. Do you want to give an extra penalty to really far predictions in the previous scenario? Then minimize mean square error instead of mean absolute error.