When we have a parametric model, we can use many procedure to estimate the parameters in the model, i.e., obtain many different estimators. We usually focus on the set of consistent estimators. For some reason, people also favors unbiased estimators. However, when the data set is small, then the bias-variance tradeoff may play a role, e.g., if an unbiased estimator has large variance, then the chance that this unbiased estimator is close to the true parameter might be less than a biased estimator with less variance.
My question is the following: all the above logic is based on the premise that the parametric model is correct, i.e., the data does come from the model and the only thing we need to do is to find the true parameter. However, there are situations when the model is misspecified (the famous quote "all models are wrong"). I am wondering when the model is wrong (so I guess in this case, the "true parameter" is meaningless), then
(1) When those "consistent" estimator converges, they converge to what? Most estimator we considered are extreme estimators, i.e., they are the solution of some optimization problem. So abstract from the statistical background, these estimators (solution to optimization problem) often converge (under some conditions on the objective function of the optimization problem), but then bring the statistical background here, what is that limit of a sequence of estimators? I vaguely remembered that for MLE, if the model is misspecified, the limit still has a meaning but I forget what it is.
(2) when the data set is small, then will the bias-variance tradeoff still apply here? I think bias-variance tradeoff makes sense only when there is a notion of true parameter, i.e., the model is correct. Because, if the model is wrong, i.e., the notion of true parameter is meaningless, then it is really hard to say like "the unbiased estimator with large variance may have less chance of being close to the true parameter than a biased estimator with less variance". So I guess in this case, i.e., when the model is wrong, the only way to judge estimator (I should say to judge the model, since if the model is wrong, then the estimator of the parameters....) is to run cross validation. Since in practice, you never know if your model is correct or wrong and you also don't know how wrong your model is (since you don't know the truth), then all those properties of estimator are kind of "toys in the theoretical world" (I am not intend to be offensive, in fact, I love statistical theory and pure math a lot, but here I just try to play the devil part). So I guess, in practice, cross validation is probably the best to judge a model in terms of prediction. But, in some cases, people not only want predicting, but also want to explain the phenomenon or making intervention based on the model. In this case, you have to look at the value of the estimators of the parameters. This is where I feel uncomfortable, since you never know whether the model is correct, and hence not sure if these estimators are meaningful, then how do you know your explanation and intervention based on reading the value of those estimators is correct? And you cannot even assign a confidence interval about this correctness.
