When we have a parametric model, we can use many procedure to estimate the parameters in the model, i.e., obtain many different estimators. We usually focus on the set of consistent estimators. For some reason, people also favors unbiased estimators. However, when the data set is small, then the bias-variance tradeoff may play a role, e.g., if an unbiased estimator has large variance, then the chance that this unbiased estimator is close to the true parameter might be less than a biased estimator with less variance.

My question is the following: all the above logic is based on the premise that the parametric model is correct, i.e., the data does come from the model and the only thing we need to do is to find the true parameter. However, there are situations when the model is misspecified (the famous quote "all models are wrong"). I am wondering when the model is wrong (so I guess in this case, the "true parameter" is meaningless), then

(1) When those "consistent" estimator converges, they converge to what? Most estimator we considered are extreme estimators, i.e., they are the solution of some optimization problem. So abstract from the statistical background, these estimators (solution to optimization problem) often converge (under some conditions on the objective function of the optimization problem), but then bring the statistical background here, what is that limit of a sequence of estimators? I vaguely remembered that for MLE, if the model is misspecified, the limit still has a meaning but I forget what it is.

(2) when the data set is small, then will the bias-variance tradeoff still apply here? I think bias-variance tradeoff makes sense only when there is a notion of true parameter, i.e., the model is correct. Because, if the model is wrong, i.e., the notion of true parameter is meaningless, then it is really hard to say like "the unbiased estimator with large variance may have less chance of being close to the true parameter than a biased estimator with less variance". So I guess in this case, i.e., when the model is wrong, the only way to judge estimator (I should say to judge the model, since if the model is wrong, then the estimator of the parameters....) is to run cross validation. Since in practice, you never know if your model is correct or wrong and you also don't know how wrong your model is (since you don't know the truth), then all those properties of estimator are kind of "toys in the theoretical world" (I am not intend to be offensive, in fact, I love statistical theory and pure math a lot, but here I just try to play the devil part). So I guess, in practice, cross validation is probably the best to judge a model in terms of prediction. But, in some cases, people not only want predicting, but also want to explain the phenomenon or making intervention based on the model. In this case, you have to look at the value of the estimators of the parameters. This is where I feel uncomfortable, since you never know whether the model is correct, and hence not sure if these estimators are meaningful, then how do you know your explanation and intervention based on reading the value of those estimators is correct? And you cannot even assign a confidence interval about this correctness.

  • $\begingroup$ Keep in mind that estimation theory doesn't apply just to model parameters, but also to features of a population such as its mean. You can sensibly talk about things like the bias of an estimator for a population mean only assuming that the mean exists and is finite, which is a much weaker assumption than a parametric model. And the median doesn't even require an assumption of existence, since every random variable has a median. $\endgroup$ – Kodiologist Jun 28 '16 at 21:23
  • $\begingroup$ Great topic! But I suggest you try to focus your questions more sharply and highlight them so that they would not disappear from sight in this long piece. Also, take a look at Francis X. Diebold's blog post on predictive consistency, "Causality and T-Consistency vs. Correlation and P-Consistency", and pay attention to the links in the post. They should be relevant to your question. $\endgroup$ – Richard Hardy Jun 29 '16 at 8:43
  • $\begingroup$ @RichardHardy thank you! What kind of key words you think I should tag this question? When I first post this question, I couldn't find suitable tags for it (or it may be also that I didn't know those concepts very clear) $\endgroup$ – KevinKim Jun 29 '16 at 14:02
  • $\begingroup$ @Kodiologist I agree. $\endgroup$ – KevinKim Jun 29 '16 at 14:10

Assume the data generating process is bounded iid With common density $p_e$ to keep things simple. Also assume a probability model defined as a set of densities $\{f(x;\theta)\}$ such that the logarithm of a particular density in the model is a continuous function of its parameters. It follows from the uniform law of large numbers that even if the model does not contain $p_e$ the negative normalized log likelihood $-(1/n)\sum_{i=1}^n \log f(X_i;\theta)$ still converges wp1 to the expected value of $-\log f(X;\theta)$ where the expectation is taken With respect to $p_e$. A strict local minimizer of the negative normalized log likelihood then converges to a strict local minimizer of the expected value of $-\log f(X;\theta)$ with respect to $p_e$ using Lemme 3 of Amemiya 1973 Econometrica. That is the quasi maximum likelihood estimator is consistent and minimizes the cross entropy between the best approximating density in the model and the data generating process.In the special case where the model is correctly specified the best approximating density is the data generating process density $p_e$.

I don't think the bias variance discussion Is meaningful in presence of model misspecification.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.