15
$\begingroup$

I have been studying statistics from many books for the last 3 years, and thanks to this site I learned a lot. Nevertheless one fundamental question still remains unanswered for me. It may have a very simple or a very difficult answer, but I know for sure it requires some deep understanding of statistics.

When fitting a model to data, be it a frequentist or a Bayesian approach, we propose a model, which may consist of a functional form for likelihood, a prior, or a kernel (non-parametric), etc. The issue is any model fits a sample with some level of goodness. One can always find a better or worse model compared to what's currently at hand. At some point we stop and start drawing conclusions, generalize to population parameters, report confidence intervals, calculate risk, etc. Hence, whatever conclusion we draw is always conditional on the model we decided to settle with. Even if we are using tools to estimate the expected KL distance such as AIC, MDL, etc., it doesn't say anything about where we stand on an absolute basis, but just improves our estimation on a relative basis. It seems there is no objectivity as the model error is completely ignored.

Now suppose that we would like to define a step by step procedure to apply to any data set when building models. What should we specify as a stopping rule? Can we at least bound the model error which will give us an objective stopping point (this is different than stopping training using a validation sample, since it also gives a stopping point within the evaluated model class rather than w.r.t. the true DGP)?

$\endgroup$
  • 1
    $\begingroup$ I think you should add other tags to the question than just inference, e.g. some modeling tags and model-selection. I think that might be relevant for this is also Occam's razor. Here is also a paper that discusses it w.r.t. Bayesian modeling. $\endgroup$ – Gumeo Oct 27 '15 at 14:26
  • $\begingroup$ Sometimes you build a particular model because it is particularly good for estimating certain parameters, not because you think the overall distribution is accurate (see M-estimation, generalized estimating equations) etc. So, if you really care about a decent estimate of location, you may be better off with a wrong model but one that is not easily thrown by noise (for your parameter of interest). In general, see Robust Estimation. $\endgroup$ – user75138 Oct 27 '15 at 19:46
  • $\begingroup$ Very interesting question. Just a comment that at least in the Bayesian setting one than also raises the question of averaging over the plausible subset of models rather than choosing one. I am not sure of any theoretical way to the OP's question and I guess practically it is done by whether the picked model is good enough for the problem we are trying to solve. Perhaps we need model selection by MCMC methods or something like that! I can envision a nested MCMC approach for this... $\endgroup$ – Luca Oct 30 '15 at 10:09
  • $\begingroup$ @Luca This has been done. However the problem remains as the space of models defined by the Bayesian prior may or may not contain the true model. Even if it did the model error is still there, that is the error of the average model with respect to the true DGP. $\endgroup$ – Cagdas Ozgenc Oct 30 '15 at 10:14
  • 1
    $\begingroup$ +1 for the question. In large part, the concerns are philosophical or epistemiological, i.e., not only "what do we know and how do we know it" but "what can we know and how can we know it?" As the physicist Richard Feynman said, "It is impossible to find an answer which someday will not be found to be wrong." In other words and unless you are religious, there is reasonable doubt whether a univocal, eternal ground truth exists on which to anchor anything.. $\endgroup$ – Mike Hunter Nov 4 '15 at 11:44
12
+50
$\begingroup$

Unfortunately, this question does not have a good answer. You can choose the best model based on the fact that it minimizes absolute error, squared error, maximizes likelihood, using some criteria that penalizes likelihood (e.g. AIC, BIC) to mention just a few most common choices. The problem is that neither of those criteria will let you choose the objectively best model, but rather the best from which you compared. Another problem is that while optimizing you can always end up in some local maximum/minimum. Yet another problem is that your choice of criteria for model selection is subjective. In many cases you consciously, or semi-consciously, make a decision on what you are interested in and choose the criteria based on this. For example, using BIC rather than AIC leads to more parsimonious models, with less parameters. Usually, for modeling you are interested in more parsimonious models that lead to some general conclusions about the universe, while for predicting it doesn't have to be so and sometimes more complicated model can have better predictive power (but does not have to and often it does not). In yet other cases, sometimes more complicated models are preferred for practical reasons, for example while estimating Bayesian model with MCMC, model with hierarchical hyperpriors can behave better in simulation than the simpler one. On the other hand, generally we are afraid of overfitting and the simpler model has the lower risk of overfitting, so it is a safer choice. Nice example for this is a automatic stepwise model selection that is generally not recommended because it easily leads to overfitted and biased estimates. There is also a philosophical argument, Occam's razor, that the simplest model is the preferred one. Notice also, that we are discussing here comparing different models, while in real life situations it also can be so that using different statistical tools can lead to different results - so there is an additional layer of choosing the method!

All this leads to sad, but entertaining, fact that we can never be sure. We start with uncertainty, use methods to deal with it and we end up with uncertanity. This may be paradoxical, but recall that we use statistics because we believe that world is uncertain and probabilistic (otherwise we would choose a career of prophets), so how could we possibly end up with different conclusions? There is no objective stopping rule, there are multiple possible models, all of them are wrong (sorry for the cliché!) because they try to simplify the complicated (constantly changing and probabilistic) reality. We find some of them more useful than others for our purposes and sometimes we do find different models useful for different purposes. You can go to the very bottom to notice that in many cases we make models of unknown $\theta$'s, that in most cases can never be known, or even do not exist (does a population has any $\mu$ for age?). Most models do not even try to describe the reality but rather provide abstractions and generalizations, so they cannot be "right", or "correct".

You can go even deeper and find out that there is no such a thing as "probability" in the reality - it is just some approximation of uncertainty around us and there are also alternative ways of approximating it like e.g. fuzzy logic (see Kosko, 1993 for discussion). Even the very basic tools and theorems that our methods are grounded on are approximations and are not the only ones that are possible. We simply cannot be certain in such a setup.

The stopping rule that you are looking for is always problem-specific and subjective, i.e. based on so called professional judgment. By the way, there are lots of research examples that have shown that professionals are often not better and sometimes even worse in their judgment than laypeople (e.g. revived in papers and books by Daniel Kahneman), while being more prone to overconfidence (this is actually an argument on why we should not try to be "sure" about our models).


Kosko, B. (1993). Fuzzy thinking: the new science of fuzzy logic. New York: Hyperion.

$\endgroup$
  • 1
    $\begingroup$ If there was "true DGP" and we were able to measure it, we wouldn't need statistics, we would measure it directly. It does not exist, so we create approximations of reality. DGP is abstract and not true and in many cases has not much to do with how the data was really generated. I want to point that I'm not saying that stats are subjective and useless, but rather that it is an approximation and there is no such things as "true DGP" or "true $\mu$" etc. $\endgroup$ – Tim Oct 31 '15 at 14:45
  • 1
    $\begingroup$ The claim is true when its assumptions are satisfied (e.g. we are given a fixed sample, which is true in practice). Taken out of context and with violations of assumptions, it can of course be made false. $\endgroup$ – Richard Hardy Nov 4 '15 at 12:46
  • 1
    $\begingroup$ @CagdasOzgenc is someone has a methodology to create a model that perfectly reflects reality, than there is no need to for stopping rule, or for measuring the model error - the model is perfect by definition. If you know the rules for building such model, there is no need to measure divergence of your model from true DGP, since knowing the true DGP just utilize such knowledge. On the other hand, if your model is a simplification based on the data you have, then the general stats rules apply, as described in my answer. $\endgroup$ – Tim Nov 4 '15 at 13:20
  • 1
    $\begingroup$ @CagdasOzgenc still, if you know the "truth", than the stopping rule is simple: stop when your model fits the "truth". If you do not know what is the truth, than "all models are [equally] wrong..." and you have to use statistics. If you don't know, you cannot measure divergence from it. $\endgroup$ – Tim Nov 4 '15 at 13:32
  • 1
    $\begingroup$ @Luca It means very much, but it is abstract. $\endgroup$ – Tim Nov 5 '15 at 14:42
4
$\begingroup$

There is a whole field called nonparametric statistics that avoids the use of strong models. However, your concern about fitting models, per se, is valid. Unfortunately there is no mechanical procedure to fitting models that would be universally accepted as "optimal". For example, if you want to define the model that maximizes the likelihood of your data, then you will be led to the empirical distribution function.

However, we usually have some background assumptions and constraints, such as continuous with finite first and second moments. For cases like these, one approach is to choose a measure like Shannon Differential Entropy and maximize it over the space of continuous distributions that satisfy your boundary constraints.

What I'd like to point out is that if you don't just want to default to the ECDF, then you'll need to add assumptions, beyond the data, to get there, and that requires subject matter expertise, and, yes, the dreaded.....professional judgement

So, is there a guaranteed stopping point to modeling...the answer is no. Is there a good enough place to stop? Generally, yes, but that point will depend on more than just the data and some statistical desiderata, you're usually going to take into account the risks of different errors, the technical limitations to implementing the models, and the robustness of its estimates, etc.

As @Luca pointed out, you can always average over a class of models, but, as you rightly pointed out, that will just push the question up to the next level of hyperparameters. Unfortunately, we seem to live within an infinitely layered onion...in both directions!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.