11
$\begingroup$

When attempting to model various phenomena, in practice, our model will usually be "wrong", but can still provide us with useful insights/predictions.

What methodologies can we follow to help us balance the ability to create a useful/informative mathematical representation of our phenomena with the "trueness" of our model?

"Trueness" meaning the relaxation of assumptions such as i.i.d/stationary/ergodic observations for probabilistic models, differentiability/continuity for dynamic models/optimization, etc.

Is a course of action to just assume that our phenomena don't really follow our assumptions (even the ones about the type of model, e.g. using probabilistic models when we don't necessarily know if our data is generated in this way), take predictive power as dogma (within reason), and whatever maximizes this is the model we choose? But what then if our there is not sufficient data to validate the the predictive strength such as in the case of rare events, etc.?

I realize that this question is similar, but, and correct me if I'm wrong, this post seems more ideology-based whereas I'm looking more for a guideline of how one would reconcile these in practice?

$\endgroup$
4
  • 5
    $\begingroup$ How does what you're describing differ from model selection in general? $\endgroup$
    – Tim
    Jun 11, 2023 at 7:48
  • 1
    $\begingroup$ @Tim I'm not sure that it really does differ, I guess what I was looking for was more of a reality check akin to Sextus Empiricus' answer to allow me to notice that my assumption that less simplifying assumptions and model "closeness" to reality always coincide when they don't necessarily have to. We can minimize assumptions all we want, but if we can't create a tractable model from that, then we aren't really any closer to our goal. and if we're after predictive performance as a measure of "closeness" to reality, and we get there by violating or making "wrong" assumptions, that can be okay? $\endgroup$
    – QMath
    Jun 11, 2023 at 11:28
  • 3
    $\begingroup$ One approach is to challenge the basic assumptions underlying model selection. For example, Occam's Razor holds that the simplest explanation that holds for some phenomena is the best explanation. Statistically, simple translates as error minimization in both model calibration and out-of-sample fit. Alternatives to Occam include Epicurus' Principle of Multiple Explanations which holds "if several theories are consistent with the observed data, retain them all". cage.ugent.be/~ci/… $\endgroup$
    – user78229
    Jun 11, 2023 at 12:43
  • $\begingroup$ This is typically studied as well - people do not blindly assume assumptions, but analyse breakdown of assumptions either theoretically or through simulations. eg normality assumption would typically relate to how well tail percentiles are matched.. $\endgroup$
    – seanv507
    Jun 14, 2023 at 11:49

4 Answers 4

8
$\begingroup$

I don't think "true" is a suitable category for models. Models are mathematical formalisms and live, if you will, in the world of mathematics, which is not (normally) the world we are interested in when doing statistics.

I tend to agree with (and often cite myself) Box's "all models are wrong, but some are useful" as a good reminder that we shouldn't expect being "true/correct" from a model, however it may be misleading to think of a model as "wrong" in the sense that there is some cure; that reality could somehow show us a truth and we could verify (in the formal world of mathematics) that "all models" disagree with it. (We can show formally based on data that certain not well chosen models look very wrong, but this doesn't generalise to "all models").

To become more practical, I first give credit to existing answers that emphasise the pragmatic/performance aspect; we could try to formalise what we want from a model, and then measure to what extent the model delivers it. The easiest way to do this is obviously predictive power, as stated in the question.

A number of remarks on this. Firstly, our ability to measure predictive power obviously relies on the availability of test/validation data. Not only are there situations in which there is not enough such data, also in some situations available test data don't allow to generalise predictive power as desired, because the model will later be applied in situations in which new data to be predicted may arise in (more or less) different ways and may turn out to be systematically different from the original test data.

Furthermore, it is useful here to distinguish between models and methods. A model (with assumed unknown true parameter values) does not predict; in order to predict you need a prediction method, which may (or may not) be based on parameter estimation. There is often more than one method (or algorithm) that qualify to get a prediction (and/or estimation) based on the same model, which particularly also means that if prediction doesn't work well, the model may not necessarily be at fault. Also prediction may not work well because the information in the data is too weak for good prediction, regardless of the quality of the used model or method.

My last aspect regarding predictive power is that this is not necessarily what is most important in a given project. Particularly, often we want to make statements about the importance of underlying factors and causality, and sometimes we want a simple message for the sake of communication. Also, it may be at least as important as prediction itself to have a reliable quantification of uncertainty. The relation of predictive power to these aims isn't always direct. Particularly, predictive power doesn't require model interpretation, but this may be the major thing we worry about.

So when being pragmatic, I think that it is important to clarify the aims of analysis. Predictive power may be the most important one, or one of several aims, or not that important at all.

I don't think it is appropriate to set up any quality measure (as for example predictive power) as a "dogma" (as said in the question) just because it can be measured. We should use quality measures to the extent that they measure what is important to us, in the given situation.

Now the question is correct suggesting (implicitly) that it is often a problem to evaluate the quality of a model in ways other than predictive quality. I will touch upon some aspects of this.

Firstly, there are direct measurements and tests of model fit, which in some sense measure the "distance" between what is implied by the model and the data (to be distinguished from the assumed truth underlying the data). An example for this is the Kolmogorov distance (and associated test) between the empirical distribution of 1-dimensional data and an assumed cdf. Also there are various test statistics to test independence of observations, linearity of regression etc. These can be used in principle, however, once more, it is important to think about whether and to what extent the outcomes of such tests/measurements are relevant to our aims.

For example, for one-dimensional data, the sample mean and the standard distribution theory used for tests and confidence intervals are based on a normality assumption, however often, for estimating the underlying "true" mean and specifying its uncertainty, they work quite well also if the underlying distribution is not normal (and, e.g., using Kolmogorov distance and the KS-test, data are quite obviously not normal). This is backed up by the Central Limit Theorem, which itself relies on certain (more general) model assumptions. But in certain situations, particularly with gross outliers in the data, it may not work well.

In fact, the claims I just made, namely what "works well" and what doesn't, need to be well defined as well of course, and are relative to the aim of analysis.

What I'm basically saying is that if we imagine different models from the one we originally assumed, and we define what we are interested in relative to these different models, and we then apply a method to the data that is based on the original model, then we can explore, by theory or simulation, to what extent the results we get are still good/useful (according to a well defined criterion such as predictive power, test error rate, number of (in)correctly selected variables, or estimator MSE).

So even though "all models are wrong", I recommend, when assuming one model, to ask, what would happen if another model would be true, that is, say, also compatible with our data, and background knowledge that we have.

Actually, this leads to investigations of robustness and sensitivity analysis. In robust statistics, typically one looks at how the performance of methods could change if another model were true that is "in some sense" similar to the assumed model, and typically one considers "worst cases" in such classes of models. The implication is often to use different, more robust methods that are still calibrated for use with the same original model, but less easily affected if there are deviations as formalised in robustness theory (most prominently outliers).

One could also generalise the model to something nonparametric in order to use methods that have "performance guarantees" over broader classes of models. (Once more always keep in mind which performance measurement is actually relevant for the situation in hand.)

Often there is a trade-off here because methods based on more general models may lose quality for specific models that we may be particularly interested in, and theoretical "performance guarantees" may require quite large data sets to be actually useful. A good recommendation is to set up one or more realistic models for your data, to try out the method(s) that you want to use/compare, and see how they perform in such an artificial situation where you know the truth (because you actually set it yourself).

That you can do such a thing is actually one of the major benefits of statistical models. You can set up an artificial world where you control the truth, and then you can see what works well! (Here you do not rely on enough available test data as you produce that yourself; note that most statistical theory actually does just that - which means you can learn from it without having to "believe" or "trust" the model.)

Of course you need to address to what extent what happens in the specific artificial world you set up is relevant for the real world, but for sure if you are accepting a certain model in the first place, it also makes sense to look at other models that may produce similar data and/or share a similar interpretation.

Here is a paper that is connected to the "philosophy" explained above, very recently accepted for publication by Journal of Data Science, Statistics, and Visualisation.

I. Shamsudheen & C. Hennig: Should we test the model assumptions before running a model-based test?

(In this work we particularly emphasise that finding model deviations that are problematic regarding the results of model based methods is not the same as finding problems regarding the model fit of the data.)

$\endgroup$
4
  • $\begingroup$ Very in-depth and insightful response! The points about how "wrongness" of a model compared to reality isn't something that can be fixed and also that that isn't really a problem as well as the alternatives to predictive performance as a metric when predictive performance isn't feasible for whatever reason definitely stuck. $\endgroup$
    – QMath
    Jun 14, 2023 at 3:26
  • $\begingroup$ As somewhat of a more specific aside, would you happen to have any recommendations for search terms/concepts/references with regards to performance analysis when data of our phenomenon, say rare events, is very minimal? I know Extreme Value Theory deals a bit with that, but how exactly do we validate this? $\endgroup$
    – QMath
    Jun 14, 2023 at 3:26
  • 1
    $\begingroup$ @QMath Maybe that'd be worth a separate question. I don't have anything specific in mind right now. Ultimately, when data is minimal, there's not much you can do based on the data, and subject matter/background knowledge is the only other thing you can go by. More interesting is the question what exactly (and at what precision) can actually be identified based on data, but this will depend on the specific problem. $\endgroup$ Jun 14, 2023 at 9:37
  • $\begingroup$ Very fair point, haha, I'll have to look into that more, thank you for the insight into that/your related question you posited! :) $\endgroup$
    – QMath
    Jun 15, 2023 at 3:13
12
$\begingroup$

What methodologies can we follow to help us balance the ability to create a useful/informative mathematical representation of our phenomena with the "trueness" of our model?

We don't balance the 'trueness' of models. Models are per definition an abstraction of reality and always untrue.

What matters is the model 'closeness' to reality.

The pursuit is the selection/optimization of models with the highest performance (and with potentially additional features like the costs to run the model). There are all sorts of methods to do this. The title of this website, cross validation, is an example.

Methodology for Reconciling "all models are wrong..."

There isn't really anything to reconcile. Yes, models are wrong, but they are close enough to give satisfying estimates/predictions. If the model performs sufficient then there is in practical terms no discrepancy and nothing to reconcile.

$\endgroup$
6
  • $\begingroup$ Ah yes, "closeness" seems to be a more fitting term, I guess I'm just hung up on the idea that a model with less simplifying/tractability assumptions is implicitly "closer" to reality even if our metric is something like predictive performance balanced with implementational feasibility and those two things, even though they may tend to coincide (at least on the less simplifying assumptions and predictive performance front), they don't necessarily have to. Thank you for the reality check on modeling in practice :) $\endgroup$
    – QMath
    Jun 11, 2023 at 10:54
  • 2
    $\begingroup$ If you don't thank any model is true in reality, how can you assess "closeness" of a model to reality in a well defined sense? (I actually agree with your "performance" point, I just think performance is valuable as a goal in itself; it isn't an indicator for "closeness" in any well defined sense.) $\endgroup$ Jun 11, 2023 at 11:43
  • 2
    $\begingroup$ A sidenote, the 'trueness' of a model can be interpreted in two ways. (1) whether a model is exactly the same as reality (2) whether a model makes true statements. In terms of the second interpretation it is possible to have a true model if the statements are about a range. For example, in a coin flip experiment the reality won't be like independent flips with a constant probability parameter. However, a model that describes a range for a varying probability parameter and degree of dependence, might be considered as possibly 'true'. $\endgroup$ Jun 11, 2023 at 12:01
  • $\begingroup$ @ChristianHennig maybe it is just semantics that differs. One might think of a concept like a 'true model' and consider the closeness of some model to this true model. In my terms this 'true model' should just not be called a model anymore, but instead 'reality'. $\endgroup$ Jun 11, 2023 at 12:08
  • $\begingroup$ OK I get that, but I prefer not to use the term "true" for this, as it invites misunderstandings. Particularly it implies that two models that are clearly different can both be called "true", which is not how most people would use the term. If you want to say "I got true statement X using model Y", why not saying just that, rather than the rather confusing "model Y is true (in sense (2))"? $\endgroup$ Jun 11, 2023 at 12:36
10
$\begingroup$

I think that there are a couple of points here.

First, remember the rest of the quote from George Box:

All models are wrong, but some are useful.

So, we can look at the utility of our model. How we measure utility will vary a lot depending on what we are modeling and why we are modeling it. Are we after explanation? Prediction? Both? How important is interpretability? What do we know about our data? (In some fields, we know a lot and in others, not so much). Do we even expect that a simple model will come close? (In some areas of physics, we do. In psychology, not so much).

Then with assumptions, we could have a similar sort of rule:

All assumptions are violated, but sometimes, it doesn't matter much.

So, which violations are violated and how badly? What do we know (from statistical theory or simulations or whatever) about how much that violation matters?

Overall, I don't think there is any algorithm to do this. You have to think, you have to use both substantive and statistical knowledge, and so on.

But hey! That's why we get the big bucks!

$\endgroup$
1
  • 1
    $\begingroup$ Very practical rephrasing! :) This definitely puts things in perspective a bit more about the sort of "looser" but still rigorously validated approach that needs to be taken in practical applications, thanks a lot! $\endgroup$
    – QMath
    Jun 13, 2023 at 6:45
4
$\begingroup$

There is nothing to reconcile: what they mean with "all models are wrong" is that the model is not the phenomenon. Alfred Korzybski said it in different way: “The map is not the territory”. There is not a single thing in this world you can do to surpass this inherent limitation.

Some maps/models are better than others, but it does not matter how good a map/model you make, it is still a map/model, and there is no way to make it more than a map/model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.