"Essentially, all models are wrong, but some are useful."

--- Box, George E. P.; Norman R. Draper (1987). Empirical Model-Building and Response Surfaces, p. 424, Wiley. ISBN 0471810339.

What exactly is the meaning of the above phrase?

  • 12
    On the same book is was earlier mentioned : Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful. Maybe this is more helpful. – usεr11852 Apr 27 '13 at 8:53

14 Answers 14

I think its meaning is best analyzed by looking at it in two parts:

"All models are wrong" that is, every model is wrong because it is a simplification of reality. Some models, especially in the "hard" sciences, are only a little wrong. They ignore things like friction or the gravitational effect of tiny bodies. Other models are a lot wrong - they ignore bigger things. In the social sciences, we ignore a lot.

"But some are useful" - simplifications of reality can be quite useful. They can help us explain, predict and understand the universe and all its various components.

This isn't just true in statistics! Maps are a type of model; they are wrong. But good maps are very useful. Examples of other useful but wrong models abound.

  • 20
    +1 Cause I like the analogy of the maps. I'll use it in the future! – usεr11852 Apr 27 '13 at 10:28
  • 4
    Many models in the "hard" sciences are quite far off, too (yesterday I attended a seminar where measurements where model was within the errorbar, but the errorbar was two orders of magnitude). – gerrit Apr 27 '13 at 16:39
  • 7
    +1. I think your key sentence is "every model is wrong because it is a simplification of reality". People often forget this - for example in naive criticisms of economics (I have my own criticisms, but they need to be more sophisticated than just that "reality is more complex than your model"). If we didn't simplify it, you have raw reality, which is too complex for us to understand. So we have to simplify it to obtain any insight. – Peter Ellis Apr 30 '13 at 20:56
  • 13
    The fantasy of a perfect map on a scale 1:1 has been used by many authors, including Lewis Carroll, Jorge Luis Borges and Umberto Eco. It would actually be no use because it would necessarily be just complicated as the area it maps and no easier to understand (not to mention the awkwardness of unfolding it and laying it out to read). – Nick Cox May 21 '13 at 22:31
  • 2
    Maybe you can also add that a model has to be a bit wrong, because otherwise it would not generalize and would thus not be applicable elsewhere. There are some answers saying this further down. But there are now too many answers to read them all. – ziggystar May 5 '14 at 8:26

It means useful insights can be provided from models which are not a perfect representation of the phenomena they model.

A statistical model is a description of a system using mathematical concepts. As such in many cases you add a certain layer of abstraction to facilitate your inferential procedure (eg. normality of measurement errors, compound symmetry in correlation structures etc.). It is almost impossible for a single model to describe perfectly a real world phenomenon given ourselves have a subjective view of the world (our sensory system is not perfect); nevertheless successful statistical inference does happen as our world does have a certain degree of consistency we exploit. So our almost always wrong models do prove useful.

(I am sure you'll get a big bold answer soon but I tried to be concise on this one!)

  • Can we say that the these useful models are provide approximate solutions? – gpuguy Apr 27 '13 at 9:29
  • 2
    @gpuguy: Sure you can. To quote John Tukey: An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem. (I actually thing J.T.'s quote is amazingly insightful.) – usεr11852 Apr 27 '13 at 10:26
  • 5
    "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." John W. Tukey 1962 The future of data analysis. Annals of Mathematical Statistics 33: 1-67 (see pp.13-14) No doubt he said similar things at other times, but that's the usual source. – Nick Cox Apr 27 '13 at 15:14
  • I c-p'ed the quote directly from the relevant CV's quotes thread. – usεr11852 Apr 27 '13 at 19:20
  • 5
    I copied mine from the original publication. – Nick Cox Apr 27 '13 at 22:55

I found this 2009 JSA talk by Thad Tarpey to provide a useful explanation and commentary on the Box passage. He argues that if we regard models as approximations to the truth, we could just as easily call all models right.

Here’s the abstract:

Students of statistics are often introduced to George Box’s famous quote: “all models are wrong, some are useful.” In this talk I argue that this quote, although useful, is wrong. A different and more positive perspective is to acknowledge that a model is simply a means of extracting information of interest from data. The truth is infinitely complex and a model is merely an approximation to the truth. If the approximation is poor or misleading, then the model is useless. In this talk I give examples of correct models that are not true models. I illustrate how the notion of a “wrong” model can lead to wrong conclusions.

For me the actual insight lies in the following aspect:

A model doesn't have to be correct to be useful.

Unfortunately in many sciences it is often forgotten that models don't necessarily need to be exact representations of reality to allow new discoveries and predictions!

So don't waste your time building a complicated model that needs accurate measurements of a myriade of variables. The true genius invents a simple model that does the job.

A model cannot provide 100% accurate predictions if there is any randomness in the outcomes. If there was no uncertainty, no randomness, and no error, then it would be considered a fact rather than a model. The first is very important, because models are frequently used for modelling expectations of events that have not occurred. This almost guarantees that there is some uncertainty about the real events.

Given perfect information, in theory it might be possible to create a model which gives perfect predictions for such precisely known events. However, even given these unlikely circumstances, such a model may be so complex as to be computationally infeasible to use, and may only be accurate at a particular moment in time as other factors change how values change with events.

Since uncertainty and randomness is present in most real-world data, efforts to obtain a perfect model are a futile exercise. Instead, it is more valuable to look at obtaining a sufficiently accurate model that is simple enough to be usable in terms of both the data and the computation required for its use. While these models are known to be imperfect, some of these flaws are well known and can be considered for decision-making based on the models.

Simpler models may be imperfect, but they are also easier to reason about, to compare to one another, and may be easier to work with because they are likely to be less computationally demanding.

If I may, than just one more comment may be useful. The version of the prase that I prefer is

(...) all models are approximations. Essentially, all models are wrong, but some are useful (...)

taken from Response Surfaces, Mixtures, and Ridge Analyses by Box and Draper (2007, p. 414, Wiley). Looking at the extended quote it is more clear what Box meant -- statistical modeling is about approximating the reality and approximation is never exact, so it is about finding the most appropriate approximation. What is appropriate for your purpose is a subjective thing, that is why it is not one of the models that is useful, but possibly some of them are, depending on purpose of modeling.

You might think of it this way. the maximum complexity (i.e., entropy) of an object obeys some form of the Bekenstein bound:

$$ I \le \frac{2\pi RE}{\hbar c\ln 2} $$

where $E$ is the total rest energy including mass, and $R$ is the radius of a sphere that encloses the object.

That's a big number, in most cases:

The Bekenstein bound for an average human brain would be $2.58991·10^{42}$ bit and represents an upper bound on the information needed to perfectly recreate the average human brain down to the quantum level. This implies that the number of different states ($Ω=2^I$) of the human brain (and of the mind if the physicalism is true) is at most $107.79640·10^{41}$.

So do you want to use "the best map", i.e. the territory itself, with all of the wave equations for all the particles in every cell? Absolutely not. Not only would it be a computational disaster, but you would be modeling things that may have essentially nothing to do with what you care about. If all you want to do is, say, identify whether or not I'm awake, you don't need to know what electron #32458 is doing in neuron #844030 ribosome #2305 molecule #2. If you don't model that, your model is indeed "wrong," but if you can identify whether or not I'm awake, your model is definitely useful.

I think Peter and user11852 gave great answers. I would also add (by negation) that if a model was really good, it would probably be useless because of overfitting (hence, not generalizable).

  • 2
    +1 for the point of overfitting. Algorithms such as Naive Bayes and linear discriminant analysis often work very well, even if you know the underlying model is incorrect (e.g. spam filtering), simply because less data is needed to estimate the parameters. – Dikran Marsupial May 25 '13 at 18:31

Because no one has added it, George Box used the phase quoted to introduce the following section in a book. I believe he does the best job of explaining what he meant:

Now it would be very remarkable if any system existing in the real world could be exactly represented by any simple model. However, cunningly chosen parsimonious models often do provide remarkably useful approximations. For example, the law $PV = RT$ relating pressure $P$, volume $V$ and temperature $T$ of an "ideal" gas via a constant $R$ is not exactly true for any real gas, but it frequently provides a useful approximation and furthermore its structure is informative since it springs from a physical view of the behavior of gas molecules.

For such a model there is no need to ask the question "Is the model true?". If "truth" is to be the "whole truth" the answer must be "No". The only question of interest is "Is the model illuminating and useful?".

Box, G. E. P. (1979), "Robustness in the strategy of scientific model building", in Launer, R. L.; Wilkinson, G. N., Robustness in Statistics, Academic Press, pp. 201–236.

My acid interpretation is: Believing that a mathematical model describes exactly all the factors, and their interactions, governing a phenomenon of interest would be too simplistic and arrogant. We do not even know if the logic we use is enough to understand our universe. However, some mathematical models represent a good enough approximation (in terms of the scientific method) that are useful to draw conclusions about such phenomenon.

As an astrostatistician (a rare breed perhaps), I find the fame of Box's dictum to be unfortunate. In the physical sciences, we often have a strong consensus for understanding the processes underlying of an observed phenomenon, and these processes can often expressed by mathematical models arising from the laws of gravitation, quantum mechanics, thermodynamics, etc. The statistical goals are to estimate the best-fit model parameters physical properties, as well as model selection and validation. A dramatic recent case arose from the March 2013 release of papers from the European Space Agency's Planck satellite's measurements of the cosmic microwave background that convincingly establishes a simple 6-parameter `LambdaCDM' model for the Big Bang. I doubt that Box's dictum would apply anywhere within the wide range of advanced statistical methods used in these 29 papers.

I have just rephrased the above answer by considering process models as focus point. The statement can be interpreted as follows:

"All models are wrong" that is, every model is wrong because it is a simplification of reality. Some models are only a little wrong. They ignore some things, For example: --> changing requirements , --> Ignoring the completion of project within the deadline, --> not considering the customer's desired level of quality etc... Other models are a lot wrong - they ignore bigger things. Classical software process models ignore a lot compared to agile Process models which ignore less.

"But some are useful" - simplifications of reality can be quite useful. They can help us explain, predict and understand the overall project and all its various components. Models are used because their features correspond to most software development programs.

I would like to give another interpretation of the term "useful". Probably not the one Box thought about.

When you have to make decisions, and this is what all information will finally be used for, then you have to measure your success in some form. When talking about decisions with uncertain information, this measure is often called utility.

So we can also think of useful models as those that enable us to make more informed decisions; to achieve our goals more effectively.

This adds another dimension on top of usual criteria, such as the ability of a model to predict something correctly: It allows us to weigh the different aspects a model is about against each other.

"All models are wrong, but some are useful". Perhaps it means: We should be doing the best we can with what we know + search for new learning?

  • 4
    (-1) Can you provide any reference suggesting that G.E.P. Box meant that? As you can find from the other answers, he meant something totally different. – Tim Nov 15 '16 at 14:50
  • The OP is perhaps taking the quote and giving it a new interpretation. I agree with Tim that Box was more or less saying don't take the model as an exact interpretation of reality but recognize that some model can describe the data well. – Michael Chernick Apr 12 '17 at 16:53

protected by Tim Nov 15 '16 at 14:48

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.