What is the meaning of "All models are wrong, but some are useful"

Question

"Essentially, all models are wrong, but some are useful."

--- Box, George E. P.; Norman R. Draper (1987). Empirical Model-Building and Response Surfaces, p. 424, Wiley. ISBN 0471810339.

What exactly is the meaning of the above phrase?

Answer 1

I think its meaning is best analyzed by looking at it in two parts:

"All models are wrong" that is, every model is wrong because it is a simplification of reality. Some models, especially in the "hard" sciences, are only a little wrong. They ignore things like friction or the gravitational effect of tiny bodies. Other models are a lot wrong - they ignore bigger things. In the social sciences, we ignore a lot.

"But some are useful" - simplifications of reality can be quite useful. They can help us explain, predict and understand the universe and all its various components.

This isn't just true in statistics! Maps are a type of model; they are wrong. But good maps are very useful. Examples of other useful but wrong models abound.

Answer 2

It means useful insights can be provided from models which are not a perfect representation of the phenomena they model.

A statistical model is a description of a system using mathematical concepts. As such in many cases you add a certain layer of abstraction to facilitate your inferential procedure (eg. normality of measurement errors, compound symmetry in correlation structures etc.). It is almost impossible for a single model to describe perfectly a real world phenomenon given ourselves have a subjective view of the world (our sensory system is not perfect); nevertheless successful statistical inference does happen as our world does have a certain degree of consistency we exploit. So our almost always wrong models do prove useful.

(I am sure you'll get a big bold answer soon but I tried to be concise on this one!)

Answer 3

I found this 2009 JSA talk by Thad Tarpey to provide a useful explanation and commentary on the Box passage. He argues that if we regard models as approximations to the truth, we could just as easily call all models right.

Here’s the abstract:

Students of statistics are often introduced to George Box’s famous quote: “all models are wrong, some are useful.” In this talk I argue that this quote, although useful, is wrong. A different and more positive perspective is to acknowledge that a model is simply a means of extracting information of interest from data. The truth is infinitely complex and a model is merely an approximation to the truth. If the approximation is poor or misleading, then the model is useless. In this talk I give examples of correct models that are not true models. I illustrate how the notion of a “wrong” model can lead to wrong conclusions.

Answer 4

Because no one has added it, George Box used the phase quoted to introduce the following section in a book. I believe he does the best job of explaining what he meant:

Now it would be very remarkable if any system existing in the real world could be exactly represented by any simple model. However, cunningly chosen parsimonious models often do provide remarkably useful approximations. For example, the law $PV = RT$ relating pressure $P$, volume $V$ and temperature $T$ of an "ideal" gas via a constant $R$ is not exactly true for any real gas, but it frequently provides a useful approximation and furthermore its structure is informative since it springs from a physical view of the behavior of gas molecules.

For such a model there is no need to ask the question "Is the model true?". If "truth" is to be the "whole truth" the answer must be "No". The only question of interest is "Is the model illuminating and useful?".

Box, G. E. P. (1979), "Robustness in the strategy of scientific model building", in Launer, R. L.; Wilkinson, G. N., Robustness in Statistics, Academic Press, pp. 201–236.

Answer 5

For me the actual insight lies in the following aspect:

A model doesn't have to be correct to be useful.

Unfortunately in many sciences it is often forgotten that models don't necessarily need to be exact representations of reality to allow new discoveries and predictions!

So don't waste your time building a complicated model that needs accurate measurements of a myriade of variables. The true genius invents a simple model that does the job.

Answer 6

A model cannot provide 100% accurate predictions if there is any randomness in the outcomes. If there was no uncertainty, no randomness, and no error, then it would be considered a fact rather than a model. The first is very important, because models are frequently used for modelling expectations of events that have not occurred. This almost guarantees that there is some uncertainty about the real events.

Given perfect information, in theory it might be possible to create a model which gives perfect predictions for such precisely known events. However, even given these unlikely circumstances, such a model may be so complex as to be computationally infeasible to use, and may only be accurate at a particular moment in time as other factors change how values change with events.

Since uncertainty and randomness is present in most real-world data, efforts to obtain a perfect model are a futile exercise. Instead, it is more valuable to look at obtaining a sufficiently accurate model that is simple enough to be usable in terms of both the data and the computation required for its use. While these models are known to be imperfect, some of these flaws are well known and can be considered for decision-making based on the models.

Simpler models may be imperfect, but they are also easier to reason about, to compare to one another, and may be easier to work with because they are likely to be less computationally demanding.

Answer 7

If I may, than just one more comment may be useful. The version of the prase that I prefer is

(...) all models are approximations. Essentially, all models are wrong, but some are useful (...)

taken from Response Surfaces, Mixtures, and Ridge Analyses by Box and Draper (2007, p. 414, Wiley). Looking at the extended quote it is more clear what Box meant -- statistical modeling is about approximating the reality and approximation is never exact, so it is about finding the most appropriate approximation. What is appropriate for your purpose is a subjective thing, that is why it is not one of the models that is useful, but possibly some of them are, depending on purpose of modeling.

Answer 8

You might think of it this way. the maximum complexity (i.e., entropy) of an object obeys some form of the Bekenstein bound:

$$ I \le \frac{2\pi RE}{\hbar c\ln 2} $$

where $E$ is the total rest energy including mass, and $R$ is the radius of a sphere that encloses the object.

That's a big number, in most cases:

The Bekenstein bound for an average human brain would be $2.58991·10^{42}$ bit and represents an upper bound on the information needed to perfectly recreate the average human brain down to the quantum level. This implies that the number of different states ($Ω=2^I$) of the human brain (and of the mind if the physicalism is true) is at most $107.79640·10^{41}$.

So do you want to use "the best map", i.e. the territory itself, with all of the wave equations for all the particles in every cell? Absolutely not. Not only would it be a computational disaster, but you would be modeling things that may have essentially nothing to do with what you care about. If all you want to do is, say, identify whether or not I'm awake, you don't need to know what electron #32458 is doing in neuron #844030 ribosome #2305 molecule #2. If you don't model that, your model is indeed "wrong," but if you can identify whether or not I'm awake, your model is definitely useful.

Answer 9

I think Peter and user11852 gave great answers. I would also add (by negation) that if a model was really good, it would probably be useless because of overfitting (hence, not generalizable).

Answer 10

My acid interpretation is: Believing that a mathematical model describes exactly all the factors, and their interactions, governing a phenomenon of interest would be too simplistic and arrogant. We do not even know if the logic we use is enough to understand our universe. However, some mathematical models represent a good enough approximation (in terms of the scientific method) that are useful to draw conclusions about such phenomenon.

Answer 11

As an astrostatistician (a rare breed perhaps), I find the fame of Box's dictum to be unfortunate. In the physical sciences, we often have a strong consensus for understanding the processes underlying of an observed phenomenon, and these processes can often expressed by mathematical models arising from the laws of gravitation, quantum mechanics, thermodynamics, etc. The statistical goals are to estimate the best-fit model parameters physical properties, as well as model selection and validation. A dramatic recent case arose from the March 2013 release of papers from the European Space Agency's Planck satellite's measurements of the cosmic microwave background that convincingly establishes a simple 6-parameter `LambdaCDM' model for the Big Bang. I doubt that Box's dictum would apply anywhere within the wide range of advanced statistical methods used in these 29 papers.

Answer 12

I have just rephrased the above answer by considering process models as focus point. The statement can be interpreted as follows:

"All models are wrong" that is, every model is wrong because it is a simplification of reality. Some models are only a little wrong. They ignore some things, For example: --> changing requirements , --> Ignoring the completion of project within the deadline, --> not considering the customer's desired level of quality etc... Other models are a lot wrong - they ignore bigger things. Classical software process models ignore a lot compared to agile Process models which ignore less.

"But some are useful" - simplifications of reality can be quite useful. They can help us explain, predict and understand the overall project and all its various components. Models are used because their features correspond to most software development programs.

Answer 13

I would like to give another interpretation of the term "useful". Probably not the one Box thought about.

When you have to make decisions, and this is what all information will finally be used for, then you have to measure your success in some form. When talking about decisions with uncertain information, this measure is often called utility.

So we can also think of useful models as those that enable us to make more informed decisions; to achieve our goals more effectively.

This adds another dimension on top of usual criteria, such as the ability of a model to predict something correctly: It allows us to weigh the different aspects a model is about against each other.

Answer 14

"All models are wrong, but some are useful". Perhaps it means: We should be doing the best we can with what we know + search for new learning?