On George Box, Galit Shmueli and the scientific method?

Question

(This question might seem like it is better suited for the Philosophy SE. I am hoping that statisticians can clarify my misconceptions about Box's and Shmueli's statements, hence I am posting it here).

George Box (of ARIMA fame) said:

"All models are wrong, but some are useful."

Galit Shmueli in her famous paper "To Explain or to Predict", argues (and cites others who agree with her) that:

Explaining and predicting are not the same, and that some models do a good job of explaining, even though they do a poor job at predicting.

I feel that these two principles are somehow contradictory.

If a model doesn't predict well, is it useful?

More importantly, if a model explains well (but doesn't necessarily predict well), then it has to be true (i.e not wrong) in some way or another. So how does that mesh with Box's "all models are wrong"?

Finally, if a model explains well, but doesn't predict well, how is it even scientific? Most scientific demarcation criteria (verificationism, falsificstionism, etc...) imply that a scientific statement has to have predictive power, or colloquially: A theory or model is correct only if it can be empirically tested (or falsified), which means that it has to predict future outcomes.

My questions:

• Are Box's statement and Shmueli's ideas indeed contradictory, or am I missing something, e.g can a model not have predictive power yet still be useful?
• If the statements of Box and Shmueli are not contradictory, then what does it mean for a model to be wrong and not predict well, yet still have explanatory power? Put it differently: If one takes away both correctness and predictive ability, what is left of a model?

What empirical validations are possible when a model has explanatory power, but not predictive power? Shmueli mentions things like: use the AIC for explanation and the BIC for prediction, etc,...but I don't see how that solves the problem. With predictive models, you can use the AIC, or the BIC, or the $R^2$, or $L1$ regularization, etc...but ultimately out of sample testing and performance in production is what determines the quality of the model. But for models that explain well, I don't see how any loss function can ever truly evaluate a model. In philosophy of science, there is the concept of underdetermination which seems pertinent here: For any given data set, one can always judiciously choose some distribution (or mixture of distributions) and loss function $L$ in such a way that they fit the data (and therefore can be claimed to explain it). Moreover, the threshold that $L$ should be under for someone to claim that the model adequately explains the data is arbitrary (kind of like p-values, why is it $p < 0.05$ and not $p < 0.1$ or $p < 0.01$?).

• Based on the above, how can one objectively validate a model that explains well, but doesn't predict well, since out of sample testing is not possible?

Answer 1

Let me start with the pithy quote by George Box, that "all models are wrong, but some are useful". This statement is an encapsulation of the methodological approach of "positivism", which is a philosophical approach that is highly influential in the sciences. This approach is described in detail (in the context of economic theory) in the classic methodological essay of Friedman (1966). In that essay, Friedman argues that any useful scientific theory necessarily constitutes a simplification of reality, and thus its assumptions must always depart from reality to some degree, and may even depart substantially from reality. He argues that the value of a scientific theory should not be judged by the closeness of its assumptions to reality --- instead it should be judged by its simplicity in reducing the complexity of the world to a manageable set of principles, and its accuracy in making predictions about reality, and generating new testable hypotheses about reality. Thus, Friedman argues that "all models are wrong" insofar as they all contain assumptions that simplify (and therefore depart from) reality, but that "some are useful" insofar as they give a simple framework to make useful predictions about reality.

Now, if you read Box (1976) (the paper where he first states that "all models are wrong"), you will see that he does not cite Friedman, nor does he mention methodological positivism. Nevertheless, his explanation of the scientific method and its characteristics is extremely close to that developed by Friedman. In particular, both authors stress that a scientific theory will make predictions about reality that can be tested against observed facts, and the error in the prediction can then be used as a basis for revision of the theory.

Now, on to the dichotomy discussed by Galit Shmueli in Shmueli (2001). In this paper, Shmueli compares causal explanation and prediction of observed outcomes and argues that these are distinct activities. Specifically, she argues that causal relations are based on underlying constructs that do not manifest directly in measureable outcomes, and so "measurable data are not accurate representations of their underlying constructs" (p. 293). She therefore argues that there is an aspect of statistical analysis that involves making inferences about unobservable underlying causal relations that are not manifested in measureable counterfactual differences in outcomes.

Unless I am misunderstanding something, I think it is fair to say that this idea is in tension with the positivist views of Box and Friedman, as represented in the quote by Box. The positivist viewpoint essentially says that there are no admissible metaphysical "constructs" beyond those that manifest in measureable outcomes. Positivism confines itself to consideration of observable data, and concepts built on this data; it excludes consideration of a priori metaphysical concepts. Thus, a positivist would argue that the concept of causality can only be valid to the extent that it is defined in terms of measureable outcomes in reality --- to the extent it is defined as something distinct from this (as Shmueli treats it), this would be regarded as metaphysical speculation, and would be treated as inadmissible in scientific discourse.

So I think you're right --- these two approaches are essentially in conflict. The positivist approach used by Box insists that valid scientific concepts be grounded entirely in their manifestations in reality, whereas the alternative approach used by Shmueli says that there are some "constructs" that are important scientific concepts (that we want to explain) but which cannot be perfectly represented when they are "operationalised" by relating them to measureable outcomes in reality.

Answer 2

A model, when used to explain things, is a simplification of reality. Simplification is just another word for "wrong in some useful way". For example, if we round the number 3.1415926535898 to 3.14 we are making an error, but this error allows us humans to focus on the most important part of that number. This is how models are used in explaining, it provides insights on some problem, but by necessity has to abstract away from a lot of other things: We humans are just not very good at looking at a thousands things simultaneously. If we primarily care about predicting we want to include those thousands things whenever possible, but with explaining the trade-off is different.

Answer 3

An example of a model that is excellent at prediction but does not explain anything is given in the Wikipedia article “All models are wrong”. The example is Newton’s model of gravitation. Newton’s model almost always gives predictions that are indistinguishable from empirical observations. Yet the model is extremely implausible: because it postulates a force that can act instantaneously over arbitrarily large distances.

Newton’s model has been supplanted by the model given in Einstein’s general theory of relativity. With general relativity, gravitational forces travel through space at finite speed (the speed of light).

Newton’s model is not a simplification of the general-relativistic model. To illustrate that, consider an apple falling down from a tree. According to general relativity, the apple falls without Earth exerting any force on the apple. (The primary reason the apple falls is that Earth warps time, so that clocks near the base of the tree run more slowly than clocks high up in the tree.) Thus, as the Wikipedia article notes, Newton’s model is completely wrong from an explanatory perspective.

The paper by Shmueli [2010] presumes that there are two purposes for a model: prediction and explanation. In fact, several authors have stated that there are three purposes (see e.g. Konishi & Kitagawa [Information Criteria and Statistical Modeling, 2008: §1.1] and Friendly & Meyer [Discrete Data Analysis, 2016: §11.6]). The three purposes correspond to the three kinds of logical reasoning:

• prediction (corresponding to deduction);
• parameter estimation (corresponding to induction);
• description of structure (corresponding to abduction).

Answer 4

I'm an undergraduate in Statistics, so I won't call myself an expert, but here are my two cents.

Models don't explain themselves; humans interpret them. Linear models are easier to understand than neural networks and random forests because they are closer to how we make decisions. Indeed, ANNs imitate the human brain, but you don't decide which restaurant to go tomorrow by doing a series of matrix multiplications. Instead, you weight some factors in your mind by their importance, which is essentially a linear combination.

"Explanatory power" measures how well a model gets along with humans' intuition, whereas "predictive power" measures how well it aligns with the underlying mechanism of the process in interest. The contradiction between them is essentially the gap between what the world is and how we can perceive/understand it. I hope this explains why "some models do a good job of explaining, even though they do a poor job at predicting".

Ian Stewart once said, "If our brains were simple enough for us to understand them, we'd be so simple that we couldn't." Unfortunately, our little human brains are actually very simple compared to the universe, or even a stock market (which involves a lot of brains :). Up to now, all models are products of human brains, so it must be more-or-less inaccurate, which leads to Box's "All models are wrong". On the other hand, a model doesn't have to be technically correct to be useful. For example, Newton's laws of motion has been disproved by Einstein, but it remains useful when an object is not ridiculously big or fast.

To address your question, I honestly can't see the incompatibility between Box and Shmueli's points. It seems that you consider "explanatory power" and "predictive power" to be binomial properties, but I think they sit at the two ends of a spectrum.