Practical thoughts on explanatory vs. predictive modeling

Question

Back in April, I attended a talk at the UMD (University of Maryland) Math Department Statistics group seminar series called "To Explain or To Predict?". The talk was given by Prof. Galit Shmueli who teaches at UMD's Smith Business School. Her talk was based on research she did for a paper titled "Predictive vs. Explanatory Modeling in IS Research", and a follow up working paper titled "To Explain or To Predict?".

Dr. Shmueli's argument is that the terms predictive and explanatory in a statistical modeling context have become conflated, and that statistical literature lacks a a thorough discussion of the differences. In the paper, she contrasts both and talks about their practical implications. I encourage you to read the papers.

The questions I'd like to pose to the practitioner community are:

• How do you define a predictive exercise vs an explanatory/descriptive one? It would be useful if you could talk about the specific application.
• Have you ever fallen into the trap of using one when meaning to use the other? I certainly have. How do you know which one to use?

Answer 1

In one sentence

Predictive modelling is all about "what is likely to happen?", whereas explanatory modelling is all about "what can we do about it?"

In many sentences

I think the main difference is what is intended to be done with the analysis. I would suggest explanation is much more important for intervention than prediction. If you want to do something to alter an outcome, then you had best be looking to explain why it is the way it is. Explanatory modelling, if done well, will tell you how to intervene (which input should be adjusted). However, if you simply want to understand what the future will be like, without any intention (or ability) to intervene, then predictive modelling is more likely to be appropriate.

As an incredibly loose example, using "cancer data".

Predictive modelling using "cancer data" would be appropriate (or at least useful) if you were funding the cancer wards of different hospitals. You don't really need to explain why people get cancer, rather you only need an accurate estimate of how much services will be required. Explanatory modelling probably wouldn't help much here. For example, knowing that smoking leads to higher risk of cancer doesn't on its own tell you whether to give more funding to ward A or ward B.

Explanatory modelling of "cancer data" would be appropriate if you wanted to decrease the national cancer rate - predictive modelling would be fairly obsolete here. The ability to accurately predict cancer rates is hardly likely to help you decide how to reduce it. However, knowing that smoking leads to higher risk of cancer is valuable information - because if you decrease smoking rates (e.g. by making cigarettes more expensive), this leads to more people with less risk, which (hopefully) leads to an expected decrease in cancer rates.

Looking at the problem this way, I would think that explanatory modelling would mainly focus on variables which are in control of the user, either directly or indirectly. There may be a need to collect other variables, but if you can't change any of the variables in the analysis, then I doubt that explanatory modelling will be useful, except maybe to give you the desire to gain control or influence over those variables which are important. Predictive modelling, crudely, just looks for associations between variables, whether controlled by the user or not. You only need to know the inputs/features/independent variables/etc.. to make a prediction, but you need to be able to modify or influence the inputs/features/independent variables/etc.. in order to intervene and change an outcome.

Answer 2

In my view the differences are as follows:

Explanatory/Descriptive

When seeking an explanatory/descriptive answer the primary focus is on the data we have and we seek to discover the underlying relationships between the data after noise has been accounted for.

Example: Is it true that exercising regularly (say 30 minutes per day) leads to lower blood pressure? To answer this question we may collect data from patients about their exercise regimen and their blood pressure values over time. The goal is to see if we can explain variations in blood pressure by variations in exercise regimen.

Blood pressure is impacted by not only exercise by wide variety of other factors as well such as amount of sodium a person eats etc. These other factors would be considered noise in the above example as the focus is on teasing out the relationship between exercise regimen and blood pressure.

Prediction

When doing a predictive exercise, we are extrapolating into the unknown using the known relationships between the data we have at hand. The known relationship may emerge from an explanatory/descriptive analysis or some other technique.

Example: If I exercise 1 hour per day to what extent is my blood pressure likely to drop? To answer this question, we may use a previously uncovered relationship between blood pressure and exercise regimen to perform the prediction.

In the above context, the focus is not on explanation, although an explanatory model can help with the prediction process. There are also non-explanatory approaches (e.g., neural nets) which are good at predicting the unknown without necessarily adding to our knowledge as to the nature of the underlying relationship between the variables.

Answer 3

One practical issue that arises here is variable selection in modelling. A variable can be an important explanatory variable (e.g., is statistically significant) but may not be useful for predictive purposes (i.e., its inclusion in the model leads to worse predictive accuracy). I see this mistake almost every day in published papers.

Another difference is in the distinction between principal components analysis and factor analysis. PCA is often used in prediction, but is not so useful for explanation. FA involves the additional step of rotation which is done to improve interpretation (and hence explanation). There is a nice post today on Galit Shmueli's blog about this.

Update: a third case arises in time series when a variable may be an important explanatory variable but it just isn't available for the future. For example, home loans may be strongly related to GDP but that isn't much use for predicting future home loans unless we also have good predictions of GDP.

Answer 4

Although some people find it easiest to think of the distinction in terms of the model/algorithm used (e.g., neural nets=predictive), that is only one particular aspect of the explain/predict distinction. Here is a deck of slides that I use in my data mining course to teach linear regression from both angles. Even with linear regression alone and with this tiny example various issues emerge that lead to different models for explanatory vs. predictive goals (choice of variables, variable selection, performance measures, etc.)

Galit

Answer 5

Example: A classic example that I have seen is in the context of predicting human performance. Self-efficacy (i.e., the degree to which a person thinks that they can perform a task well) is often a strong predictor of task performance. Thus, if you put self-efficacy into a multiple regression along with other variables such as intelligence and degree of prior experience, you often find that self-efficacy is a strong predictor.

This has lead some researchers to suggest that self-efficacy causes task performance. And that effective interventions are those which focus on increasing a person's sense of self-efficacy.

However, the alternative theoretical model sees self-efficacy largely as a consequence of task performance. I.e., If you are good, you'll know it. In this framework interventions should focus on increasing actual competence and not perceived competence.

Thus, including a variable like self-efficacy might increase prediction, but assuming you adopt the self-efficacy-as-consequence model, it should not be included as a predictor if the aim of the model is to elucidate causal processes influencing performance.

This of course raises the issue of how to develop and validate a causal theoretical model. This clearly relies on multiple studies, ideally with some experimental manipulation, and a coherent argument about dynamic processes.

Proximal versus distal: I've seen similar issues when researchers are interested in the effects of distal and proximal causes. Proximal causes tend to predict better than distal causes. However, theoretical interest may be in understanding the ways in which distal and proximal causes operate.

Variable selection issue: Finally, a huge issue in social science research is the variable selection issue. In any given study, there is an infinite number of variables that could have been measured but weren't. Thus, interpretation of models need to consider the implications of this when making theoretical interpretations.

Answer 6

I haven't read her work beyond the abstract of the linked paper, but my sense is that the distinction between "explanation" and "prediction" should be thrown away and replaced with the distinction between the aims of the practitioner, which are either "causal" or "predictive". In general, I think "explanation" is such a vague word that it means nearly nothing. For example, is Hooke's Law explanatory or predictive? On the other end of the spectrum, are predictively accurate recommendation systems good causal models of explicit item ratings? I think we all share the intuition that the goal of science is explanation, while the goal of technology is prediction; and this intuition somehow gets lost in consideration of the tools we use, like supervised learning algorithms, that can be employed for both causal inference and predictive modeling, but are really purely mathematical devices that are not intrinsically linked to "prediction" or "explanation".

Having said all of that, maybe the only word that I would apply to a model is interpretable. Regressions are usually interpretable; neural nets with many layers are often not so. I think people sometimes naively assume that a model that is interpretable is providing causal information, while uninterpretable models only provide predictive information. This attitude seems simply confused to me.

Answer 7

Statistical Modeling: Two Cultures (2001) by L. Breiman is, perhaps, the best paper on this point. His main conclusions (see also the replies from other prominent statisticians in the end of the document) are as follows:

• "Higher predictive accuracy is associated with more reliable information about the underlying data mechanism. Weak predictive accuracy can lead to questionable conclusions."
• "Algorithmic models can give better predictive accuracy than data models, and provide better information about the underlying mechanism."

Answer 8

I am still a bit unclear as to what the question is. Having said that, to my mind the fundamental difference between predictive and explanatory models is the difference in their focus.

Explanatory Models

By definition explanatory models have as their primary focus the goal of explaining something in the real world. In most instances, we seek to offer simple and clean explanations. By simple I mean that we prefer parsimony (explain the phenomena with as few parameters as possible) and by clean I mean that we would like to make statements of the following form: "the effect of changing $x$ by one unit changes $y$ by $\beta$ holding everything else constant". Given these goals of simple and clear explanations, explanatory models seek to penalize complex models (by using appropriate criteria such as AIC) and prefer to obtain orthogonal independent variables (either via controlled experiments or via suitable data transformations).

Predictive Models

The goal of predictive models is to predict something. Thus, they tend to focus less on parsimony or simplicity but more on their ability to predict the dependent variable.

However, the above is somewhat of an artificial distinction as explanatory models can be used for prediction and sometimes predictive models can explain something.

Answer 9

as others have already said, the distinction is somewhat meaningless, except in so far as the aims of the researcher are concerned.

Brad Efron, one of the commentators on The Two Cultures paper, made the following observation (as discussed in my earlier question):

Prediction by itself is only occasionally sufficient. The post office is happy with any method that predicts correct addresses from hand-written scrawls. Peter Gregory undertook his study for prediction purposes, but also to better understand the medical basis of hepatitis. Most statistical surveys have the identification of causal factors as their ultimate goal.

Certain fields (eg. Medicine) place a heavy weight on model fitting as explanatory process (the distribution, etc.), as a means to understanding the underlying process that generates the data. Other fields are less concerned with this, and will be happy with a "black box" model that has a very high predictive success. This can work its way into the model building process as well.

Answer 10

With respect, this question could be better focused. Have people ever used one term when the other was more appropriate? Yes, of course. Sometimes it's clear enough from context, or you don't want to be pedantic. Sometimes people are just sloppy or lazy in their terminology. This is true of many people, and I'm certainly no better.

What's of potential value here (discussing explanation vs. prediction on CV), is to clarify the distinction between the two approaches. In short, the distinction centers on the role of causality. If you want to understand some dynamic in the world, and explain why something happens the way it does, you need to identify the causal relationships amongst the relevant variables. To predict, you can ignore causality. For example, you can predict an effect from knowledge about its cause; you can predict the existence of the cause from knowledge that the effect occurred; and you can predict the approximate level of one effect by knowledge of another effect that is driven by the same cause. Why would someone want to be able to do this? To increase their knowledge of what might happen in the future, so that they can plan accordingly. For example, a parole board may want to be able to predict the probability that a convict will recidivate if paroled. However, this is not sufficient for explanation. Of course, estimating the true causal relationship between two variables can be extremely difficult. In addition, models that do capture (what are thought to be) the real causal relationships are often worse for making predictions. So why do it, then? First, most of this is done in science, where understanding is pursued for its own sake. Second, if we can reliably pick out true causes, and can develop the ability to affect them, we can exert some influence over the effects.

With regard to the statistical modeling strategy, there isn't a large difference. Primarily the difference lies in how to conduct the study. If your goal is to be able to predict, find out what information will be available to users of the model when they will need to make the prediction. Information they won't have access to is of no value. If they will most likely want to be able to predict at a certain level (or within a narrow range) of the predictors, try to center the sampled range of the predictor on that level and oversample there. For instance, if a parole board will mostly want to know about criminals with 2 major convictions, you might gather info about criminals with 1, 2, and 3 convictions. On the other hand, assessing the causal status of a variable basically requires an experiment. That is, experimental units need to be assigned at random to prespecified levels of the explanatory variables. If there is concern about whether or not the nature of the causal effect is contingent on some other variable, that variable must be included in the experiment. If it is not possible to conduct a true experiment, then you face a much more difficult situation, one that is too complex to go into here.

Answer 11

Most of the answers have helped clarify what modeling for explanation and modeling for prediction are and why they differ. What is not clear, thus far, is how they differ. So, I thought I would offer an example that might be useful.

Suppose we are intereted in modeling College GPA as a function of academic preparation. As measures of academic preparation, we have:

1. Aptitude Test Scores;
2. HS GPA; and
3. Number of AP Tests passed.

Strategy for Prediction

If the goal is prediction, I might use all of these variables simultaneously in a linear model and my primary concern would be predictive accuracy. Whichever of the variables prove most useful for predicting College GPA would be included in the final model.

Strategy for Explanation

If the goal is explanation, I might be more concerned about data reduction and think carefully about the correlations among the independent variables. My primary concern would be interpreting the coefficients.

Example

In a typical multivariate problem with correlated predictors, it would not be uncommon to observe regression coefficients that are "unexpected". Given the interrelationships among the independent variables, it would not be surprising to see partial coefficients for some of these variables that are not in the same direction as their zero-order relationships and which may seem counter intuitive and tough to explain.

For example, suppose the model suggests that (with Aptitude Test Scores and Number of AP Tests Successfully Completed taken into account) higher High School GPAs are associated with lower College GPAs. This is not a problem for prediction, but it does pose problems for an explanatory model where such a relationship is difficult to interpret. This model might provide the best out of sample predictions but it does little to help us understand the relationship between academic preparation and College GPA.

Instead, an explanatory strategy might seek some form of variable reduction, such as principal components, factor analysis, or SEM to:

1. focus on the variable that is the best measure of "academic performance" and model College GPA on that one variable; or
2. use factor scores/latent variables derived from the combination of the three measures of academic preparation rather than the original variables.

Strategies such as these might reduce the predictive power of the model, but they may yield a better understanding of how Academic Preparation is related to College GPA.

Answer 12

I would like to offer a model-centered view on the matter.

Predictive modeling is what happens in most analyses. For example, a researcher sets up a regression model with a bunch of predictors. The regression coefficients then represent predictive comparisons between groups. The predictive aspect comes from the probability model: the inference is done with regard to a superpopulation model which may have produced the observed population or sample. The purpose of this model is to predict new outcomes for units emerging from this superpopulation. Often, this is a vain objective because things are always changing, especially in the social world. Or because your model is about rare units such as countries and you cannot draw a new sample. The usefulness of the model in this case is left to the appreciation of the analyst.

When you try to generalize the results to other groups or future units, this is still prediction but of a different kind. We may call it forecasting for example. The key point is that the predictive power of estimated models is, by default, of descriptive nature. You compare an outcome across groups and hypothesize a probability model for these comparisons, but you cannot conclude that these comparisons constitute causal effects.

The reason is that these groups may suffer from selection bias. Ie, they may naturally have a higher score in the outcome of interest, irrespective of the treatment (the hypothetical causal intervention). Or they may be subject to a different treatment effect size than other groups. This is why, especially for observational data, the estimated models are generally about predictive comparisons and not explanation. Explanation is about the identification and estimation of causal effect and requires well designed experiments or thoughtful use of instrumental variables. In this case, the predictive comparisons are cut from any selection bias and represent causal effects. The model may thus be regarded as explanatory.

I found that thinking in these terms has often clarified what I was really doing when setting up a model for some data.

Answer 13

We can learn a lot more than we think from Black box "predictive" models. The key is in running different types of sensitivity analyses and simulations to really understand how model OUTPUT is affected by changes in the INPUT space. In this sense even a purely predictive model can provide explanatory insights. This is a point that is often overlooked or misunderstood by the research community. Just because we do not understand why an algorithm is working doesn't mean the algorithm lacks explanatory power...

Overall from a mainstream point of view, probabilityislogic's succinct reply is absolutely correct...

Answer 14

There is distinction between what she calls explanatory and predictive applications in statistics. She says we should know every time we use one or another which one exactly is being used. She says we often mix them up, hence conflation.

I agree that in social science applications, the distinction is sensible, but in natural sciences they are and should be the same. Also, I call them inference vs. forecasting, and agree that in social sciences one should not mix them up.

I'll start with the natural sciences. In physics we're focused on explaining, we're trying to understand how the world works, what causes what etc. So, the focus is on causality, inference and such. On the other hand, the predictive aspect is also a part of the scientific process. In fact, the way you prove a theory, which already explained observations well (think of in-sample), is to predict new observations then check how prediction worked. Any theory that lack predictive abilities will have big trouble gaining acceptance in physics. That's why experiments such as Michelson-Morley's are so important.

In social sciences, unfortunately, the underlying phenomena are unstable, unrepeatable, unreproducible. If you watch nuclei decay you'll get the same results every time you observe them, and the same results that I or a dude one hundred years ago got. Not in economics or finance. Also, the ability to conduct experiments is very limited, almost non existent for all practical purposes, we only observe and conduct random samples of observations. I can keep going on but the idea's that the phenomena that we deal with are very unstable, hence our theories are not of the same quality as in physics. Therefore, one of the ways we deal with the situation is to focus on either inference (when you try to understand what causes what or impact what) or forecasting (just say what you think will happen to this or that ignore the structure).

Answer 15

A Structural Model would give explanation and a predictive model would give prediction. A structural model would have latent variables. A structural model is a simultaneous culmination of regression and factor analysis

The latent variables are manifested in the form of multi collinearity in predictive models (regression).

Answer 16

Explanatory model has also been used in medicine and the health area as well, with a very different meaning. Basically what people have as internal beliefs or meanings can be quite different from accepted explanations. For example a religious person may have an explanatory model that an illness was due to punishment or karma for a past behaviour along with accepting th biological reasons as well.

https://thehealthcareblog.com/blog/2013/06/11/the-patient-explanatory-model/

https://pdfs.semanticscholar.org/0b69/ffd5cc4c7bb2f401be6819c946a955344880.pdf