# Practical thoughts on explanatory vs. predictive modeling

This question has been bugging me for some time, and I was going to write a blog post about it. However, I think it is better left for discussion in this forum.

Back in April, I attended a talk at the UMD Math Department Statistics group seminar series called "To Explain or To Predict?". The talk was given by Prof. Galit Shmueli who teaches at UMD's Smith Business School. Her talk was based on research she did for a paper titled "Predictive vs. Explanatory Modeling in IS Research", and a follow up working paper titled "To Explain or To Predict?".

Dr. Shmueli's argument is that the terms predictive and explanatory in a statistical modeling context have become conflated, and that statistical literature lacks a a thorough discussion of the differences. In the paper, she contrasts both and talks about their practical implications. I encourage you to read the papers.

The question I'd like to pose to the practitioner community is: have you ever fallen into the trap of using one when meaning to use the other? I certainly have. How do you know which one to use? How do you define a predictive excercise vs an exlanatory/descriptive one? It would be useful if you could talk about the specific application.

-
This question is proposed to be closed. See: meta.stats.stackexchange.com/questions/213/… I see that it has 2 votes. Could the up-voters or the OP comment on why they would like to see the question stay open at the meta thread? – user28 Aug 3 '10 at 20:33
Rather than saying "this should be closed. Someone should defend it" how about starting with explaining why you want it closed. Too vague? Then ask for clarification. This seems a reasonable question to me. The asker presents a paper and asks about the difference is between predictive and explanatory statistics. The only change I would make to the question is to clarify exactly the question thus making it easier to vote. – JD Long Aug 3 '10 at 20:39
I have already offered a reason on the meta thread. I feel that 'meta discussions' about the question would clutter up this particular page. – user28 Aug 3 '10 at 20:41
Nevertheless, it should be community wiki. – mbq Aug 3 '10 at 22:17
Could you add proper links to the talk/papers mentioned above? – chl Nov 25 '11 at 10:19

In one sentence

Predictive modelling is all about "what is likely to happen?", whereas explanatory modelling is all about "what can we do about it?"

In many sentences

I think the main difference is what is intended to be done with the analysis. I would suggest explanation is much more important for intervention than prediction. If you want to do something to alter an outcome, then you had best be looking to explain why it is the way it is. Explanatory modelling, if done well, will tell you how to intervene (which input should be adjusted). However, if you simply want to understand what the future will be like, without any intention (or ability) to intervene, then predictive modelling is more likely to be appropriate.

As an incredibly loose example, using "cancer data".

Predictive modelling using "cancer data" would be appropriate (or at least useful) if you were funding the cancer wards of different hospitals. You don't really need to explain why people get cancer, rather you only need an accurate estimate of how much services will be required. Explanatory modelling probably wouldn't help much here. For example, knowing that smoking leads to higher risk of cancer doesn't on its own tell you whether to give more funding to ward A or ward B.

Explanatory modelling of "cancer data" would be appropriate if you wanted to decrease the national cancer rate - predictive modelling would be fairly obsolete here. The ability to accurately predict cancer rates is hardly likely to help you decide how to reduce it. However, knowing that smoking leads to higher risk of cancer is valuable information - because if you decrease smoking rates (e.g. by making cigarettes more expensive), this leads to more people with less risk, which (hopefully) leads to an expected decrease in cancer rates.

Looking at the problem this way, I would think that explanatory modelling would mainly focus on variables which are in control of the user, either directly or indirectly. There may be a need to collect other variables, but if you can't change any of the variables in the analysis, then I doubt that explanatory modelling will be useful, except maybe to give you the desire to gain control or influence over those variables which are important. Predictive modelling, crudely, just looks for associations between variables, whether controlled by the user or not. You only need to know the inputs/features/independent variables/etc.. to make a prediction, but you need to be able to modify or influence the inputs/features/independent variables/etc.. in order to intervene and change an outcome.

-
+1, nicely done! I hate to nitpick, but I want to note that prediction doesn't have to be about the future. For example, an archeologist may want to determine (i.e., predict) the level of rainfall in an area at a point in the past by knowledge of traces (i.e., effects of rainfall) that are left. – gung Nov 26 '11 at 17:33
@gung - I thought I worded my response so that this didn't happen. Clearly, I missed a spot :-) – probabilityislogic Nov 26 '11 at 23:19
Nice answer. I think we need in many cases to know what the future looks like and why. Suppose, when studying customer churn, you want to know how many customer (and exactly which customer) churn next N month and then why they churn so marketing can intervene to retain them. Then we need both predictive (to learn future number and customers) and explanatory to tell us why, so we can reduce the churners. So, do we have hybrid model of both or one suffices? Varty touches it up by saying "The known relationship may emerge from an explanatory/descriptive analysis or some other technique" – Espanta Oct 31 '15 at 3:47

In my view the differences are as follows:

Explanatory/Descriptive

When seeking an explanatory/descriptive answer the primary focus is on the data we have and we seek to discover the underlying relationships between the data after noise has been accounted for.

Example: Is it true that exercising regularly (say 30 minutes per day) leads to lower blood pressure? To answer this question we may collect data from patients about their exercise regimen and their blood pressure values over time. The goal is to see if we can explain variations in blood pressure by variations in exercise regimen.

Blood pressure is impacted by not only exercise by wide variety of other factors as well such as amount of sodium a person eats etc. These other factors would be considered noise in the above example as the focus is on teasing out the relationship between exercise regimen and blood pressure.

Prediction

When doing a predictive exercise, we are extrapolating into the unknown using the known relationships between the data we have at hand. The known relationship may emerge from an explanatory/descriptive analysis or some other technique.

Example: If I exercise 1 hour per day to what extent is my blood pressure likely to drop? To answer this question, we may use a previously uncovered relationship between blood pressure and exercise regimen to perform the prediction.

In the above context, the focus is not on explanation, although an explanatory model can help with the prediction process. There are also non-explanatory approaches (e.g., neural nets) which are good at predicting the unknown without necessarily adding to our knowledge as to the nature of the underlying relationship between the variables.

-
Well said (very easy to understand!) – Adhesh Josh Nov 25 '11 at 20:22
@AdheshJosh You should up-vote the answer if you think its good. – probabilityislogic Nov 25 '11 at 22:32
+1 This reply largely avoids confusing association with causation by using the language of explanation, description, and relationship. This lends it a desirable degree of clarity. – whuber Nov 26 '11 at 4:13
Under Explanation you wrote "the primary focus is on the data we have" -- I think you are trying to say that the task is retrospective (as opposed to the prospective nature of prediction). In explanation (read "causal explanation") there is actually a large focus on theory and domain knowledge and the data are used to test these assumptions/theories. In contrast, in prediction it is more data-driven and you are more open-minded about relationships, because you are not searching for causality but rather for correlation. – Galit Shmueli Nov 26 '11 at 4:42
@GalitShmueli Reg theory/domain knowledge- yes, I agree with that point. I was simply trying to contrast prediction vis-a-vis explanation by focusing on what seems to me the key distinction- extrapolating a variable's value vs unearthing the relationship between variables. In the process, I am of course guilty of neglecting subtle nuances between the two paradigms. – varty Nov 26 '11 at 4:59

One practical issue that arises here is variable selection in modelling. A variable can be an important explanatory variable (e.g., is statistically significant) but may not be useful for predictive purposes (i.e., its inclusion in the model leads to worse predictive accuracy). I see this mistake almost every day in published papers.

Another difference is in the distinction between principal components analysis and factor analysis. PCA is often used in prediction, but is not so useful for explanation. FA involves the additional step of rotation which is done to improve interpretation (and hence explanation). There is a nice post today on Galit Shmueli's blog about this.

Update: a third case arises in time series when a variable may be an important explanatory variable but it just isn't available for the future. For example, home loans may be strongly related to GDP but that isn't much use for predicting future home loans unless we also have good predictions of GDP.

-
Why/how would an important explanatory variable reduce predictive accuracy? – user28 Aug 4 '10 at 2:34
@Srikant. This can happen when the explanatory variable has a weak but significant relationship with the response variable. Then the coefficient can be statistically significant but hard to estimate. Consequently, the MSE of predictions can increase when the variable is included compared to when it is omitted. (The bias is reduced with its inclusion but the variance is increased.) – Rob Hyndman Aug 4 '10 at 5:15
First paragraph is a very, very good point. Still sometimes is even worse; here PMID: 18052912 is a great example that sometimes a better model can be made on the noise part of the set than on a true one -- it is obvious that one can do a good model on random data, but this is a bit shocking. – mbq Aug 4 '10 at 14:20
forgive my ingorance, but isn't rotation normally a part of PCA as well as FA? – richiemorrisroe Mar 15 '11 at 11:40
A statistically sig. but weak predictor is rarely effective either for prediction or explanation. E.g., if a linear regression solution has an RSQ of .40 without including the predictor X1, and if inclusion of X1 adds .01 to that RSQ, then X1 is "important" neither for prediction nor explanation. – rolando2 Oct 2 '12 at 22:36

Although some people find it easiest to think of the distinction in terms of the model/algorithm used (e.g., neural nets=predictive), that is only one particular aspect of the explain/predict distinction. Here is a deck of slides that I use in my data mining course to teach linear regression from both angles. Even with linear regression alone and with this tiny example various issues emerge that lead to different models for explanatory vs. predictive goals (choice of variables, variable selection, performance measures, etc.)

Galit

-
Thank you for sharing those! – whuber Nov 27 '11 at 18:19
Out of curiosity, is it intentional that in your discussion of regression for prediction (starting p. 33) you choose predictors (step 1) before partitioning into training and validation datasets (step 3)? I have thought that the most objective and honest procedure would be to partition at the outset, even before looking at scatterplots (step 2). If regressors are chosen based on the entire dataset, wouldn't that inflate apparent significance levels on many tests even when they are subsequently applied to the validation data? – whuber Nov 27 '11 at 18:25
I think the more general question is whether you perform data visualization before keeping a holdout. When the dataset is large, then it doesn't really matter. With a small sample, using visualization to choose predictors is indeed dangerous. In my slides I don't mean using visualization for variable selection. The "select predictors" is more generally "select a potential set of available predictors that are reasonable". It's more about incorporating domain knowledge for selecting a reasonable set. – Galit Shmueli Feb 14 '12 at 18:15
Continuing the topic "To explain or to predict", I have a related question here. I would appreciate if you took a look since the question is mostly based on your paper. – Richard Hardy Oct 17 '15 at 22:38

Example: A classic example that I have seen is in the context of predicting human performance. Self-efficacy (i.e., the degree to which a person thinks that they can perform a task well) is often a strong predictor of task performance. Thus, if you put self-efficacy into a multiple regression along with other variables such as intelligence and degree of prior experience, you often find that self-efficacy is a strong predictor.

This has lead some researchers to suggest that self-efficacy causes task performance. And that effective interventions are those which focus on increasing a person's sense of self-efficacy.

However, the alternative theoretical model sees self-efficacy largely as a consequence of task performance. I.e., If you are good, you'll know it. In this framework interventions should focus on increasing actual competence and not perceived competence.

Thus, including a variable like self-efficacy might increase prediction, but assuming you adopt the self-efficacy-as-consequence model, it should not be included as a predictor if the aim of the model is to elucidate causal processes influencing performance.

This of course raises the issue of how to develop and validate a causal theoretical model. This clearly relies on multiple studies, ideally with some experimental manipulation, and a coherent argument about dynamic processes.

Proximal versus distal: I've seen similar issues when researchers are interested in the effects of distal and proximal causes. Proximal causes tend to predict better than distal causes. However, theoretical interest may be in understanding the ways in which distal and proximal causes operate.

Variable selection issue: Finally, a huge issue in social science research is the variable selection issue. In any given study, there is an infinite number of variables that could have been measured but weren't. Thus, interpretation of models need to consider the implications of this when making theoretical interpretations.

-
There is also a problem in social science of "weak hypothesis" (eg effect is positive vs negative). And in that "self efficacy" example, you could view that as an internal predictor of performance that each person has built up. So it is probably similar to using a "black box" prediction as an explanatory variable. – probabilityislogic Jul 14 '14 at 5:54

Statistical Modeling: Two Cultures (2001) by L. Breiman is, perhaps, the best paper on this point. His main conclusions (see also the replies from other prominent statisticians in the end of the document) are as follows:

• "Higher predictive accuracy is associated with more reliable information about the underlying data mechanism. Weak predictive accuracy can lead to questionable conclusions."
• "Algorithmic models can give better predictive accuracy than data models, and provide better information about the underlying mechanism."
-
Just to make a link with an earlier related question: The Two Cultures: statistics vs. machine learning? – chl Nov 27 '11 at 11:16
The problem with algorithmic models is that they are hard to understand. This makes it hard to diagnose and fix potential problems that arise. A structural model is much easier to assess because you know what each component should look like. – probabilityislogic Oct 16 '12 at 9:15

I haven't read her work beyond the abstract of the linked paper, but my sense is that the distinction between "explanation" and "prediction" should be thrown away and replaced with the distinction between the aims of the practitioner, which are either "causal" or "predictive". In general, I think "explanation" is such a vague word that it means nearly nothing. For example, is Hooke's Law explanatory or predictive? On the other end of the spectrum, are predictively accurate recommendation systems good causal models of explicit item ratings? I think we all share the intuition that the goal of science is explanation, while the goal of technology is prediction; and this intuition somehow gets lost in consideration of the tools we use, like supervised learning algorithms, that can be employed for both causal inference and predictive modeling, but are really purely mathematical devices that are not intrinsically linked to "prediction" or "explanation".

Having said all of that, maybe the only word that I would apply to a model is interpretable. Regressions are usually interpretable; neural nets with many layers are often not so. I think people sometimes naively assume that a model that is interpretable is providing causal information, while uninterpretable models only provide predictive information. This attitude seems simply confused to me.

-

I am still a bit unclear as to what the question is. Having said that, to my mind the fundamental difference between predictive and explanatory models is the difference in their focus.

Explanatory Models

By definition explanatory models have as their primary focus the goal of explaining something in the real world. In most instances, we seek to offer simple and clean explanations. By simple I mean that we prefer parsimony (explain the phenomena with as few parameters as possible) and by clean I mean that we would like to make statements of the following form: "the effect of changing $x$ by one unit changes $y$ by $\beta$ holding everything else constant". Given these goals of simple and clear explanations, explanatory models seek to penalize complex models (by using appropriate criteria such as AIC) and prefer to obtain orthogonal independent variables (either via controlled experiments or via suitable data transformations).

Predictive Models

The goal of predictive models is to predict something. Thus, they tend to focus less on parsimony or simplicity but more on their ability to predict the dependent variable.

However, the above is somewhat of an artificial distinction as explanatory models can be used for prediction and sometimes predictive models can explain something.

-
+1 for mentioning complexity which was not directly mentioned by the top answers. However, the challenge arises when explanatory models are used for interventions. How does one ensure that the estimated coefficients are not biased which is a common problem resulting from parsimony? – Thomas Speidel Dec 23 '13 at 21:18

With respect, this question could be better focused. Have people ever used one term when the other was more appropriate? Yes, of course. Sometimes it's clear enough from context, or you don't want to be pedantic. Sometimes people are just sloppy or lazy in their terminology. This is true of many people, and I'm certainly no better.

What's of potential value here (discussing explanation vs. prediction on CV), is to clarify the distinction between the two approaches. In short, the distinction centers on the role of causality. If you want to understand some dynamic in the world, and explain why something happens the way it does, you need to identify the causal relationships amongst the relevant variables. To predict, you can ignore causality. For example, you can predict an effect from knowledge about its cause; you can predict the existence of the cause from knowledge that the effect occurred; and you can predict the approximate level of one effect by knowledge of another effect that is driven by the same cause. Why would someone want to be able to do this? To increase their knowledge of what might happen in the future, so that they can plan accordingly. For example, a parole board may want to be able to predict the probability that a convict will recidivate if paroled. However, this is not sufficient for explanation. Of course, estimating the true causal relationship between two variables can be extremely difficult. In addition, models that do capture (what are thought to be) the real causal relationships are often worse for making predictions. So why do it, then? First, most of this is done in science, where understanding is pursued for its own sake. Second, if we can reliably pick out true causes, and can develop the ability to affect them, we can exert some influence over the effects.

With regard to the statistical modeling strategy, there isn't a large difference. Primarily the difference lies in how to conduct the study. If your goal is to be able to predict, find out what information will be available to users of the model when they will need to make the prediction. Information they won't have access to is of no value. If they will most likely want to be able to predict at a certain level (or within a narrow range) of the predictors, try to center the sampled range of the predictor on that level and oversample there. For instance, if a parole board will mostly want to know about criminals with 2 major convictions, you might gather info about criminals with 1, 2, and 3 convictions. On the other hand, assessing the causal status of a variable basically requires an experiment. That is, experimental units need to be assigned at random to prespecified levels of the explanatory variables. If there is concern about whether or not the nature of the causal effect is contingent on some other variable, that variable must be included in the experiment. If it is not possible to conduct a true experiment, then you face a much more difficult situation, one that is too complex to go into here.

-
I wonder about the role of causality. For instance, suppose we have a dataset of the dimensions and volumes of a set of blocks, $(x,y,z,v)$, and we are modeling their association by regressing $z$ on $(x,y,v)$ (and interactions thereof). In what sense can it be said that two dimensions and a volume "cause" the third dimension? The distinction, therefore, between explanation and prediction appears to be based on something fundamentally different; namely, the purpose of the analysis. Re your last paragraph, there are many accounts on this site attesting to sharp differences in strategy. – whuber Nov 26 '11 at 4:10
You're right this hinges on the purpose of the study. I suppose I didn't make that explicit (I only talked about what you want to achieve). It's also true that explanation needn't be concerned with causality exactly--something analogous to causality fits as well (e.g., the dimensions - volume case is one of logical / mathematical implication). However, most explanatory modeling centers on causality; I guess I thought I could skip that sort of thing for the sake of simplicity. Finally, strategy does differ during study design & data collection, but regressing y on x is pretty much the same. – gung Nov 26 '11 at 16:55
Thank you for the reply. From other exchanges on this site I have learned to understand universal statements like "most explanatory modeling centers on causality" to reflect the writer's background and experience, rather than as being literally true. In the physical and "hard" sciences this statement may be correct, but in the social and "soft" sciences I doubt practitioners would make such a strong claim. Often, in fact, the relationships under study are believed to have common hidden causes but do not reflect direct causation between regressors and the regressand. – whuber Nov 27 '11 at 16:37
@whuber it's certainly true that my ideas are influenced by my background and experience. If this answer is not useful (I notice that is hasn't gotten any votes), I can delete it. A number of others have provided answers that cover the ideas I meant to convey. – gung Nov 27 '11 at 18:10
@whuber - a good example of soft causailty is "smoking causes cancer" -though i'm sure you could find a chain smoker who doesn't have cancer. The notion of causality is inter-linked with the timing of events. The cause must happen before the effect - which explains why the cube example makes no sense. – probabilityislogic Feb 11 '12 at 10:25

as others have already said, the distinction is somewhat meaningless, except in so far as the aims of the researcher are concerned.

Brad Efron, one of the commentators on The Two Cultures paper, made the following observation (as discussed in my earlier question):

Prediction by itself is only occasionally sufficient. The post office is happy with any method that predicts correct addresses from hand-written scrawls. Peter Gregory undertook his study for prediction purposes, but also to better understand the medical basis of hepatitis. Most statistical surveys have the identification of causal factors as their ultimate goal.

Certain fields (eg. Medicine) place a heavy weight on model fitting as explanatory process (the distribution, etc.), as a means to understanding the underlying process that generates the data. Other fields are less concerned with this, and will be happy with a "black box" model that has a very high predictive success. This can work its way into the model building process as well.

-

Most of the answers have helped clarify what modeling for explanation and modeling for prediction are and why they differ. What is not clear, thus far, is how they differ. So, I thought I would offer an example that might be useful.

Suppose we are intereted in modeling College GPA as a function of academic preparation. As measures of academic preparation, we have:

1. Aptitude Test Scores;
2. HS GPA; and
3. Number of AP Tests passed.

Strategy for Prediction

If the goal is prediction, I might use all of these variables simultaneously in a linear model and my primary concern would be predictive accuracy. Whichever of the variables prove most useful for predicting College GPA would be included in the final model.

Strategy for Explanation

If the goal is explanation, I might be more concerned about data reduction and think carefully about the correlations among the independent variables. My primary concern would be interpreting the coefficients.

Example

In a typical multivariate problem with correlated predictors, it would not be uncommon to observe regression coefficients that are "unexpected". Given the interrelationships among the independent variables, it would not be surprising to see partial coefficients for some of these variables that are not in the same direction as their zero-order relationships and which may seem counter intuitive and tough to explain.

For example, suppose the model suggests that (with Aptitude Test Scores and Number of AP Tests Successfully Completed taken into account) higher High School GPAs are associated with lower College GPAs. This is not a problem for prediction, but it does pose problems for an explanatory model where such a relationship is difficult to interpret. This model might provide the best out of sample predictions but it does little to help us understand the relationship between academic preparation and College GPA.

Instead, an explanatory strategy might seek some form of variable reduction, such as principal components, factor analysis, or SEM to:

1. focus on the variable that is the best measure of "academic performance" and model College GPA on that one variable; or
2. use factor scores/latent variables derived from the combination of the three measures of academic preparation rather than the original variables.

Strategies such as these might reduce the predictive power of the model, but they may yield a better understanding of how Academic Preparation is related to College GPA.

-
Regarding the counter-intuitive sign, i wonder if it is because our intuition is interpretting the wrong covariate - like a main effect as if it was nested or interaction effect. – probabilityislogic Feb 11 '12 at 10:32

I would like to offer a model-centered view on the matter.

Predictive modeling is what happens in most analyses. For example, a researcher sets up a regression model with a bunch of predictors. The regression coefficients then represent predictive comparisons between groups. The predictive aspect comes from the probability model: the inference is done with regard to a superpopulation model which may have produced the observed population or sample. The purpose of this model is to predict new outcomes for units emerging from this superpopulation. Often, this is a vain objective because things are always changing, especially in the social world. Or because your model is about rare units such as countries and you cannot draw a new sample. The usefulness of the model in this case is left to the appreciation of the analyst.

When you try to generalize the results to other groups or future units, this is still prediction but of a different kind. We may call it forecasting for example. The key point is that the predictive power of estimated models is, by default, of descriptive nature. You compare an outcome across groups and hypothesize a probability model for these comparisons, but you cannot conclude that these comparisons constitute causal effects.

The reason is that these groups may suffer from selection bias. Ie, they may naturally have a higher score in the outcome of interest, irrespective of the treatment (the hypothetical causal intervention). Or they may be subject to a different treatment effect size than other groups. This is why, especially for observational data, the estimated models are generally about predictive comparisons and not explanation. Explanation is about the identification and estimation of causal effect and requires well designed experiments or thoughtful use of instrumental variables. In this case, the predictive comparisons are cut from any selection bias and represent causal effects. The model may thus be regarded as explanatory.

I found that thinking in these terms has often clarified what I was really doing when setting up a model for some data.

-
+1, there's good information here. I would be cautious regarding the statement "Predictive modeling is what happens in most analyses", however. Whether or not predictive modeling is more common will vary by discipline, etc. My guess would be that most modeling in academia is explanatory, & that a lot of modeling / data mining that is done in the private sector (eg identify potential repeat customers) is predictive. I could easily be wrong, but it will be tough to say, a priori, which happens most of the time. – gung Oct 16 '12 at 12:58
Well, in my view, most modeling of observational data is predictive, even if the aim is explanatory. If you do not randomize the attribution of the treatment and actually induce a change in an experimental setup, your regression coefficients will only have a descriptive value, ie, they only provide the means to predictive comparisons. For example, you can predict success at school based on demographic characteristics but that does not mean that these demographics are explanatory causal effects. The reason is that comparative predictions are exposed to selection bias. – lionel Oct 16 '12 at 13:47

We can learn a lot more than we think from Black box "predictive" models. The key is in running different types of sensitivity analyses and simulations to really understand how model OUTPUT is affected by changes in the INPUT space. In this sense even a purely predictive model can provide explanatory insights. This is a point that is often overlooked or misunderstood by the research community. Just because we do not understand why an algorithm is working doesn't mean the algorithm lacks explanatory power...

Overall from a mainstream point of view, probabilityislogic's succinct reply is absolutely correct...

-
It is unclear what "explanatory insights" can be gleaned in this way, if by that phrase you imply causality. – gung Sep 2 '14 at 19:40