Is machine learning less useful for understanding causality, thus less interesting for social science?

Question

My understanding of the difference between machine learning/other statistical predictive techniques vs. the kind of statistics that social scientists (e.g., economists) use is that economists seem very interested in understanding the effect of a single or several variables -- both in terms of magnitude and detecting whether the relationship is causal. For this, you end up concerning yourself with experimental and quasi-experimental methods, etc.

Machine learning or statistical modeling that is predictive often entirely neglects this aspect and in many cases doesn't give you a specific degree to which one variable affects the outcome (logit and probit do seem to do both).

A related question is to what extent do theoretically inspired economic or behavioral models have an advantage over atheoretical models when predicting to new domains? What would a machine learning or prediction-oriented statistician say to the criticism that without an economic model, you wouldn't be able to correctly predict new samples where covariates were very different.

I'd be really happy to hear people's take on this from all perspectives.

Answer 1

There are IMHO no formal differences that distinguish machine learning and statistics at the fundamental level of fitting models to data. There may be cultural differences in the choice of models, the objectives of fitting models to data, and to some extend the interpretations.

In the typical examples I can think of we always have

• a collection of models $M_i$ for $i \in I$ for some index set $I$,
• and for each $i$ an unknown component $\theta_i$ (the parameters, may be infinite dimensional) of the model $M_i$.

Fitting $M_i$ to data is almost always a mathematical optimization problem consisting of finding the optimal choice of the unknown component $\theta_i$ to make $M_i$ fit the data as measured by some favorite function.

The selection among the models $M_i$ is less standard, and there is a range of techniques available. If the objective of the model fitting is purely predictive, the model selection is done with an attempt to get good predictive performance, whereas if the primary objective is to interpret the resulting models, more easily interpretable models may be selected over other models even if their predictive power is expected to be worse.

What could be called old school statistical model selection is based on statistical tests perhaps combined with step-wise selection strategies, whereas machine learning model selection typically focuses on the expected generalization error, which is often estimated using cross-validation. Current developments in and understandings of model selection do, however, seem to converge towards a more common ground, see, for instance, Model Selection and Model Averaging.

Inferring causality from models

The crux of the matter is how we can interpret a model? If the data obtained are from a carefully designed experiment and the model is adequate it is plausible that we can interpret the effect of a change of a variable in the model as a causal effect, and if we repeat the experiment and intervene on this particular variable we can expect to observe the estimated effect. If, however, the data are observational, we can not expect that estimated effects in the model correspond to observable intervention effects. This will require additional assumptions irrespectively of whether the model is a "machine learning model" or a "classical statistical model".

It may be that people trained in using classical statistical models with a focus on univariate parameter estimates and effect size interpretations are of the impression that a causal interpretation is more valid in this framework than in a machine learning framework. I would say it is not.

The area of causal inference in statistics does not really remove the problem, but it does make the assumptions upon which causal conclusions rest explicit. They are referred to as untestable assumptions. The paper Causal inference in statistics: An overview by Judea Pearl is a good paper to read. A major contribution from causal inference is the collection of methods for the estimation of causal effects under assumptions where there actually are unobserved confounders, which is otherwise a major concern. See Section 3.3 in the Pearl paper above. A more advanced example can be found in the paper Marginal Structural Models and Causal Inference in Epidemiology.

It is a subject matter question whether the untestable assumptions hold. They are precisely untestable because we can not test them using the data. To justify the assumptions other arguments are required.

As an example of where machine learning and causal inference meets, the ideas of targeted maximum-likelihood estimation as presented in Targeted Maximum Likelihood Learning by Mark van der Laan and Daniel Rubin typically exploit machine learning techniques for non-parametric estimation followed by the "targeting" towards a parameter of interest. The latter could very well be a parameter with a causal interpretation. The idea in Super Learner is to rely heavily on machine learning techniques for estimation of parameters of interest. It is an important point by Mark van der Laan (personal communication) that classical, simple and "interpretable" statistical models are often wrong, which leads to biased estimators and too optimistic assessment of the uncertainty of the estimates.

Answer 2

There is a (fairly limited) set of statistical tools for so-called "causal inference". These are designed for actually assessing causal relationships and are proven to do this correctly. Excellent, but not for the meek of heart (or brain, for that matter).

Apart from that, in many instances, the ability to imply causality is much more a consequence of your design than of the techniques at hand: if you have control over 'all' the variables in your experiment, and you see something happening everytime you (only) change one variable, it is reasonable to call the thing that happens a 'consequence' of the thing you change (unfortunately, in real research, these extreme cases rarely actually occur). Another intuitive but sound reasoning is time-based: if you randomly (but in a controlled manner) change a variable and another changes the day after, causality is also around the corner.

All of my second paragraph essentially works regardless of which methods you use to find which variables changed in which conditions, so at least in theory there is no reason why Machine Learning (ML) would be worse than Statistics based methods.

Disclaimer: Highly subjective paragraph folowing

However, in my experience, too often ML techniques are just let loose on a blob of data without consideration of where the data came from or how it was collected (i.e. disregarding the design). In those cases, ever so often a result bobs up, but it will be extremely hard to say something useful about causality. This will be exactly the same when some statistically sound method is run upon that same data. However, people with a strong statistics background are trained to be critical towards these matters, and if all goes well, will avoid these pitfalls. Perhaps it is simply the mindset of early (but sloppy) adopters of ML techniques (typically not the developers of new techniques but those eager to 'prove' some results with them in their field of interest) that has given ML its bad reputation on this account. (note that I am not saying statistics is better than ML, or that all people doing ML are sloppy and those doing stats aren't)

Answer 3

My view is that the models used in economics and the other social sciences are useful only insofar as they have predictive power in the real world - a model which doesn't predict the real world is just some clever math. A favorite saying of mine to colleagues is that "data is king".

It seems to me that your question raises two critiques of a predictive approach. First, you point out that the models produced by machine learning techniques may not be interpretable. Second, you suggest that the methods used by those in the social sciences are more useful for uncovering causal relationships than machine learning.

To address the first point, I'd offer the following counter argument. The present fad in machine learning favours methods (like SVMs and NN) which are not at all easy for a layperson to understand. This does not mean that all machine learning techniques have this property. For example, the venerable C4.5 decision tree is still widely used 20 years after reaching the final stage of its development, and produces as output a number of classification rules. I would argue that such rules lend themselves better to interpretation than do concepts like the log odds ratio, but that's a subjective claim. In any case, such models are interpretable.

In addressing the second point, I will concede that if you train a machine learning model in one environment, and test it in another, it will likely fail, however, there is no reason to suppose a priori that this is not also true of a more conventional model: if you build your model under one set of assumptions, and then evaluate it under another, you'll get bad results. To co-opt a phrase from computer programming: "garbage in, garbage out" applies equally well to both machine learning and designed models.

Answer 4

No. Causal inference is an active area of research in machine learning, for instance see the proceedings of this workshop and this one. I would however point out that even if causal inference or model interpretation is your primary interest, it is still a good idea to try an opaque purely predictive approach in parallel, so that you will know if there is a significant performance penalty involved in insisting on an interpretable model.

Answer 5

I will not re-iterate the very good points already made in other answers, but would like to add a somewhat different perspective. What I say here is somewhat philosophical, not necessarily drawn from professional experience, but from a mixed background in the physical sciences, complex systems theory and machine learning (and, I have to admit, largely undergraduate statistics).

One substantial difference between machine learning and classical statistical approaches (that I am aware of) is in the set of assumptions that are made. In classical statistics, many assumptions about the underlying processes and distributions are fixed and tend to be taken for granted. In machine learning, however, these assumptions are explicitly chosen for each model, resulting in a much broader set of possibilities and perhaps a greater awareness of the assumptions being made.

We are seeing more and more that systems in the world around us behave in complex, non-linear ways, and that many processes do not obey assumptions of normality etc. typically present in classical statistics. I would argue that, due to the flexibility and variety of model assumptions, machine learning approaches will often lead to a more robust model in such cases.

There are strong model assumptions built into phrases such as "magnitude of effect", "causal relation", and "degree to which one variable affects the outcome". In a complex system (such as an economy), these assumptions will only be valid within a certain window of possible system states. With some observables and processes, this window may be large, leading to relatively robust models. With others it may be small or even empty. Perhaps the greatest danger is the middle ground: a model may appear to be working, but when the system shifts, fail in a sudden and surprising ways.

Machine learning is no panacea. Rather, I see it as a search for new ways of gleaning meaning from our observations, seeking new paradigms that are needed if we are to deal effectively with the complexity we are starting to perceive in the world around us.