3
$\begingroup$

I was just reading the Reddit thread "My issue with data science" in r/datascience. One of the main points made in the thread is that prediction is fundamentally a different game to causal inference. When we deal with real-world data, it often isn't feasible to design controlled experiments so that we can perform causal inference. In that case, and since we often primarily care about prediction (that is, we often don't care why something happens – just that it happens), people just use predictive methods.

In the real world, we often have (1) limited data that was (2) not generated through any kind of controlled experiment. My understanding is that this is the worst situation: Having large amounts of data – even if it is not generated through any kind of controlled experiment – enables us to make good predictions (using, for instance, Deep Learning), and having limited data that was generated by a strictly controlled experiment also enables us to make good predictions.

So what statistical methods/tools are suitable (statistically sound) for use in such cases? What statistical methods can we use to squeeze as much predictive value out of limited data that was generated without any experimental design/controls? Are there any machine learning tools that are appropriate here, or are they all only suitable with lots of data? What is the research that I should be looking at? Someone mentioned that Bayesian methods are good for this, but I don't know enough to have an opinion.

$\endgroup$
5
  • 2
    $\begingroup$ I think you misunderstood the post. OP is not saying that you cannot do causal inference with observational data, he is saying that people calling them selfs data scientists, do not have the skills to do causal inference using observational data and that they are misinterpreting their results as causal, even tho they did not run an appropriate analysis. Again, due to a lack of knowledge and skills, not due to a lack of available methods, which there are plenty. $\endgroup$ – rep_ho Sep 24 '20 at 8:28
  • $\begingroup$ @rep_ho So we don't need any experimental design to perform causal inference (even though it would make it much easier and/or more accurate)? $\endgroup$ – The Pointer Sep 24 '20 at 8:32
  • 2
    $\begingroup$ that is correct. We didn't run a 20year old randomized controlled trial on smoking to show that smokng causes cancer for example. It's not always possible with any dataset to get causal answers, but it many cases it is. $\endgroup$ – rep_ho Sep 24 '20 at 8:41
  • $\begingroup$ @rep_ho Interesting. It seems that I did indeed misunderstand the OP. Would you please elaborate on what the OP was referring to with regards to most data scientists not having the skills? What skills are usually required to perform causal inference with observational data? $\endgroup$ – The Pointer Sep 24 '20 at 8:50
  • $\begingroup$ Lots of data does not necessarily yield good predictions; it is perfectly possible, probably common, to have a massive dataset that is biased or confounded. $\endgroup$ – einar Sep 24 '20 at 9:51
2
+50
$\begingroup$

I am sorry for being too verbose.

You started with a premise, that we need a controlled experiment to do causal inference. This is not correct. Although randomized controlled experiments are the gold standard, it's not the only way to do causal inference. Sometimes it's just unfeasible or unethical to do experiments, but we would still like to know causes. For example, nobody randomized people to smoking and non-smoking groups and followed them for 30 years to show that smoking causes cancer.

What methods are there to do causal inference with non-experimental data?

So what can we do with non-experimental data? First, if we know a confound and we measured it, we can adjust for the confound, using standard regression methods. We can also resample or reweight or sample to have "treatment" and "control" groups with the same levels of confounding variables on average. For example, smoking correlates with socioeconomic status, so we can collect data from the population, adjust for SES using ANCOVA, or create a balanced sample with the same amount of poor, middle-income, wealthy people in smoking and non-smoking groups.

Second, we can look for quasi-experiments, where the treatment and control groups were created "by accident" without researchers directly randomizing people. E.g., a ban on smoking, which affects all people in the specific area regardless of SES or other confounds.

Third, we can look for regression discontinuity, where the people bellow some threshold should be the same that people just above some threshold, but the "intervention" happens only to the people above the threshold, thus they are the control group. I.e. studying the efficiency of a free lunches program using people just eligible for the program and people that are just not eligible for the program. Or maybe how good is education for you, studying people who just made the entry exam and who just failed the exam. Similarly, this can be used for events that happend in time, so comparing results from before the program was introduced and after.

Forth, we can use instrumental variables, which are variables that are for sure not related to our confounds but are related to the exposure. E.g., increasing taxes for cigarettes for sure won't give you a genetic resilience against lung cancer, but it will lower the number of cigarettes people smoke.

Fifth, we can use structural equation models when the constructs are not measured directly or a complicated (assumed) causal relationship between several variables.

I am sure there are more methods, and I oversimplified stuff, but it should give you an idea of possible. There are whole fields, notably epidemiology and economics, focused on doing causal inference using non-experimental data.

Do we need causal inference for prediction?

Causal inference is specifically important for predicting what will/would happen if we do some intervention. Did people get to our website because we advertised for it, or would they get there anyway? Will vitamin D supplement cure depression? Or do depressed people have lower vitamin D only because they don't like going out that much, and therefore the supplement will do nothing?

Causal inference also helps make models more robust. If our predictive model is based on causal effects, it will work fine even when the population changes. Causal inference is also important for evaluating machine learning models. Is my model predicting well because it's a fancy DL model using 50k variables, or because it's just predicting age?

Causal inference for data science

Your role as a data scientist should be to know these things and propose solutions to your business. You shouldn't just say, I don't know if this customer got here thanks to our ads, you should propose ways to create experiments or quasi-experiments, randomize people, and so on.

Can we use machine learning for causal inference?

Yes, any matching or adjustment that is performed using regression methods can also be performed using machine learning methods. The benefit is that ML methods might be able to learn more complicated relationships from the data or learn information from high-dimensional datasets. Look for Atlantic causal inference conference data challenge.

What is the research you should be looking at?

I would start with any introductory statistics book that talks about confound adjustment, interpreting standard linear regression, ANCOVA, and so on. Focus on interpretation and less on math. You can read on experimental and quasi-experimental designs. There is a whole field of causal inference; however, most of the stuff coming from there are quite unreadable. Pearl's "Book of why" is OK. Atlantic causal inference challenge papers are the state fo the art for using ML to do causal inference. You can also look at "Elements of Causality" by Peters, Janzig, and Scholkopf, which is related to causal inference for machine learning.

$\endgroup$
2
  • 1
    $\begingroup$ Wow, excellent answer! The had no idea the Atlantic causal inference challenge even existed. $\endgroup$ – The Pointer Sep 24 '20 at 10:13
  • $\begingroup$ Well written answer. (+1) $\endgroup$ – Sextus Empiricus Sep 24 '20 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.