# How to perform a what-if study using observational data?

A team fit a Random Forest model to a dataset $$S=\{\mathbf{x}_i,y_i\}_{i=1}^N$$, where $$\mathbf{x}$$ is a vector of continuous and categorical variables, and $$y$$ is a binary response. The model has a low CV-error, and the precision-recall curve looks good.

Now, for new samples they would like to perform what-if studies, i.e. for a fixed variable $$x_j$$ they would like to see how $$p(y=1|x_j)$$ changes as a function of $$x_j$$, all other things being equal. The goal of this activity is the following: for new samples $$\mathbf{x}$$, they want to modify the value of $$x_j$$ to $$x'_j$$ , so that $$x'_jp(y=1|(x_1,\dots,x'_j,\dots,x_n))$$ is maximized. However, by doing so, it seems to me that they modify the distribution $$p(\mathbf{x},y)$$, thus the results are not reliable anymore. I think this is related to causal inference. Is there a way to modify this process so that it actually works? Which methods should I study to help this team?

EDIT: An important point I forgot to mention is that $$x_j$$ is one of the continuous variables, unfortunately.

• This is a subschool of study under causal inference, usually referred to as "counter-factual reasoning". This is quite a good paper to get familiar with it: microsoft.com/en-us/research/wp-content/uploads/2013/11/… – Zhubarb Jun 10 at 16:13
• Thanks @Zhubarb. Actually, I thought counterfactuals would apply to a different problem - what would have happened, had I set $x_j$ to $c$ rather than to $d$. In this case, the new sample $\mathbf{x}$ is modified before I get the corresponding new result $y$. I thought do calculus would be more useful in our case. However, I know very little about both approaches, thus I may well be wrong. I'll read your reference with interest. If there's anything else you'd like to suggest, I'm all ears, as long as it's not an overly long book 🙂 – DeltaIV Jun 10 at 16:33
• @Zhubarb by the way, I forgot to specify that $x_j$ is continuous. I guess this complicates things quite a bit, doesn't it? – DeltaIV Jun 10 at 16:35
• The paper is focusing on "continuously valued variables with meaningful confidence intervals". – Zhubarb Jun 10 at 16:36
• I recommend this for reading: ftp.cs.ucla.edu/pub/stat_ser/r354-corrected-reprint.pdf It's a great primer for causal inference written by Pearl. Essentially, your problem is one of identifiability. In principle, purely observational data cannot yield any causal inference. However, if you assume a causal model, then a quantity such as $P(Y=y \mid do(x=x'))$ may or may not be identifiable considering what part of your model is observed (appears in the data). Note, this is not the same as $P(Y=y \mid x=x')$, the latter being a purely associational (not causal) quantity. – Bridgeburners Jun 12 at 18:15

The key part of the application of the g-formula is the identifiability assumptions. Common ones are exchangeability, causal consistency, and positivity. This paper describes the conditions. Exchangeability is the assumption that there is no common cause of $$x_j$$ and $$y$$. This is the d-separation rule of do-calculus. Causal consistency implies that you have sufficiently specified $$x_j$$ so that there is no difference in $$y$$ by residual differences in $$x_j$$. You can think about this as assuming taking an aspirin in the morning versus night has no difference on mortality. Positivity is the assumption that all individuals have a non-zero probability of having all values of $$x_j$$. Westreich & Cole provide further details.
Lastly, since you mentioned that $$x_j$$ is a continuous variable, you will need to consider how $$x_j$$ is changed. You can imagine that everyone's $$x_j$$ is shifted by some quantity $$\delta$$. Alternatively, you can set a threshold, $$\gamma$$, where those below (above) are shifted to $$\gamma$$. The remainder of individuals retain their $$x_j$$ value
• There is also a cross-fit procedure. Basically, you divide your sample into two pieces and fit whatever algorithm you want to the data. You then use the algorithm from one sample split to predict the outcomes under the different distribution of $x_j$ for the other sample split. This procedure is suppose to attain root-n convergence (even for estimators that converge slower than root-n), so bootstrapping would be a valid way to get confidence intervals. This paper gives the mathematics arxiv.org/abs/1801.09138 It is still a fairly new approach, so I haven't seen any applications yet – pzivich Jun 13 at 15:21