2
$\begingroup$

A team fit a Random Forest model to a dataset $S=\{\mathbf{x}_i,y_i\}_{i=1}^N$, where $\mathbf{x}$ is a vector of continuous and categorical variables, and $y$ is a binary response. The model has a low CV-error, and the precision-recall curve looks good.

Now, for new samples they would like to perform what-if studies, i.e. for a fixed variable $x_j$ they would like to see how $p(y=1|x_j)$ changes as a function of $x_j$, all other things being equal. The goal of this activity is the following: for new samples $\mathbf{x}$, they want to modify the value of $x_j$ to $x'_j$ , so that $x'_jp(y=1|(x_1,\dots,x'_j,\dots,x_n))$ is maximized. However, by doing so, it seems to me that they modify the distribution $p(\mathbf{x},y)$, thus the results are not reliable anymore. I think this is related to causal inference. Is there a way to modify this process so that it actually works? Which methods should I study to help this team?

EDIT: An important point I forgot to mention is that $x_j$ is one of the continuous variables, unfortunately.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ This is a subschool of study under causal inference, usually referred to as "counter-factual reasoning". This is quite a good paper to get familiar with it: microsoft.com/en-us/research/wp-content/uploads/2013/11/… $\endgroup$
    – Zhubarb
    Jun 10, 2019 at 16:13
  • $\begingroup$ Thanks @Zhubarb. Actually, I thought counterfactuals would apply to a different problem - what would have happened, had I set $x_j$ to $c$ rather than to $d$. In this case, the new sample $\mathbf{x}$ is modified before I get the corresponding new result $y$. I thought do calculus would be more useful in our case. However, I know very little about both approaches, thus I may well be wrong. I'll read your reference with interest. If there's anything else you'd like to suggest, I'm all ears, as long as it's not an overly long book 🙂 $\endgroup$
    – DeltaIV
    Jun 10, 2019 at 16:33
  • $\begingroup$ @Zhubarb by the way, I forgot to specify that $x_j$ is continuous. I guess this complicates things quite a bit, doesn't it? $\endgroup$
    – DeltaIV
    Jun 10, 2019 at 16:35
  • 1
    $\begingroup$ The paper is focusing on "continuously valued variables with meaningful confidence intervals". $\endgroup$
    – Zhubarb
    Jun 10, 2019 at 16:36
  • 1
    $\begingroup$ I recommend this for reading: ftp.cs.ucla.edu/pub/stat_ser/r354-corrected-reprint.pdf It's a great primer for causal inference written by Pearl. Essentially, your problem is one of identifiability. In principle, purely observational data cannot yield any causal inference. However, if you assume a causal model, then a quantity such as $P(Y=y \mid do(x=x'))$ may or may not be identifiable considering what part of your model is observed (appears in the data). Note, this is not the same as $P(Y=y \mid x=x')$, the latter being a purely associational (not causal) quantity. $\endgroup$
    – Bridgeburners
    Jun 12, 2019 at 18:15

1 Answer 1

Reset to default
3
+100
$\begingroup$

This is linked to causal inference. The potential outcomes/counterfactual model will be useful for conceptualization. One easy way to do this is to use the g-formula. The g-formula is essentially the same as do-calculus. For an introductory paper, I would recommend this article by Snowden et al. 2011

The key part of the application of the g-formula is the identifiability assumptions. Common ones are exchangeability, causal consistency, and positivity. This paper describes the conditions. Exchangeability is the assumption that there is no common cause of $x_j$ and $y$. This is the d-separation rule of do-calculus. Causal consistency implies that you have sufficiently specified $x_j$ so that there is no difference in $y$ by residual differences in $x_j$. You can think about this as assuming taking an aspirin in the morning versus night has no difference on mortality. Positivity is the assumption that all individuals have a non-zero probability of having all values of $x_j$. Westreich & Cole provide further details.

Having said that, I would be careful with the usage of machine learning for causal inference. In general, confidence intervals for the g-formula are calculated by bootstrapping. There has been a lot of work showing that confidence intervals based on bootstrapping procedures is invalid for machine learning algorithms, like random forests. For an example, see Naimi & Edwards.

Lastly, since you mentioned that $x_j$ is a continuous variable, you will need to consider how $x_j$ is changed. You can imagine that everyone's $x_j$ is shifted by some quantity $\delta$. Alternatively, you can set a threshold, $\gamma$, where those below (above) are shifted to $\gamma$. The remainder of individuals retain their $x_j$ value

Improve this answer
$\endgroup$
4
  • $\begingroup$ Ah! The fact that bootstrapping is invalid for Random Forests is very interesting (also, very unfortunate). Would xgboost be better? The problem is that the team tried to use logistic regression, but the predictive accuracy was real crap. I will have a hard time convincing this team to change their ways, if, in order to predict consistently the results of interventions, they need to use a model with low predictive accuracy. Would a GAM model be ok, for what it concerns causal inference? $\endgroup$
    – DeltaIV
    Jun 13, 2019 at 14:19
  • 1
    $\begingroup$ The issue is not only random forests (the Naimi paper linked above use xgboost as an option). The issue is that the convergence rate for a lot of machine learning is below root-n. For bootstrapping to work, you need your estimator to have root-n convergence (how fast the estimator converges on the true value as a function of sample size). Methods like random forest or xgboost have much slower convergence. As a result, the confidence intervals are overly narrow. GAMs would be alright to use as far as I know $\endgroup$
    – pzivich
    Jun 13, 2019 at 15:16
  • 2
    $\begingroup$ There is also a cross-fit procedure. Basically, you divide your sample into two pieces and fit whatever algorithm you want to the data. You then use the algorithm from one sample split to predict the outcomes under the different distribution of $x_j$ for the other sample split. This procedure is suppose to attain root-n convergence (even for estimators that converge slower than root-n), so bootstrapping would be a valid way to get confidence intervals. This paper gives the mathematics arxiv.org/abs/1801.09138 It is still a fairly new approach, so I haven't seen any applications yet $\endgroup$
    – pzivich
    Jun 13, 2019 at 15:21
  • 1
    $\begingroup$ Wanted to follow-up on this, I have a pre-print on cross-fitting and machine learning. This shows why the g-formula approach with machine learning isn't the best choice. Our paper also talks about best practices: arxiv.org/abs/2004.10337 $\endgroup$
    – pzivich
    Apr 23, 2020 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.