6
$\begingroup$

Consider the following hypothetical. I am an employee of Acme Inc., and I want to quantify the impact on spending (in $) of joining the Customer Loyalty Club (CLC).

Of course, only customers who were already particularly interested in/enthusiastic about will Acme join the CLC, so we can't quantify the treatment effect via a simple difference in mean spend between those who join the CLC and those who don't.

Now, interest in Acme is unobservable. (Let's suppose that, anyway.) We might search, then, for a proxy variable that we can adjust for instead. Spend in the previous year seems like it may make sense. Someone who is interested in Acme's products is likely to have spent more money in the previous year than a customer who is less interested. Indeed, a customer's interest in Acme plausibly has a causal impact on the amount that customer spends at Acme in any given year.

Here's my question. I don't see how to justify adjusting for spend in the previous year, assuming the DAG shown below. Am I missing something?

EDIT: To clarify, the back-door criterion doesn't seem to help, since conditioning on Spend_in_Prev_Year doesn't block any back-door paths. Is there some other justification that can be given for conditioning on Spend_in_Prev_Year (where that justification may be a plausible way of re-drawing this DAG)?

enter image description here

$\endgroup$
2
  • 2
    $\begingroup$ I'm still digging through the topic myself, but... if there's a causal link between the interest and last year's spending, you'd need to ensure whatever value you propose for the interest's impact on the current year's spending also matches up to your expectation on last year's spending conditional on that proposed value, right? Otherwise you'd be violating the established weights of either the red or the black arrows. $\endgroup$
    – jkm
    Dec 24, 2019 at 0:21
  • $\begingroup$ Hrm, I'm not sure. Can you say more? $\endgroup$ Dec 24, 2019 at 1:01

2 Answers 2

5
$\begingroup$

Since Interest_in_Acme is unobservable, the average causal effect of Loyalty Club Membership on Spend is unidentifiable. However, there is an important exception to that rule, that is if Interest_in_Acme is perfectly correlated ($r=1.0$ or $r=0.0$) with Spend_in_Prev_Year. If those two variables are perfectly correlated (i.e. contain the same information), then Spend_in_Prev_Year can be adjusted for instead and used to identify the average causal effect.

In the much more likely scenario of Interest_in_Acme being somewhat correlated with Spend_in_Prev_Year, a somewhat biased estimate of the average causal effect can be obtained. The more that the two are correlated, the less biased the estimate adjusted for Spend_in_Prev_Year.

A simple simulation study

To demonstrate the concept, below is a simple simulation study (Python 3.5+ code). Let $L$ be Interest_in_Acme, $L^*$ be Spend_in_Prev_Year, $A$ be Loyalty Club Membership, $Y(a)$ be the potential Spend under treatment plan $a$, and $Y$ be the observed spending. For simplicity, my simulation uses binary variables. To reduce variability to sample size, I set $n=1,000,000$. For the estimator of the average causal effect, I used the standardized mean difference (i.e. g-formula, do-calculus, etc.)

import numpy as np
import pandas as pd

# Simulation parameters
n = 1000000
correlation = 1.0
np.random.seed(20191223)

# Simulating data set
df = pd.DataFrame()
df['L'] = np.random.binomial(n=1, p=0.25, size=n)
df['L*'] = np.random.binomial(n=1, p=correlation*df['L'] + (1-correlation)*(1-df['L']), size=n)
df['A'] = np.random.binomial(1, p=(0.25 + 0.5*df['L']), size=n)
df['Ya0'] = np.random.binomial(1, p=(0.75 - 0.5*df['L']), size=n)
df['Ya1'] = np.random.binomial(1, p=(0.75 - 0.5*df['L'] - 0.1*1 -0.1*1*df['L']), size=n)
df['Y'] = (1-df['A'])*df['Ya0'] + df['A']*df['Ya1']

# True causal effect
print("True Causal Effect:", np.mean(df['Ya1'] - df['Ya0']))

# Standardized Mean Estimator
l1 = np.mean(df['L*'])
l0 = 1 - l1
r1_l0 = np.mean(df.loc[(df['A']==1) & (df['L*']==0)]['Y'])
r1_l1 = np.mean(df.loc[(df['A']==1) & (df['L*']==1)]['Y'])
r0_l0 = np.mean(df.loc[(df['A']==0) & (df['L*']==0)]['Y'])
r0_l1 = np.mean(df.loc[(df['A']==0) & (df['L*']==1)]['Y'])
rd_stdmean = (r1_l0*l0 + r1_l1*l1) - (r0_l0*l0 + r0_l1*l1)
print('Standardized Mean Risk Difference:', rd_stdmean)

Below are the results for some various correlations (you can also run this code and change the correlation parameter to see the result of the various changes. Note that $r=0.50$ is no correlation

True Average Causal Effect: -0.124

$r=1.0$: -0.123

$r=0.99$: -0.136

$r=0.50$: -0.347

$r=0.05$: -0.180

Summary

As a justification, you may believe that Interest_in_Acme and Spend_in_Prev_Year are highly correlated meaning you may be close to the true average causal effect. While you can't fully identify, you may believe that those two variables are highly correlated so your estimate is close to the truth. As a final note, this problem becomes more complicated for continuous variables since functional forms of variables may differ.

$\endgroup$
3
  • $\begingroup$ Fantastic. Thank you! A follow-up question, if you have time: where did you get the formula for p in the line that defines df['L*']? I'm curious because I was just checking that it generates a r.v. with the given correlation, but numpy is suggesting it doesn't (unless I'm doing something stupid here, which is quite possible!): correlation = 0.9 df = pd.DataFrame() df['L'] = np.random.binomial(n=1, p=0.25, size=n) df['L*'] = np.random.binomial(n=1, p=correlation*df['L'] + (1-correlation)*(1-df['L']), size=n) np.corrcoef(df['L'], df['L*']) $\endgroup$ Dec 24, 2019 at 2:24
  • $\begingroup$ Sorry about the formatting above, I'm trying to figure out how to fix that. Also, do you know of any place where Pearl discusses the exception you mention? I've only read Book of Why all the way through (and I'm in the middle of the Primer), and I don't recall seeing him mention that but would like to follow-up. $\endgroup$ Dec 24, 2019 at 2:29
  • 1
    $\begingroup$ Regarding df['L*'] that is a trick to generate correlation between the variables (it isn't the same as $r$ in linear regression since we are working with binary variables). Essentially if L == 1 then L* will equal 1 correlation percent of the time. Everything after the + is to take care of the case when L == 0. As for a source, I couldn't point you to where Pearl discusses it. My knowledge of this relationship came from my epidemiology PhD classes. $\endgroup$
    – pzivich
    Dec 24, 2019 at 2:47
5
$\begingroup$

Exact point identification is not possible here, but adjusting for Spend_in_Prev_Year does partially block the backdoor path, so that would be the rationale for it. As a general advice, you should adjust for the proxy in the absence of the true confounder (there are exceptions, the proxy could be opening other backdoor paths for instance, but that's not the case in your example).

Now I should add, since you know you didn't fully block the backdoor path, you should perform a sensitivity analysis---we know by construction that your estimate is biased, so we want to judge how biased it could be.

For instance, if you are using a linear model, you can perform a fairly general, yet simple, sensitivity analysis by simply comparing how much more variation the true variable could explain of your treatment and your outcome, as compared with the proxy variable you have measured (see Cinelli and Hazlett 2020 - ungated version). If you think the proxy does a good job, and the the true variable can't be much stronger than the proxy, then it is likely your estimate is not much biased.

I will show here an example in R using the package sensemakr. Suppose you measured the confounder $X^*$ instead of $X$, and you obtained the following estimates,

set.seed(10)
n <- 1e4
x <- rnorm(n)
xs <- x + rnorm(n)
d <- rbinom(n, 1, plogis(x))
y <- d + x + rnorm(n)

model <- lm(y ~ d + xs)
model
#> 
#> Call:
#> lm(formula = y ~ d + xs)
#> 
#> Coefficients:
#> (Intercept)            d           xs  
#>     -0.2411       1.4882       0.4537

Now you wonder whether the whole estimate of $1.48$ could be due to bias, because you didn't control for the "true" $X$.

Here is a sensitivity plot showing how much stronger the true $X$ would need to be, both with its association with the treatment $D$ and with the outcome $Y$, to fully explain away the observed association (as compared to the proxy, and above what the proxy already explains). As you can see in the example, the true variable would need to be 3 times as strong as the proxy to fully explain away your estimate. If you think that's unlikely and that you think that the true variable could (additionally) explain only as much or twice as much as what has already explained by the proxy, then you can claim the true effect is not less than 0.54 (in our case we know it is 1).

library(sensemakr)
#> See details in:
#> Carlos Cinelli and Chad Hazlett (2020). Making Sense of Sensitivity: Extending Omitted Variable Bias. Journal of the Royal Statistical Society Series B.
sense <- sensemakr(model = model, treatment = "d", 
                   benchmark_covariates = "xs", 
                   kd = 1:3)
plot(sense)

enter image description here

$\endgroup$
3
  • 1
    $\begingroup$ Thank you! Very interesting. I'm familiar with Rosenbaum's sensitivity analysis technique for matched datasets, but this technique is new to me. I will read your paper. (BTW, your link to the ungated version isn't working, but it appears that this URL works instead.) $\endgroup$ Dec 24, 2019 at 12:02
  • 2
    $\begingroup$ Thanks @KerasNewcomer just fixed the link (this one here researchgate.net/publication/…) $\endgroup$ Dec 24, 2019 at 18:09
  • $\begingroup$ Hi Carlos, thanks so much. I have a question regarding your simulation. What if the simulation would be like this: xs <- x + z + rnorm(n). And lets say we measured z and it is only related to xs. Am I right that we then also should include z into the regression that uses the proxy? At least in my simulation the estimate of the proxy gets closer to the true x effect then. However, normally it seems to be advised to not include predictors of the predictors if they are not related to the outcome themselves (see model 9 in your article "Good and Bad Controls"). $\endgroup$
    – Jaynes01
    Feb 21, 2022 at 9:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.