1
$\begingroup$

I have a question about how far the naive treament effect estimator can go wrong when there is appreciable within group variance vs between group variance in an RCT, and what we can do about it practically.

I am running an experiment where I have the within group variance is appreciably larger than the between group variance. To put some numbers on this we are trying to see if there is ~\$2 between groups, but the within group variance is ~\$20. For a host of operational reasonsm there's not much I can do to change this.

The problem is that the pre-experiment differences in the groups can be ~$1 or ~$2 already. How can I go about estimating the treatment effect (the between group difference) in this case?

When doing an RCT, the treatment effect is always the simple difference between the treated and control groups:

$$ \Delta_{ATE} = \frac{1}{n_T}\sum_{i\in n_T}Y_{i} - \frac{1}{n_C}\sum_{i\in n_C}Y_{i}$$

And we can use pre-experimental differences to improve the precision (lower the standard errors) of this estimate, but there's fundamentally nothing we can do to alter the estimate.

To verify this, I have simulated the simple difference, CUPED, Difference in Differences, Latent Analysis of Covariance, and Analysis of Changes estimators. I found that in expectation the estimators all had the same value (the simple difference), but their standard errors differed when there was essentially no within group variance. When there was some between group variance then all these estimators had the same standard errors.

is this correct? Is there a preferred estimator in this case, or am I unable to do anything better than the simple difference? Would e.g. a panel data model help (I do not know much about these)? Also, in future, what are the sample size guidelines for this case, we have currently been using Lehr's rule:

$$ N = 16\frac{\sigma^2}{\Delta^2}$$

to estimate sample sizes.

Treatment effect estimates when the within group variance is ~ between groups

Treatment effect estimates when the within group variance is > between groups

I generated my data with the following function:

import pandas as pd
import numpy as np
import seaborn as sns

def generate_data_v2(alpha, beta, gamma, delta, rng, between_variation, N, seed=42):
# Individuals
i = range(1, N+1)

# Treatment status
d = rng.binomial(1, 0.5, N)

# Individual outcome pre-treatment
# Pre-treatment outcomes are generated with some base level of variation
y0 = alpha + beta*d + rng.normal(0, 1, N)

# Modifying post-treatment outcomes to incorporate between_variation control
# This will ensure the post-treatment variation across units is driven by between_variation parameter
y1 = y0 + gamma + delta*d + rng.normal(0, between_variation, N) * d  # Amplifying variation for treated units

# Generate the dataframe
df = pd.DataFrame({'i': i, 'ad_campaign': d, 'revenue0': y0, 'revenue1': y1})

return df

df = generate_data_v2(alpha, beta, gamma, delta, rng, between_variation=5, N=100)

results_no_group_var = simulate(
    func=generate_data_v2,  # Your data generating function
    func_args={
        'alpha': alpha, 
        'beta': beta, 
        'gamma': gamma, 
        'delta': 0.0, 
        'rng': rng, 
        'N': N, 
    'between_variation': 1, 
    },
    K=K,
    x='revenue0'
)

results_group_var = simulate(
    func=generate_data_v2,  # Your data generating function
    func_args={
        'alpha': alpha, 
        'beta': beta, 
        'gamma': gamma, 
        'delta': 0.0, 
        'rng': rng, 
        'N': N, 
    'between_variation': 5, 
    },
    K=K,
    x='revenue0'
)

sns.kdeplot(data=results, x="Estimate", hue="Estimator");
plt.title('Simulated Distributions');
$\endgroup$
2
  • $\begingroup$ I'm not really sure I follow what the problem is. Could you demonstrate what the problem is using your data? $\endgroup$ Apr 11 at 15:41
  • $\begingroup$ @DemetriPananos the issue is that I'm not certain what the problem is, other than a stakeholder telling me that our software is not showing the 'correct' treatment effect! $\endgroup$
    – Tom Kealy
    Apr 11 at 20:11

1 Answer 1

2
$\begingroup$

So from what I understand:

  • You randomized users to one of two exposures
  • The hope is that you can detect a differences of \$ 2 dollars per user between groups
  • However, the pre-experiment data suggests that this difference existed prior to the randomization, and that hence any detected effects may not be caused by the exposure, but are simply an artifact of between group differences that already existed.

I don't think there is much to think about here. The two groups should have a pre-expsosure mean outcome which is the same in expectation. You should be able to just compare means, but I understand why some may be concerned.

This leaves techniques like Difference in Differences, which should adjust for any pre-existing differences between groups. The additional parameters for this model are not really needed, but if the data are plentiful then I don't see an issue with this approach per se.

The point estimate in either case should be similar, or at the very least very close. Through some simulation, I've found that treatment differences can have very high MSE when you first check to see if there is a pre-treatment difference and then choose to do diff in diff, so that is the only thing I can really forsee changing the estimate greatly.

$\endgroup$
1
  • $\begingroup$ Thank you! I feel like I'm going mad: that's what I've said already! $\endgroup$
    – Tom Kealy
    Apr 11 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.