1
$\begingroup$

Say I have an obstacle course, which not everyone completes, though, globally, most do. I hypothesize that the treatment, drinking Gatorade, will cause an increase in the obstacle course completion rate (CR).

Due to a fluke in randomization, the test group has a pre-experiment 95% CR but an in-experiment 98% CR. Whereas the control has a pre-experiment 94% CR and a 96% in-experiment CR.

In both time windows, the test group is beating the control group; given that this bias exists, I cannot simply do a Z proportions test on the experimental data. I need to correct for this pre-experiment bias in some way.

My question is: How is this most frequently / easily done?

DAG

Edit 1: DAG added (courtesy Excalidraw). The fundamental issue is that I believe that two factors affect Completion Rate, both ability and treatment (gatorade.) Because pre-experiment bias exists in my data (it could be anything from better running shoes, increased stamina, height, etc) which is observed by the CR of test > CR control pre-experiment. I believe that ability is moderating the effect of gatorade such that drawing conclusions from the experimental results alone, might tell an incomplete story.

Edit 2: Perhaps this situation is best solved with a regression model, eg the x=pre-experiment ratios, y=post-experiment ratios and what really matters are the slopes for exposure (yes/no)?

$\endgroup$
4
  • $\begingroup$ Pre experiment completion rate should not matter. You need to compare between those who received the intervention and those who did not. So long as you can randomize to control and test effectively, and adhering to the treatment is not hindered in any way, then a test of proportions should be fine. $\endgroup$ Aug 19, 2022 at 21:17
  • $\begingroup$ @DemetriPananos, I don't understand why pre-experiment completion rate does not matter; in this example, is the treatment's effect not moderated by the latent variable, ability, of which (in pre-experiment) the test group has elevated values in comparison to control? (eg I'm viewing completion rate as an instrument variable for the latent variable, ability.) $\endgroup$
    – jbuddy_13
    Aug 19, 2022 at 21:31
  • $\begingroup$ It isn’t necessarily the case the effect is moderated from your description (unless the proportions you list are the true counter factual proportions). Can you include a DAG of your assumptions? $\endgroup$ Aug 19, 2022 at 21:41
  • $\begingroup$ @DemetriPananos, sure added! Plus a description of the DAG and concerns in the edit section. $\endgroup$
    – jbuddy_13
    Aug 19, 2022 at 21:49

2 Answers 2

0
$\begingroup$

Thanks for the DAG.

As it is written, the intervention has no confounders so you’re free to just compare sample means with an appropriate test.

Ability may modulate the effect of the intervention, and adjusting for pre experiment ability may reduce the variance, but it is not necessary to adjust for. There is no bias here.

Now, this assumes that you have a simple random sample. If your sample consists mostly of people who have innate ability, and this is not reflected in the population, then the inference is troubled. Is this the case you’re trying to describe?

$\endgroup$
1
  • $\begingroup$ >If your sample consists mostly of people who have innate ability, and this is not reflected in the population, then the inference is troubled. I'm not positive that there is this issue but I believe that it is a possible risk given that the test CR > ctrl CR pre experiment. And what I didn't capture above is that the difference, pre-exp, is statistically significant. $\endgroup$
    – jbuddy_13
    Aug 19, 2022 at 22:19
0
$\begingroup$

I ended up using logistic regression and controlling for the confounding variables, which were likely to have caused the pre-experiment bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.