# Correcting for pre-experiment bias in proportions test?

Say I have an obstacle course, which not everyone completes, though, globally, most do. I hypothesize that the treatment, drinking Gatorade, will cause an increase in the obstacle course completion rate (CR).

Due to a fluke in randomization, the test group has a pre-experiment 95% CR but an in-experiment 98% CR. Whereas the control has a pre-experiment 94% CR and a 96% in-experiment CR.

In both time windows, the test group is beating the control group; given that this bias exists, I cannot simply do a Z proportions test on the experimental data. I need to correct for this pre-experiment bias in some way.

My question is: How is this most frequently / easily done?

Edit 1: DAG added (courtesy Excalidraw). The fundamental issue is that I believe that two factors affect Completion Rate, both ability and treatment (gatorade.) Because pre-experiment bias exists in my data (it could be anything from better running shoes, increased stamina, height, etc) which is observed by the CR of test > CR control pre-experiment. I believe that ability is moderating the effect of gatorade such that drawing conclusions from the experimental results alone, might tell an incomplete story.

Edit 2: Perhaps this situation is best solved with a regression model, eg the x=pre-experiment ratios, y=post-experiment ratios and what really matters are the slopes for exposure (yes/no)?

• Pre experiment completion rate should not matter. You need to compare between those who received the intervention and those who did not. So long as you can randomize to control and test effectively, and adhering to the treatment is not hindered in any way, then a test of proportions should be fine. Aug 19, 2022 at 21:17
• @DemetriPananos, I don't understand why pre-experiment completion rate does not matter; in this example, is the treatment's effect not moderated by the latent variable, ability, of which (in pre-experiment) the test group has elevated values in comparison to control? (eg I'm viewing completion rate as an instrument variable for the latent variable, ability.) Aug 19, 2022 at 21:31
• It isn’t necessarily the case the effect is moderated from your description (unless the proportions you list are the true counter factual proportions). Can you include a DAG of your assumptions? Aug 19, 2022 at 21:41
• @DemetriPananos, sure added! Plus a description of the DAG and concerns in the edit section. Aug 19, 2022 at 21:49