8
$\begingroup$

Let's say there are two types of treatment, namely treatment A and treatment B.

A subject can be in one of these categories:

  1. get treatment A and then treatment B.
  2. get treatment B and then treatment A.
  3. get only treatment A.
  4. get only treatment B.
  5. get no treatment.

There are several ways we can construct the control group. For example, we can:

  1. construct a pure control group who gets no treatment.
  2. construct control groups for treatment A and treatment B independently. A portion of the control subjects for treatment A may have gotten treatment B. In other words there can be overlap.

Edit: Note that I only have ability to design and assign control group in the experiment. The treatment order happens naturally in the test group without moderation.

Also the post-experiment analysis to estimate treatment effects get tricky since there are two types of treatment, and they may interact with each other.

Anyone have good pointers for reference I can read on?


Edit:

Context was requested in the comment section, so I'd like to provide some exemplar cases:

Weight study

  • Treatment A: doing yoga daily
  • Treatment B: running 3 miles daily
  • Metric: body weight in kg

Button on a web page

  • Treatment A: change button's location
  • Treatment B: change button's color
  • Metric: click-through-rate

Completing online course

  • Treatment A: chunking hour-long videos into smaller sessions
  • Treatment B: sending reminder emails
  • Metric: course completion rate

In general, carrying out multiple experiments to estimate effect of two types of treatment on the same metric requires more time and/or subjects and increases cost. It also ignores potential interaction effect of treatment A and treatment B.

$\endgroup$
6
  • 1
    $\begingroup$ Some shameless-self promo (hence a comment): I looked at some experiment designs for situations similar to what you've described from an effect size / test power perspective. It doesn't consider the case where you apply two treatments sequentially though - arxiv.org/pdf/2007.11638.pdf . $\endgroup$
    – B.Liu
    Nov 19, 2021 at 22:32
  • 1
    $\begingroup$ The experiment design should correspond to the the questions that you seek to answer and the setting. Those details are missing in the information you have provided. For instance, an experiment on gasoline additives for race cars would look very different from a preschool intervention. $\endgroup$
    – dimitriy
    Nov 20, 2021 at 5:42
  • $\begingroup$ @dimitriy edited question. $\endgroup$ Nov 20, 2021 at 7:24
  • $\begingroup$ Please see my answer for what I have done. If there are better answers that go beyond the basics, I will accept it. If not, I will leave this question open. $\endgroup$ Nov 23, 2021 at 23:32
  • $\begingroup$ Which control group are you saying you can assign and which can you not assign? $\endgroup$
    – Ben
    Nov 23, 2021 at 23:59

3 Answers 3

1
+100
$\begingroup$

OP mentioned that they ended up selecting control groups for the treatments independently.

In this case, if there is no clear bias in the assignment mechanisms of treatment A and treatment B (e.g. somehow applying treatment B also increases probability of treatment A), the subtraction in the calculation of average treatment effect should naturally cancel out the impact of the other treatment on both treatment and control group.

If one suspects that there is some dependency between treatment A and treatment B, it is always possible to manually test for the independence of the two random events.

In the case of treatment A and treatment B being dependent, without loss of generality, one can adjust for the bias from treatment B by conducting regression adjustment with the propensity scores of treatment A given treatment B. I suspect that's what OP went for directly without testing for independence.

$\endgroup$
2
$\begingroup$

When conducting these kinds of experiments there is a natural control-group for each marginal and conditional effect in the experiment. The figure below shows an experimental flowchart for the possible categories of each participant. The "No Treat" category acts as the control group for determining the marginal effects of Treatment A or B, the "Treat A" category acts as the control group for determining the conditional effect of Treatment B given Treatment A, and the "Treat B" category acts as the control group for determining the conditional effect of Treatment A given Treatment B. (And of course, these last two groups might end up being effectively the same thing if the order of the treatments makes no difference; in case the order of treatment does make a difference, we leave them separately for now.)

Now, ideally you would be able to randomly assign your participants to these five groups and then undertake estimation of all the marginal and conditional causal effects in the process. However, if you have no control over the occurrence of a second treatment given a first treatment then you cannot make reliable causal inferences about the conditional causal effects. You can still estimate the relevant conditional distributions in a statistical sense, but the connection to causality is effectively lost.

As to references for causal inference, this depends largely on your existing knowledge of statistics and mathematics, and the degree to which you want a formalisation of the rules of causality. A standard text in the field is Pearl (2009), but this is written for people who have some quantitative training in mathematics or statistics. The main thing to remember when conducting causal inference is that it works by undertaking regular statistical modelling/inference (usually using regression analysis) in a context where we can sever causal relationships between variables using randomisation of assigned variables and blinding protocols.

enter image description here

$\endgroup$
1
  • $\begingroup$ Thank you. Please see my answer for what I have done. $\endgroup$ Nov 23, 2021 at 22:20
0
$\begingroup$

I opted to construct control groups for treatment A and treatment B independently. Each control group is randomly selected. It allowed the two teams at my work to save time and effort for communications.

I used matching to adjust for any potential bias in the measurement of treatment effect of A and B introduced by the other treatment (which turned out to be an insignificant adjustment). I won't go into the specifics here, cause it requires context that I am not allowed to share.

To estimate interaction effect, it's done with a CUPED-like method.

If there are better answers that go beyond the basics, I will accept it. If not, I will leave this question open.

Special thanks to the paper shared in the comment section for giving me a starting point of thinking about the problem.

$\endgroup$
2
  • $\begingroup$ Can you please clarify what you mean here? When you say you contructed control groups for treatment A and treatment B, so you mean that those are the control groups (i.e., the blue and red circles in above answer) or do you mean that the control group for them is the one composed of people with no treatment (i.e., the black circle in the above answer). $\endgroup$
    – Ben
    Nov 24, 2021 at 0:01
  • $\begingroup$ @Ben Thank you for the question. You can consider it as two teams conducting two independent, parallel experiments on the same sample subjects. Each team randomly selects their own control group without knowledge of what the other team is doing. For example, a portion of the control group for treatment A will have treatment B applied to them. The main challenge is how to correct for potential bias from difference in treatment B' impact between treatment A's test&control group. $\endgroup$ Nov 25, 2021 at 1:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.