3
$\begingroup$

I’m getting a bit confused about who to include in the control and treatment pools and would appreciate the help. I need to estimate the effect of treatment where 100% of the population was assigned a push notification that leads to a treatment, but not all of them saw it and are therefore not treated (or know they were assigned). The pool of control/treatment candidates can be broken down as:

  1. People that don’t know they were sent a notification and therefore never receive treatment
  2. People that see the notification, but then don’t choose it (treatment). It’s possible that knowledge of the notification has a causal effect on outcome.
  3. People that see the notification and choose to take treatment by clicking it
  4. People that see and select the notifications, but their selection into treatment is driven by unobserved variables that would likely also influence outcome.

My approach was going to be using matching methods between treatment (Group 3) and control (Group 1), removing group 2 and 4 from the control/treatment samples entirely. I was going to condition on variables that would cause someone to self select into treatment group 3 that also influence outcome. Am I estimating the treatment effect in the right way here?

$\endgroup$
1
  • $\begingroup$ There is a distinction between Groups 1 and 3 in that those in Group 3 (and 2 and 4) see notifications while those in Group 1 do not. How do you know this is not "driven by unobserved variables that would likely also influence outcome"? $\endgroup$
    – Henry
    May 7, 2023 at 0:24

3 Answers 3

1
$\begingroup$

The first question you have to ask yourself when you are facing a problem like this is "What exactly am I trying to estimate?", and be precise. This will usually start to guide you in the right direction. These are two (of several) possible estimands that may be of interest in an experiment like this:

  1. The causal effect of seeing a push notification on the outcome of interest. (Similar to Intention to Treat Effect [ITT])

  2. The causal effect of clicking a push notification on the outcome of interest among people who would click on it if they were to see it. (Similar to Complier Average Causal Effect [CACE])

Typically, you would want to estimate the values of these estimands using techniques based on the randomized assignment of an intervention to a treatment group (as in a clinical trial). The challenge with your experiment is that the the original treatment (sending the push notification) was assigned to everyone, and thus there is no control group for this paticular intervention. Therefore you need to rely on methods for causal inference in observational studies because the only possible "treatment" for which you can construct a control group is a person seeing the push notification. I think this is more reasonable than having a treatment and control group based on whether or not someone clicked the push notification.

Thus, under this framework, your treatment group is comprised of everyone who sees the push notification, and the control group is comprised of everyone who does not. However, there are multiple complications:

First, obviously, the assignment to seeing or not seeing the push notification is not random, and is likely dependent on characteristics of the individual and their environment. In this scenario, many people will rely on methods associated with the propensity score to establish treatment and control groups that are "similar" in their probability of having been assigned the intervention of interest (seeing the push notification in this case). It is important to understand the assumptions underlying to the propensity score in order to justify its use. For example, you need to assume:

  • Everyone in your sample had a non-zero probability of seeing the push notification
  • The set of variables you have that describe the individual and their environment is sufficient to create independence between the potential outcomes and whether or not they saw the notification. (See strongly ignorable treatment assignment in section 5.1 )

Second, you have to decide which of the two estimands, written above as 1. and 2., you want to estimate. The first simply describes the effect of seeing the push notification on the outcome of interest, regardless of click status. The second requires you to consider who in the control group would have clicked on the push notification were they to have seen the push notification. This is an important distinction, and the consideration is necessary because it doesn't make sense to estimate the effect of clicking on the notification among people who would never click on it, regardless of whether they saw it or not. This is often called the "complier average causal effect" (See here). There is a lot of work that has gone into the appropriate methods for estimating each of these two estimands, however it is important to understand which one is more suitable to answer your question of interest. That is something you must decide based on your goals.

$\endgroup$
1
$\begingroup$

The attrition issues you are facing here are common issues in trials involving a treatment imposed by the researchers. In many contexts this is dealt with by considering causal effects on the basis of intention-to-treat rather than on the basis of progression into the substantive treatment. This is a trade-off, insofar as you ask a less valuable question (i.e., what is the causal effect of the initial notification), but you can make a robust inference about this effect that does not hinge on assumptions that progression to the substantive treatment is independent of confounding effects.

Having said this, your big problem here is not attrition --- it is the fact that you do not appear to have a proper control group. If you want to proceed using an RCT with intention-to-treat analysis, you ought to have a larger group that is divided randomly into a control and treatment group, and you would only send the notification to the treatment group. My recommendation would be to rethink the underlying experimental design, and consider adopting an alternative design where you have a control group composed of people who do not receive the notification for treatment.

$\endgroup$
0
$\begingroup$

As Henry pointed out, if the four groups are defined and each individual be able to be clearly identified into one of the group?

Assume the groups are correctly defined, from the analysis viewpoint, yes, I think the approach is on the right way. However the removal of the group 2 and 4 in the matching step is ok but not necessary as the data of group 2 & 4 may also provide valuable information of the selection mechanism. After the matching the weights arising from the matching should be considered in the analysis, no matter what analysis method is to be used in the next step.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.