The first question you have to ask yourself when you are facing a problem like this is "What exactly am I trying to estimate?", and be precise. This will usually start to guide you in the right direction. These are two (of several) possible estimands that may be of interest in an experiment like this:
The causal effect of seeing a push notification on the outcome of interest. (Similar to Intention to Treat Effect [ITT])
The causal effect of clicking a push notification on the outcome of interest among people who would click on it if they were to see it. (Similar to Complier Average Causal Effect [CACE])
Typically, you would want to estimate the values of these estimands using techniques based on the randomized assignment of an intervention to a treatment group (as in a clinical trial). The challenge with your experiment is that the the original treatment (sending the push notification) was assigned to everyone, and thus there is no control group for this paticular intervention. Therefore you need to rely on methods for causal inference in observational studies because the only possible "treatment" for which you can construct a control group is a person seeing the push notification. I think this is more reasonable than having a treatment and control group based on whether or not someone clicked the push notification.
Thus, under this framework, your treatment group is comprised of everyone who sees the push notification, and the control group is comprised of everyone who does not. However, there are multiple complications:
First, obviously, the assignment to seeing or not seeing the push notification is not random, and is likely dependent on characteristics of the individual and their environment. In this scenario, many people will rely on methods associated with the propensity score to establish treatment and control groups that are "similar" in their probability of having been assigned the intervention of interest (seeing the push notification in this case). It is important to understand the assumptions underlying to the propensity score in order to justify its use. For example, you need to assume:
- Everyone in your sample had a non-zero probability of seeing the push notification
- The set of variables you have that describe the individual and their environment is sufficient to create independence between the potential outcomes and whether or not they saw the notification.
(See strongly ignorable treatment assignment in section 5.1 )
Second, you have to decide which of the two estimands, written above as 1. and 2., you want to estimate. The first simply describes the effect of seeing the push notification on the outcome of interest, regardless of click status. The second requires you to consider who in the control group would have clicked on the push notification were they to have seen the push notification. This is an important distinction, and the consideration is necessary because it doesn't make sense to estimate the effect of clicking on the notification among people who would never click on it, regardless of whether they saw it or not. This is often called the "complier average causal effect" (See here). There is a lot of work that has gone into the appropriate methods for estimating each of these two estimands, however it is important to understand which one is more suitable to answer your question of interest. That is something you must decide based on your goals.
