Propensity Score Stratification: Design Clarifications I have a pool of users and I am able to send them push notifications. I want to measure the effect of receiving a push notification on a usage metric, such as using a feature in the game within the next 7 days. So I want to measure the effect of a dichotomous variable (seeing a notification) on a dichotomous variable (using the feature within 7 days of receiving the notification).
So I can choose to send my notifications to only part of my user base. However, there are many variables, some of which I know (device OS, time of reception, user permissions...) and some of which I don't know (possibly user sociodemographic profile, user interest in the app...) that will lead a user to actually seeing or not seeing my notification. For this reason, the group of users who saw a notification is not equivalent / comparable to the group of users that I have sent notifications to, but who did not actually see it.
I want to correct for this using propensity score stratification or matching methods.
Given my pool of 100K users, I will send notifications to 90K users (Notification Sent) and keep 10K users aside (Notification Not Sent). Devices will be assigned to one of these two groups randomly.
Then, let's say only 60K users from the Notification Sent group will actually see the notification (Notification Seen). The devices in Notification Not Sent will, by definition, never see a notification.
Now I can do the following: within the Notification Sent group, I will compute propensity scores, by fitting a logistic regression model, using the Notification Sent dataset, which are the predicted probabilities of seeing the notification given a set of cofounders.
Then, within the Notification Sent group, I will build propensity scores stratas for the subgroup which saw the notification and the subgroup that did not see the notification (for instance, the propensity score quintiles). Then, for each stratum, I will calculate the risk ratio for my dependent dichotomous variable: RR = P(use_feature=1 | saw_notification=1) / P(use_feature=1 | saw notification = 0) and compute the average risk ratio from my five strata.
I could do something a little different. I am not sure if it adds any value, if it's "better" or "worse". After computing the propensity scores, I can now use my trained logistic regression model to compute propensity scores for devices in the Notification Not Sent group. Then, I can compare two subgroups, split in similar strata: the subgroup within Notification Sent that saw the notification and the Notification Not Sent group (which by definition did not see the notification). I can then calculate the risk ratio in a similar fashion).
Is this method valid?
Is it "better" or "worse" (or equivalent) than simply using my observational data as is?
I feel like it's overall pretty pointless and that it might have some issues. That is, I will be matching people from the Notification Not Sent group with people from the Notification Sent group, some of whom would have seen the notification if I had not artifically kept them aside. Does this have any consequence? 
Let me know if you need clarifications, it's pretty hard to explain.
 A: You are actually in a somewhat rare instrumental variable situation, and might consider taking advantage of it instead of using propensity scores. With an instrumental variable, you need 4 conditions to identify the causal effect of seeing the notification on your outcome. I'm going to denote Z being sent the notification, A seeing the notification, and Y performing the action within 7 days.


*

*Relevance: Z is associated with A (it is; Z is a direct cause of A)

*Exclusion: Z has no effect on Y except through A (the only way the push can affect the outcome is if users see the notification)

*Exchangeability: There are no common causes of Z and Y (true; Z is randomly assigned)

*Monotonicity: There are no "defiers", i.e., people who always take the opposite treatment they are assigned (there can't be definers because people who don't receive the notification can't choose to see it).


Under these assumptions, you can identify the "complier average treatment effect" (CATE), where "compliance" is seeing the notification when it is pushed and not seeing it when it is not pushed. So, the CATE which is the causal effect of receiving and seeing the notification relative to not receiving and not seeing it. 
This can be identified as $\frac{E[Y|Z = 1]−E[Y|Z = 0]}{E[A|Z = 1]−E[A|Z = 0]}$ but since $E[A|Z = 0] =0$ you can compute it simply as $\frac{E[Y|Z = 1]−E[Y|Z = 0]}{E[A|Z = 1]}$
Skip propensity scores when you have randomization!
