I have a pool of users and I am able to send them push notifications. I want to measure the effect of receiving a push notification on a usage metric, such as using a feature in the game within the next 7 days. So I want to measure the effect of a dichotomous variable (seeing a notification) on a dichotomous variable (using the feature within 7 days of receiving the notification).
So I can choose to send my notifications to only part of my user base. However, there are many variables, some of which I know (device OS, time of reception, user permissions...) and some of which I don't know (possibly user sociodemographic profile, user interest in the app...) that will lead a user to actually seeing or not seeing my notification. For this reason, the group of users who saw a notification is not equivalent / comparable to the group of users that I have sent notifications to, but who did not actually see it.
I want to correct for this using propensity score stratification or matching methods. Given my pool of 100K users, I will send notifications to 90K users (Notification Sent) and keep 10K users aside (Notification Not Sent). Devices will be assigned to one of these two groups randomly. Then, let's say only 60K users from the Notification Sent group will actually see the notification (Notification Seen). The devices in Notification Not Sent will, by definition, never see a notification.
Now I can do the following: within the Notification Sent group, I will compute propensity scores, by fitting a logistic regression model, using the Notification Sent dataset, which are the predicted probabilities of seeing the notification given a set of cofounders.
Then, within the Notification Sent group, I will build propensity scores stratas for the subgroup which saw the notification and the subgroup that did not see the notification (for instance, the propensity score quintiles). Then, for each stratum, I will calculate the risk ratio for my dependent dichotomous variable: RR = P(use_feature=1 | saw_notification=1) / P(use_feature=1 | saw notification = 0) and compute the average risk ratio from my five strata.
I could do something a little different. I am not sure if it adds any value, if it's "better" or "worse". After computing the propensity scores, I can now use my trained logistic regression model to compute propensity scores for devices in the Notification Not Sent group. Then, I can compare two subgroups, split in similar strata: the subgroup within Notification Sent that saw the notification and the Notification Not Sent group (which by definition did not see the notification). I can then calculate the risk ratio in a similar fashion). Is this method valid?
Is it "better" or "worse" (or equivalent) than simply using my observational data as is?
I feel like it's overall pretty pointless and that it might have some issues. That is, I will be matching people from the Notification Not Sent group with people from the Notification Sent group, some of whom would have seen the notification if I had not artifically kept them aside. Does this have any consequence?
Let me know if you need clarifications, it's pretty hard to explain.