1
$\begingroup$

I have a pool of users and I am able to send them push notifications. I want to measure the effect of receiving a push notification on a usage metric, such as using a feature in the game within the next 7 days. So I want to measure the effect of a dichotomous variable (seeing a notification) on a dichotomous variable (using the feature within 7 days of receiving the notification).

So I can choose to send my notifications to only part of my user base. However, there are many variables, some of which I know (device OS, time of reception, user permissions...) and some of which I don't know (possibly user sociodemographic profile, user interest in the app...) that will lead a user to actually seeing or not seeing my notification. For this reason, the group of users who saw a notification is not equivalent / comparable to the group of users that I have sent notifications to, but who did not actually see it.

I want to correct for this using propensity score stratification or matching methods. Given my pool of 100K users, I will send notifications to 90K users (Notification Sent) and keep 10K users aside (Notification Not Sent). Devices will be assigned to one of these two groups randomly. Then, let's say only 60K users from the Notification Sent group will actually see the notification (Notification Seen). The devices in Notification Not Sent will, by definition, never see a notification.

Now I can do the following: within the Notification Sent group, I will compute propensity scores, by fitting a logistic regression model, using the Notification Sent dataset, which are the predicted probabilities of seeing the notification given a set of cofounders.

Then, within the Notification Sent group, I will build propensity scores stratas for the subgroup which saw the notification and the subgroup that did not see the notification (for instance, the propensity score quintiles). Then, for each stratum, I will calculate the risk ratio for my dependent dichotomous variable: RR = P(use_feature=1 | saw_notification=1) / P(use_feature=1 | saw notification = 0) and compute the average risk ratio from my five strata.

I could do something a little different. I am not sure if it adds any value, if it's "better" or "worse". After computing the propensity scores, I can now use my trained logistic regression model to compute propensity scores for devices in the Notification Not Sent group. Then, I can compare two subgroups, split in similar strata: the subgroup within Notification Sent that saw the notification and the Notification Not Sent group (which by definition did not see the notification). I can then calculate the risk ratio in a similar fashion). Is this method valid?

Is it "better" or "worse" (or equivalent) than simply using my observational data as is?

I feel like it's overall pretty pointless and that it might have some issues. That is, I will be matching people from the Notification Not Sent group with people from the Notification Sent group, some of whom would have seen the notification if I had not artifically kept them aside. Does this have any consequence?

Let me know if you need clarifications, it's pretty hard to explain.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Maybe you can tell us a bit more about your study and give us context? Then we might be able to provide some added direction. For example other causal inference methods might be appropriate. It's difficult to determine what exactly you are trying to do with these propensity scores without context. By the way, subjects must have a non-zero probability of receiving the treatment for propensity models to be valid. $\endgroup$
    – StatsStudent
    May 29, 2017 at 21:23
  • $\begingroup$ I have completely modified my question. I hope that it's much clearer. My two core questions are: "which method is best and why?" and "Are both methods completely invalid?" $\endgroup$
    – Konrad
    May 30, 2017 at 7:40

1 Answer 1

Reset to default
1
$\begingroup$

You are actually in a somewhat rare instrumental variable situation, and might consider taking advantage of it instead of using propensity scores. With an instrumental variable, you need 4 conditions to identify the causal effect of seeing the notification on your outcome. I'm going to denote Z being sent the notification, A seeing the notification, and Y performing the action within 7 days.

  1. Relevance: Z is associated with A (it is; Z is a direct cause of A)
  2. Exclusion: Z has no effect on Y except through A (the only way the push can affect the outcome is if users see the notification)
  3. Exchangeability: There are no common causes of Z and Y (true; Z is randomly assigned)
  4. Monotonicity: There are no "defiers", i.e., people who always take the opposite treatment they are assigned (there can't be definers because people who don't receive the notification can't choose to see it).

Under these assumptions, you can identify the "complier average treatment effect" (CATE), where "compliance" is seeing the notification when it is pushed and not seeing it when it is not pushed. So, the CATE which is the causal effect of receiving and seeing the notification relative to not receiving and not seeing it.

This can be identified as $\frac{E[Y|Z = 1]−E[Y|Z = 0]}{E[A|Z = 1]−E[A|Z = 0]}$ but since $E[A|Z = 0] =0$ you can compute it simply as $\frac{E[Y|Z = 1]−E[Y|Z = 0]}{E[A|Z = 1]}$

Skip propensity scores when you have randomization!

Improve this answer
$\endgroup$
9
  • $\begingroup$ This is very interesting. I'm unfamiliar with this but I'm going to keep thinking about it / looking at some sources. I just want to confirm: when you write E, you're computing expectancies. So for instance, E(Y|Z=1) is equal to the number of people in Z=1 who had Y=1 over the total number of people in Z=1, yes? So an expected probability. Similarly, E(A|Z=1) will be an expected probability. I'm trying to wrap my head around this, but in the meantime, could you tell me if there's any difference in the two (less efficient) methods I described? $\endgroup$
    – Konrad
    May 30, 2017 at 20:06
  • $\begingroup$ Also, I assume I must interpret the result as a difference, not a ratio: I could say that the difference between the probability of doing the action and that of treated compliers is the CATE. But the CATE shouldn't be interpreted as a ratio / multiplier. Am I correct? $\endgroup$
    – Konrad
    May 30, 2017 at 20:11
  • $\begingroup$ I would have loved if you had answered these questions, but given that your reply has been very helpful to me and that you've disappeared, I'll mark this as answered. $\endgroup$
    – Konrad
    Jun 5, 2017 at 9:38
  • 1
    $\begingroup$ Sorry for disappearing! By E I do mean expectation, so E[A|Z=1] = P(A=1|Z=1). Regarding the methods you proposed, if you truly have enough variables in your data set to eliminate confounding, then you can combine the NS/not-seen group and the NNS group to serve as a single large control group. Remember that the point of PS is to achieve sample balance on the confounders, so as long as you do that your method should be fine. I would actually include the NNS group in the PS estimation sample. $\endgroup$
    – Noah
    Jun 7, 2017 at 3:14
  • 1
    $\begingroup$ To clarify, you want two groups, one of which saw the notification and one of which did not, and you want their covariates to be balanced. The fact that a portion of your group could not have seen the notifications (NNS group) is irrelevant because they are not systematically different from the NS group thanks to randomization. So you can safely combine them into your control group. Your PS model will obviously not be a good selection model because you know that for a portion of your group, selection was forced (the NNS group). But a PS model isn't a selection model. $\endgroup$
    – Noah
    Jun 7, 2017 at 3:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.