6
$\begingroup$

A client's website was operating under a certain policy for membership sign ups for over a year. At the start of October 2012 the client implemented a new policy for sign ups that was supposed to encourage more registrations. This policy was applied site wide so there is no control group to compare against (except for the before period).

I have access to the entire history of sign ups and I am wondering if there is a test that I could use to determine if there is a statistically significant difference in the number of registrations before and since the policy change? Could I do something as simple as average sign ups per day before and after the policy change and do a t-test?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

You are describing "intervention analysis" or "interrupted time series". It refers to estimating how much an intervention has changed a time series. (Intervention-analysis is even one of the tags here, so I am proposing an edit to add it to your question.)

Among other ways, it can be done using an autoregressive integrated moving average (ARIMA) model. ARIMA should be done on a stationary time series but you can estimate a seasonal component and control for it if necessary. And NickAdams is right that you don't want to use raw numbers but rather use proportion of visitors who sign up.

Improve this answer
$\endgroup$
6
  • $\begingroup$ It's nice having you around, @mb3041023. It sounds like you know a lot about this (I'm afraid I really don't). If you think it's appropriate, you can go ahead and add the tag to the question. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 13, 2012 at 2:14
  • $\begingroup$ I don't think I have the tag-changing superpower. $\endgroup$
    – MattBagg
    Commented Nov 13, 2012 at 2:19
  • 1
    $\begingroup$ Thanks, but I'm not the OP ;-). You should be able to suggest an edit to the post, & then higher-rep users can approve it for you. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 13, 2012 at 2:22
  • 1
    $\begingroup$ @mb3041023 Googling "intervention analysis" and "interrupted time series" has provided me with a wealth of new information and I am now able to attack my problem with confidence. I found a 2010 paper by Lagarde titled "How to do (or not to do)...Assessing the impact of a policy change with routine longitudinal data" to be particularly helpful. Thank you. $\endgroup$
    – No Name Required
    Commented Nov 13, 2012 at 3:01
  • 1
    $\begingroup$ No prob :-) For others, that reference is: Lagarde, Mylene. "How to do (or not to do)… Assessing the impact of a policy change with routine longitudinal data." Health policy and planning 27.1 (2012): 76-83. heapol.oxfordjournals.org/content/27/1/76.full.pdf+html $\endgroup$
    – MattBagg
    Commented Nov 13, 2012 at 14:10
1
$\begingroup$

Yes, you can simply do a t-test, although you may very well have confounding variables that will affect how you want to go about this, and perhaps you may want to use an ANOVA with blocks.

One confounding variable that you may want to watch out for is effects over time. Does the site have more sign-ups in certain parts of the year over others? Have there been more sign-ups this year than in other years? You may also want to control for traffic: is there more traffic now than in the past? Perhaps now, more people are seeing the sign-up sheet than before.

A better metric may be (sign-ups)/(site visitor), and you could find this out with some preliminary ANOVA tables.

Improve this answer
$\endgroup$
3
  • $\begingroup$ There is some seasonality in the registration history. I have considered constructing a time series model by fitting the pre-policy daily registrations and then using the model to calculate the residual for both pre and post policy sign ups. I thought performing the t-test on the average daily residuals might remove some of the confounding variables. Does this seem reasonable? $\endgroup$
    – No Name Required
    Commented Nov 13, 2012 at 1:53
  • 1
    $\begingroup$ By residual are you referring to (actual values after treatment applied) - (time series model prediction)? In that case, it seems reasonable to me. mb3041023 is right about wanting to use something like ARIMA: arima() and predict.arima() in R could be suited to that. $\endgroup$
    – Nick Adams
    Commented Nov 13, 2012 at 2:59
  • $\begingroup$ Yes, that is what I meant by residual. The Lagarde paper I mention in my comment to mb3041023 details using an ARIMA model as you both have recommended. Thank you for the R suggestion. $\endgroup$
    – No Name Required
    Commented Nov 13, 2012 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.