# What test should I use to determine if a policy change had a statistically significant impact on website registrations?

A client's website was operating under a certain policy for membership sign ups for over a year. At the start of October 2012 the client implemented a new policy for sign ups that was supposed to encourage more registrations. This policy was applied site wide so there is no control group to compare against (except for the before period).

I have access to the entire history of sign ups and I am wondering if there is a test that I could use to determine if there is a statistically significant difference in the number of registrations before and since the policy change? Could I do something as simple as average sign ups per day before and after the policy change and do a t-test?

-

You are describing "intervention analysis" or "interrupted time series". It refers to estimating how much an intervention has changed a time series. (Intervention-analysis is even one of the tags here, so I am proposing an edit to add it to your question.)

Among other ways, it can be done using an autoregressive integrated moving average (ARIMA) model. ARIMA should be done on a stationary time series but you can estimate a seasonal component and control for it if necessary. And NickAdams is right that you don't want to use raw numbers but rather use proportion of visitors who sign up.

-
It's nice having you around, @mb3041023. It sounds like you know a lot about this (I'm afraid I really don't). If you think it's appropriate, you can go ahead and add the tag to the question. –  gung Nov 13 '12 at 2:14
I don't think I have the tag-changing superpower. –  MattBagg Nov 13 '12 at 2:19
Thanks, but I'm not the OP ;-). You should be able to suggest an edit to the post, & then higher-rep users can approve it for you. –  gung Nov 13 '12 at 2:22
Editing my comment to reflect that. :-) Thanks for the welcome and heads-up! –  MattBagg Nov 13 '12 at 2:24
@mb3041023 Googling "intervention analysis" and "interrupted time series" has provided me with a wealth of new information and I am now able to attack my problem with confidence. I found a 2010 paper by Lagarde titled "How to do (or not to do)...Assessing the impact of a policy change with routine longitudinal data" to be particularly helpful. Thank you. –  Joseph Ryan Glover Nov 13 '12 at 3:01

Yes, you can simply do a t-test, although you may very well have confounding variables that will affect how you want to go about this, and perhaps you may want to use an ANOVA with blocks.

One confounding variable that you may want to watch out for is effects over time. Does the site have more sign-ups in certain parts of the year over others? Have there been more sign-ups this year than in other years? You may also want to control for traffic: is there more traffic now than in the past? Perhaps now, more people are seeing the sign-up sheet than before.

A better metric may be (sign-ups)/(site visitor), and you could find this out with some preliminary ANOVA tables.

-
There is some seasonality in the registration history. I have considered constructing a time series model by fitting the pre-policy daily registrations and then using the model to calculate the residual for both pre and post policy sign ups. I thought performing the t-test on the average daily residuals might remove some of the confounding variables. Does this seem reasonable? –  Joseph Ryan Glover Nov 13 '12 at 1:53
By residual are you referring to (actual values after treatment applied) - (time series model prediction)? In that case, it seems reasonable to me. mb3041023 is right about wanting to use something like ARIMA: arima() and predict.arima() in R could be suited to that. –  Nick Adams Nov 13 '12 at 2:59
Yes, that is what I meant by residual. The Lagarde paper I mention in my comment to mb3041023 details using an ARIMA model as you both have recommended. Thank you for the R suggestion. –  Joseph Ryan Glover Nov 13 '12 at 3:05