We recently released a new redesigned e-commerce website for a client. They are claiming to be seeing a 40% drop in revenue due to the changes. We have daily sales data going back a few years. We are trying to determine if a recent drop in revenue after a site relaunch can be attributed to the redesign or just due to normal variance.
What tools/methods can we use to either prove or disprove that the most recent decrease in revenue is statistically significant or not?
Thanks in advance.
You could do a simple t test of the means of the two periods. Be sure that there is not some kind of upward or downward trend in the data over the two years, because this can lead to the wrong conclusions of the t test value.
you may need to use Welch's t test(R uses this as standard) which is a modification of the standard Student's t test to take into account different variance and sample sizes. It should be a one sided alternative hypothesis since you are testing of whether with period_after_change is less than period_before_change.
This is the simplest test of whether the change has had a negative or no effect.
You can calculate the t test score in excel using Welch's t test (googles first hit) if the test is significant then you have done something wrong and the drop can be explained by the relaunch(if we only look at that as an explanatory variable). if it is not significant then you have done nothing wrong.
this is the statistically method, another side of this is to explain it to the client... I won't dive in to this here ... ;)
Nothing will be able to prove it one way or another, because even if you find revenue has dropped from that time, you will not be able to dismiss the possibility of other structural change (eg new competitor, changed regulatory environment, changed fashion, something you can't even think of...).
You can use time series techniques to identify if the timing of the changed system is associated with a decline in revenue; or, better for your aims, you might be able to dismiss that claim (and if there's no obvious decline, there's nothing to explain, right? well, maybe... the problem is constructing a counterfactual.).
Problems you will have to deal with will include; seasonality (both micro eg weekly and macro eg summer v winter); growth or other trends; and serial correlation of your observations.
In the plot below from simulated data you can see one way of going about this.
You can fit a model of some sort based on there being no change over time - in this case, I've fit a linear model with the response variable on a logarithmic scale, which is equivalent to saying the daily revenue is growing at a constant rate. This null hypothesis is shown with the black line.
The red and blue lines, on the other hand, show an alternative, more complex model, which allows both a change in the level of revenue and a change in the growth rate (in this case to negative) as a result of the introduction of the new system. Once you've fit this more complex model as well, you can then test for statistically significant evidence that this model is needed rather than the simple black line model.
(Note that in this case, if you did a simple t test comparing before and after situations as @pgericson suggests, you'd conclude you'd significantly increased revenue with the new system, because of the underlying growth rate in the months leading up to the new system.)
Now, the danger to look out for, is you can't just fit a model in the way you would with cross sectional data. You need to allow for the fact that a revenue observation one day does not add that much to the revenue observation the day before - they are probably highly correlated and not completely new information. Any stats or econometric package worth its salt will allow this; in R you can use either gls() in the nlme library or arima() to do it.
My R code that simulated this data and did some basic analysis on it is pasted below.
#simulate data
set.seed(80)
x <- ts(100*exp(1:1000*0.001), frequency=7)
e <- rnorm(1002,0,10)
x <- x+ 0.5*e[1:1000] + 0.8*e[2:1001] + e[3:1002]
changed <- rep(c(0,1), c(800,200))
x <- x + cumsum(changed)^0.4 * rnorm(1000, -8,1)
# check it looks ok
par(mfrow=c(2,1))
plot(x, main="Daily revenue ($'000)", xlab="weeks", ylab="(original)")
abline(v=801/7, col="grey50")
plot(x, ylab="(logarithmic)", xlab="weeks", log="y")
abline(v=801/7, col="grey50")
# t test makes it look like you've increased revenue! -
# because it ignores the trend
t.test(x~changed)
# Much better is to illustrate in some kind of model
# that can take into account any growth trend.
# With real data this will be quite complex, but
# in my simulated data the growth is nice and regular
# so it is easy to see if it is disrupted.
x.df <- data.frame(x=x, changed=changed, day=1:1000)
win.graph()
x.lm1 <- lm(log(x)~day, data=x.df)
plot(x, ylab="(logarithmic)", xlab="weeks", log="y", bty="l")
abline(v=801/7, col="grey50")
lines(1:1000/7, exp(predict(x.lm1)), lwd=3)
x.lm2 <- lm(log(x)~day*changed, data=x.df)
lines(1:800/7, exp(predict(x.lm2))[1:800], col="red", lwd=3)
lines(801:1000/7, exp(predict(x.lm2))[801:1000], col="blue", lwd=3)
anova(x.lm2) # shows "changed" is significant
summary(x.lm2) # could be used to estimate how much change has happened
# The problem with this approach though is that
# the errors are serially correlated and hence the inferences
# based on them being iid will not be justifiable. As shown by this diagnostic
# plot:
acf(residuals(x.lm2))
library(nlme)
x.lm3 <- gls(log(x)~day*changed, data=x.df, correlation=corAR1())
anova(x.lm3) # "changed" is still significant but much higher p values
summary(x.lm3)
# I'd like to fit a model with more lags in the autoregression structure
# but the following code takes frigging ages for some reason (eventually came out OK)
x.lm4 <- update(x.lm3, correlation=corARMA(c(0.6, 0.2), p=2, q=0),
control=glsControl(msVerbose=TRUE))
You might consider using either CausalImpact in R to estimate the difference, but CausalImpact wants a secondary dataset (control) to compare the "treated" data - in this case website data in question.
CausalImpact requires a secondary dataset to model against the exploratory data, something like an separate but very similar market. You could also provide a regression model as your comparative dataset. A better explanation is here: https://google.github.io/CausalImpact/
You might also consider using ITS, Interrupted Time Series analysis. This would entail building a strong time series model that includes variables x1) change has/has not occurred and x2) day 1,2,3... of the new website changes.
My recommendation would be to use ITS if you do not have a parallel dataset or model to plug into CausalImpact. Using ITS would require the following:
Build a strong time-series model of the "pre-period", hopefully something that accounts for any and all seasonality and cyclical trends leading into the treatment or post-period. You will want a model that you feel would have accurately predicted the "counter-factual," or what would have happened had no changes to the website been made. This is model 1.
Add the two variables I mentioned above to your model to signify the start and trend of the "test". Re-run your model to include the test days and new test variables, and see if your test shows it is statistically significant or not (very low p-values). This is model 2.
Plot the resulting prediction points for these two models into Excel - all the way through the test. The difference between the two models during the test days is your result, as long as the two new "test variable" coefficients from the previous step are statistically significant. They will also begin to show your the magnitude of the change if there is one.
Of course, you have to know for certain that there wasn't some other event that happened at the exact same time as the new event - correlation is not causation, yada yada.