7
$\begingroup$

We recently released a new redesigned e-commerce website for a client. They are claiming to be seeing a 40% drop in revenue due to the changes. We have daily sales data going back a few years. We are trying to determine if a recent drop in revenue after a site relaunch can be attributed to the redesign or just due to normal variance.

What tools/methods can we use to either prove or disprove that the most recent decrease in revenue is statistically significant or not?

Thanks in advance.

$\endgroup$
4
  • 1
    $\begingroup$ Can you be a little more specific in terms of data you have? $\endgroup$
    – Ken
    Apr 27, 2012 at 15:27
  • $\begingroup$ We have their revenue numbers in dollars of products sold for about the last two years on a daily basis. We are trying to determine if a recent drop in revenue after a site relaunch can be attributed to the redesigned web site functionality or just a normal variance. $\endgroup$
    – jermeyz
    Apr 27, 2012 at 15:36
  • $\begingroup$ f you actually post your data I will try and help as I have had extensive experience in modelling daily data taking into account day-of-the-week , month of the year, holiday effects , particular days of the month effects , level shift effects and local time trends. What needs to be done is to empirically detect a substantive change which could be assignable to a number of possible "causes". $\endgroup$
    – IrishStat
    Apr 27, 2012 at 19:58
  • 1
    $\begingroup$ I may be wrong, but did you try simple correlation to see if it even makes sense? Correlating sales with new design for that time period vs that of the old one? Intuitively this makes sense for a cursory analysis before diving into details, but I'll leave it up to the experts to comment on this $\endgroup$
    – PhD
    Apr 27, 2012 at 20:02

3 Answers 3

3
$\begingroup$

You could do a simple t test of the means of the two periods. Be sure that there is not some kind of upward or downward trend in the data over the two years, because this can lead to the wrong conclusions of the t test value.

you may need to use Welch's t test(R uses this as standard) which is a modification of the standard Student's t test to take into account different variance and sample sizes. It should be a one sided alternative hypothesis since you are testing of whether with period_after_change is less than period_before_change.

This is the simplest test of whether the change has had a negative or no effect.

You can calculate the t test score in excel using Welch's t test (googles first hit) if the test is significant then you have done something wrong and the drop can be explained by the relaunch(if we only look at that as an explanatory variable). if it is not significant then you have done nothing wrong.

this is the statistically method, another side of this is to explain it to the client... I won't dive in to this here ... ;)

$\endgroup$
2
  • 2
    $\begingroup$ Won't there be a problem because the t test assumes the observations are iid, whereas the data in the OP's post will probably be autocorrelated? $\endgroup$ Apr 27, 2012 at 22:45
  • $\begingroup$ You are right about the autocorrelation, there is probably some kind of random walk with drift. This is a quick and practical test by looking at e.g. 1 month before the change of daily revenue data and 1 month after the change to see the levels of whether the drop was statistically significant. It was not meant as a perfect academic solution. $\endgroup$
    – pgericson
    Apr 28, 2012 at 0:06
4
$\begingroup$

Nothing will be able to prove it one way or another, because even if you find revenue has dropped from that time, you will not be able to dismiss the possibility of other structural change (eg new competitor, changed regulatory environment, changed fashion, something you can't even think of...).

You can use time series techniques to identify if the timing of the changed system is associated with a decline in revenue; or, better for your aims, you might be able to dismiss that claim (and if there's no obvious decline, there's nothing to explain, right? well, maybe... the problem is constructing a counterfactual.).

Problems you will have to deal with will include; seasonality (both micro eg weekly and macro eg summer v winter); growth or other trends; and serial correlation of your observations.

In the plot below from simulated data you can see one way of going about this.

You can fit a model of some sort based on there being no change over time - in this case, I've fit a linear model with the response variable on a logarithmic scale, which is equivalent to saying the daily revenue is growing at a constant rate. This null hypothesis is shown with the black line.

The red and blue lines, on the other hand, show an alternative, more complex model, which allows both a change in the level of revenue and a change in the growth rate (in this case to negative) as a result of the introduction of the new system. Once you've fit this more complex model as well, you can then test for statistically significant evidence that this model is needed rather than the simple black line model.

enter image description here

(Note that in this case, if you did a simple t test comparing before and after situations as @pgericson suggests, you'd conclude you'd significantly increased revenue with the new system, because of the underlying growth rate in the months leading up to the new system.)

Now, the danger to look out for, is you can't just fit a model in the way you would with cross sectional data. You need to allow for the fact that a revenue observation one day does not add that much to the revenue observation the day before - they are probably highly correlated and not completely new information. Any stats or econometric package worth its salt will allow this; in R you can use either gls() in the nlme library or arima() to do it.

My R code that simulated this data and did some basic analysis on it is pasted below.

#simulate data
set.seed(80)
x <- ts(100*exp(1:1000*0.001), frequency=7)
e <- rnorm(1002,0,10)
x <- x+ 0.5*e[1:1000] + 0.8*e[2:1001] + e[3:1002]
changed <- rep(c(0,1), c(800,200))
x <- x + cumsum(changed)^0.4 * rnorm(1000, -8,1)

# check it looks ok
par(mfrow=c(2,1))
plot(x, main="Daily revenue ($'000)", xlab="weeks", ylab="(original)")
abline(v=801/7, col="grey50")
plot(x, ylab="(logarithmic)", xlab="weeks", log="y")
abline(v=801/7, col="grey50")




# t test makes it look like you've increased revenue! -
# because it ignores the trend
t.test(x~changed)

# Much better is to illustrate in some kind of model
# that can take into account any growth trend.
# With real data this will be quite complex, but
# in my simulated data the growth is nice and regular
# so it is easy to see if it is disrupted.

x.df <- data.frame(x=x, changed=changed, day=1:1000)
win.graph()
x.lm1 <- lm(log(x)~day, data=x.df)
plot(x, ylab="(logarithmic)", xlab="weeks", log="y", bty="l")
abline(v=801/7, col="grey50")
lines(1:1000/7, exp(predict(x.lm1)), lwd=3)

x.lm2 <- lm(log(x)~day*changed, data=x.df)
lines(1:800/7, exp(predict(x.lm2))[1:800], col="red", lwd=3)
lines(801:1000/7, exp(predict(x.lm2))[801:1000], col="blue", lwd=3)
anova(x.lm2) # shows "changed" is significant
summary(x.lm2) # could be used to estimate how much change has happened


# The problem with this approach though is that 
# the errors are serially correlated and hence the inferences
# based on them being iid will not be justifiable.  As shown by this diagnostic
# plot:
acf(residuals(x.lm2))


library(nlme)
x.lm3 <- gls(log(x)~day*changed, data=x.df, correlation=corAR1())
anova(x.lm3) # "changed" is still significant but much higher p values
summary(x.lm3)

# I'd like to fit a model with more lags in the autoregression structure
# but the following code takes frigging ages for some reason (eventually came out OK)
x.lm4 <- update(x.lm3, correlation=corARMA(c(0.6, 0.2), p=2, q=0), 
    control=glsControl(msVerbose=TRUE))
$\endgroup$
1
  • $\begingroup$ The problem is that one doesn't know a priori howto construct the null hypothesis.You created a simulated example where you knew the state of nature.In practice one needs to form a model based upon the data to empirically detect the unspecified effect via model diagnostics.Intervention Detection would suggest the slope change and intercept change in your data.Alternative data might suggest "parameter changes" and/or error variance changes. This is why I asked the OP to actually post his data so that objective analytics could be used to construct the hypothesis which was most rejectable. $\endgroup$
    – IrishStat
    Apr 28, 2012 at 13:53
1
$\begingroup$

You might consider using either CausalImpact in R to estimate the difference, but CausalImpact wants a secondary dataset (control) to compare the "treated" data - in this case website data in question.

CausalImpact requires a secondary dataset to model against the exploratory data, something like an separate but very similar market. You could also provide a regression model as your comparative dataset. A better explanation is here: https://google.github.io/CausalImpact/

You might also consider using ITS, Interrupted Time Series analysis. This would entail building a strong time series model that includes variables x1) change has/has not occurred and x2) day 1,2,3... of the new website changes.

My recommendation would be to use ITS if you do not have a parallel dataset or model to plug into CausalImpact. Using ITS would require the following:

  1. Build a strong time-series model of the "pre-period", hopefully something that accounts for any and all seasonality and cyclical trends leading into the treatment or post-period. You will want a model that you feel would have accurately predicted the "counter-factual," or what would have happened had no changes to the website been made. This is model 1.

  2. Add the two variables I mentioned above to your model to signify the start and trend of the "test". Re-run your model to include the test days and new test variables, and see if your test shows it is statistically significant or not (very low p-values). This is model 2.

  3. Plot the resulting prediction points for these two models into Excel - all the way through the test. The difference between the two models during the test days is your result, as long as the two new "test variable" coefficients from the previous step are statistically significant. They will also begin to show your the magnitude of the change if there is one.

Of course, you have to know for certain that there wasn't some other event that happened at the exact same time as the new event - correlation is not causation, yada yada.

$\endgroup$
3
  • $\begingroup$ CausalImpact is a package; your answer would be improved by explaining (or at least mentioning) the methods behind the specific implementation. It would also benefit from an explanation of why these methods are a good fit for the question (as you partially do in the case of ITS). $\endgroup$
    – mkt
    Dec 22, 2017 at 16:45
  • $\begingroup$ Thanks for the feedback, mkt. CausalImpact requires a secondary dataset to model against the exploratory data, something like an separate but very similar market. $\endgroup$ Dec 22, 2017 at 17:50
  • $\begingroup$ (+1) Thanks for your expanded contribution. $\endgroup$
    – mkt
    Dec 22, 2017 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.