My company just completed a major website redesign and launched it a month ago. We're seeing some changes in daily visitors and new signups from the original website (comparing month over month), and we're wondering if there's a statistically significant change in traffic and conversions. I've run some t-tests and chi-squared tests over the average daily visitors and conversions, and they point to insignificant changes. Even if I can find statistical measures that show significant changes in our traffic, I'm skeptical of the any of those findings since there are a number of confounding variables (seasonality, holidays, external marketing campaigns) at play. Outside of going back in time and performing an A/B Test with the new site and the old site, are there any methods I can use to understand whether or not the traffic after the redesign is different from the original?
1 Answer
$\begingroup$
$\endgroup$
You've mentioned simple t-test and chi-squared on absolute number of visitors already. If these point to insignificance, it already sounds like you're a long way toward an answer (which would be that redesign did not have any significant effect).
If your history is long enough, you can try to decompose of seasonality, and then redo the t-test with seasonally adjusted figures.
