Statistical validity of before and after tests I'm working at a company developing web applications and due to the load on our testing team that can execute and analyze A/B or multi-variate tests, we're being asked to run "before and after" tests--also called "launch and learn" tests.
With these before and after tests, if we make a change to a web page, we're being asked to measure a key performance indicator (KPI) for a week before the change and then a week after the change.  However, in many cases, the traffic to that page is lumpy and not necessarily consistent.
As an example, a link on one part of a web page may be served up 1000 times in a 7 day period and clicked on 500 times during that week, but during the 2nd week (8 days, not 7)--once a change has been made and the link has been moved to another part of the page--it might be served up 1200 times and clicked on 550 times.
My gut tells me that this is not an "apples to apples" comparison, and so comparing 500 clicks (A, control) to 550 click (B) would not be valid.  Can someone please validate or let me know if I'm wrong?
 A: This is quite invalid, but there are ways to deal with it.
First, you can just graph clicks over time. This way, you can visually account for the secular trends that you discuss. Alternatively but equivalently, you could fit some complex model that accounts for these secular trends.
Second, you can create a control group. While you're measuring performance on the web page that you are changing (Call this "Peanut Butter".), measure the performance on a comparable web page (Call this "Jelly".) at the same time. Then check whether the change in the performance of Peanut Butter is different from the change in performance of Jelly.
And it's even better if you use both of these approaches together.
A: While I'm not 100% certain of the statistical validity, many testers do this sort of thing anyways. Problem is that external factors like promotions, etc. One way to mitigate against a false result is to repeat the test 2 or 3 times. If there is a true difference between the versions, the results should be repeatable.
