Context: website A/B testing
When calculating P-value and confidence intervals for each variation in an experiment with more than 2 variations (control, variation 1, variation 2 ...) should there be some sort of correction that takes into account the existence of other variations or is it OK to calculate these metrics in isolation (variation 1 vs control, variation 2 vs control etc.) - Z-test and chi-squared
The most straightforward answer is that you should adjust the p-values based on the number of comparisons you're making in the experiment. So, if you're comparing control vs v1, control vs v2, and v1 vs v2, then your p-value should be corrected for the fact that you're looking at the same data 3 different ways. The most straightforward way of doing this is to just divide the critical p-value (typically .05) by the number of comparisons you're making. For 3 comparisons, that would give you a new critical p-value of .0167 (.05/3). This procedure is called a bonferroni correction. There are a whole menagerie of other ways of doing these corrections. See this wikipedia entry for some background
The more complicated answer is that the strict interpretation of a p-value depends on a whole bunch of other things besides the number of comparisons you're making. In practice, people usually just correct for how many comparsions they're making, but under the logic of null-hypothesis testing, you should also correct for any other comparisons from other experiments that you've run on the same idea, or perhaps, related ideas, or perhaps even based on experiments that you should have run, or could have run, or thought about running, but didn't. See the first four and a half-ish pages of this paper: http://www.indiana.edu/~kruschke/articles/Kruschke2010WIRES.pdf
