How do I measure the statistical difference between two measures? I am measuring rendering time on Stack Overflow, specifically pre/post changes.
I have data as such
pre_post | RouteName      | count | avg | st_dev | min |  max
---------+----------------+-------+-----+--------+-----+-----
1. pre   | Questions/Show | 24736 |  11 |     15 |   0 | 1004
2. post  | Questions/Show |   842 |  16 |     26 |   1 |  453

From this I would like to understand:


*

*Are the requests statistically slower or faster after the change, or there is no discernible change?

*How many measurements do I need before having reasonable certainty of point 1.?



Here is what the peaks look like (after cleaning, the data is not the same as above!)

Here is a link to the raw data.

Here's what the raw data looked like before cleaning and normalizing

 A: Basic questions are good.
In this application you have lots of data and the opportunity to collect many more.  This makes the calculations so easy you can practically do them in your head using fundamental principles.  I will share the standard progression of thinking that a statistician will automatically go through when presented with data like these.


*

*The standard error (SE) of the "avg" (presumably, arithmetic mean) is the standard deviation divided by the square root of the count.  We may extend the table thus:
pre_post | RouteName      | count | avg | st_dev | min |  max |    SE
---------+----------------+-------+-----+--------+-----+-------------
1. pre   | Questions/Show | 24736 |  11 |     15 |   0 | 1004 | 0.095
2. post  | Questions/Show |   842 |  16 |     26 |   1 |  453 | 0.896


*The standard error of a difference satisfies the Pythagorean Theorem: it is the root of the sum of squares.  Therefore the difference of averages, $5=16-11$, has a standard error of $$\sqrt{15^2/24736 + 26^2/842} \approx \sqrt{0.095^2+0.896^2} = 0.901.$$
The form of the general formula (in terms of two averages, two standard deviations, and two counts) should be apparent from this example.

*Differences should be measured as multiples of their standard error, a so-called "Z score."  A "change" is a comparison to $0$, whence the apparent amount of change (on average) is 
$$\frac{(16-11) - 0}{\sqrt{15^2/24736 + 26^2/842}} \approx 5.55.$$

*$5.55$ standard errors is big.  For instance, if rendering times had Normal distributions (they do not), this would give us about $99.99999856$% confidence that the difference is real and not just the result of chance variations in the observations.  (Rounding in the table suggests a more precisely computed Z-score would be somewhere in the $4.5$ to $6.5$ range, but these are large Z-scores regardless and correspond to ridiculously high computed confidences.)

*However, these data are not Normally distributed.  One definite sign of this is that their standard deviations are larger than their averages.  Usually we have to work harder and ask additional questions related to details of the data distribution.

*Nevertheless, because there are so many pre values and a reasonably large number of post values, it's a fair bet that a change has occurred.  (It would take a lengthy argument to justify this and to put some quantitative bounds on "fair bet," so let's just say this is experience talking.)  A good estimate of the amount of change is the difference $16-11=5$.

Please don't resort to "power calculations" or anything like that: they could be deceptive due to the (strong) non-Normality of the data.  If you need more detailed insights, then follow up by providing additional information about the data.  Histograms of the pre and post values would go a long way towards any further analysis.

Update
There is no significant difference in average times based on the data that were posted, as "cleaned" to include only durations between 5 and 65 ms: I compute a Z statistic of $0.91$, which is not large at all.  However, there is a difference that is visible in the histograms.  It can be detected with a chi-squared test (which is extremely significant, having a p-value essentially zero: the chi-squared statistic is $323$ with $60$ degrees of freedom).  The nature of the difference is clear in the histograms:

The shorter-duration times at 9 and 10 ms appear to have been lengthened (on average) to 12-20 ms.  A more powerful way to evaluate this change is to plot the standardized chi-squared residuals.  These are values that, like z-statistics, should usually lie between around $-2$ and $2$ and rarely exceed $3$ in size.  For a given duration, a positive residual for the post value means that after the change there were more occurrences than expected (assuming no change had occurred) and a negative residual means there were fewer occurrences of that duration.  Here is the plot, with lines drawn at $\pm 3$ as a reference:

The evident downward trend (from just significantly positive at 12-14 ms to just significantly negative at 57-64 ms) in the residuals for durations greater than 11 ms suggests the renderings that disappeared from the 8-11 ms group have reappeared scattered throughout the 13-65+ ms range.  This suggests, as a working hypothesis, that there was a group of short rendering times (comprising 5 to 10% of the total) that were lengthened after the change but otherwise nothing else seems to have altered.
A: To test for changes pre and post within one group you should carry out a repeated-measures ANOVA (repeated-measures T-test). If you want to know how many measurements you would likely need to obtain 1% (.01) certainty you can calculate this doing a statistical power calculator. 
I hope that helps!
