1
$\begingroup$

Suppose we do the following study. We have 2 web-servers (server A and server B) which are processing user requests. Both run on the same hardware, but one of them (server B) uses a slightly different algorithm for request processing. When a user request comes in we have 50/50 chance of processing it on each server. So the work being done by each server is the same in long term.

We have 1 hour system-log containing information about all the requests in this hour (all the population, not sample): what server processed the request and how long it took to process.

For example:

21:00:00 SERVER-A 0.11s
21:00:00 SERVER-A 0.13s
21:00:00 SERVER-B 0.09s
21:00:00 SERVER-A 0.16s
21:00:00 SERVER-B 0.11s
21:00:01 SERVER-A 0.14s

The log is quite large (~250K requests from each server, ~500K in total).

It turns out, that server B is slightly faster (mean response time is smaller). The question is which test should I use to show statistical significance in difference of response times of those two servers?

$\endgroup$
1
  • 1
    $\begingroup$ If, as you say, you have the population rather than a sample, then it is arguable whether you can or should do any significance tests at all (this has been discussed here quite a bit). $\endgroup$ – Peter Flom Jan 24 '13 at 11:18
2
$\begingroup$

If, as you say, you have the population rather than a sample, then it is arguable whether you can or should do any significance tests at all (this has been discussed here quite a bit).

But unless the algorithms or the servers are going to be changed in some way after that 1 hour, I'd say you have a sample, not a population. You could, then, use a t-test unless the distributions are very non-normal; or you could use a non-parametric test of two means.

$\endgroup$
2
  • 1
    $\begingroup$ What conditions should be met to imply that Student t-test is giving strong inference? I mean under what circumstances the data is very non-normal? $\endgroup$ – Denis Bazhenov Jan 24 '13 at 11:46
  • $\begingroup$ I would look at quantile normal plots of the distribution of times for each server. If in doubt, I'd run both the t-test and the nonparametric test and see if there are differences. $\endgroup$ – Peter Flom Jan 24 '13 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.