# Statistical test for computer-systems performance analysis?

Suppose we do the following study. We have 2 web-servers (server A and server B) which are processing user requests. Both run on the same hardware, but one of them (server B) uses a slightly different algorithm for request processing. When a user request comes in we have 50/50 chance of processing it on each server. So the work being done by each server is the same in long term.

We have 1 hour system-log containing information about all the requests in this hour (all the population, not sample): what server processed the request and how long it took to process.

For example:

21:00:00 SERVER-A 0.11s
21:00:00 SERVER-A 0.13s
21:00:00 SERVER-B 0.09s
21:00:00 SERVER-A 0.16s
21:00:00 SERVER-B 0.11s
21:00:01 SERVER-A 0.14s


The log is quite large (~250K requests from each server, ~500K in total).

It turns out, that server B is slightly faster (mean response time is smaller). The question is which test should I use to show statistical significance in difference of response times of those two servers?

• If, as you say, you have the population rather than a sample, then it is arguable whether you can or should do any significance tests at all (this has been discussed here quite a bit). – Peter Flom Jan 24 '13 at 11:18

## 1 Answer

If, as you say, you have the population rather than a sample, then it is arguable whether you can or should do any significance tests at all (this has been discussed here quite a bit).

But unless the algorithms or the servers are going to be changed in some way after that 1 hour, I'd say you have a sample, not a population. You could, then, use a t-test unless the distributions are very non-normal; or you could use a non-parametric test of two means.

• What conditions should be met to imply that Student t-test is giving strong inference? I mean under what circumstances the data is very non-normal? – Denis Bazhenov Jan 24 '13 at 11:46
• I would look at quantile normal plots of the distribution of times for each server. If in doubt, I'd run both the t-test and the nonparametric test and see if there are differences. – Peter Flom Jan 24 '13 at 12:16