I'd like to test if requests to my system are succeeding at least 90% of the time. This system receives about 10 requests/minute. I'm ok getting a 1-in-20 false alarm.

If my question ended here, I guess the answer would be "do a t-test with CI=95%".

The thing is grouping these requests together will miss issues because the caller influences the probability of success (long story short, my system can fail because the caller is using an outdated firmware, or a broken client, etc.).

So my idea is to break the results by the dimensions (for instance model and client version). If I do that, I get at least n=5 from 30 statistics or so. For instance:

| model | version | n  |   p  | 
|   A   |    2    |  5 | 0.20 |
|   B   |    1    | 20 | 0.90 |
|   C   |    1    | 10 | 1.00 |
|  ...  |   ...   | .. | ...  |

But now, if I have 30 of these and I still use CI 95%, I will get more than 1-in-20 false alarms.

So it seems obvious that I need to calculate the desired CI like:

CI^30 = 0.95 -> CI = 99.83%

But.. is it? Using a CI=99.83% feels like this alarm will only fire on extreme situations.

  • $\begingroup$ What do you mean by "grouping these requests together will miss issues?" Can you explain a bit more about the testing process you originally thought about setting up but think you couldn't because the grouping will supposedly create problems? $\endgroup$ – StatsStudent Jan 9 at 4:35
  • 1
    $\begingroup$ You either calculate the p-values for each test (30 univariate tests total) and then adjust the p-values accordingly (for ex. Benjamini-Hochberg method), this will give you corrected p-values for each estimate. If you do not want to deal with p-values and only with confidence intervals, then you need to adjust the standard errors of your estimates (for ex. using a multivariate t-distribution). $\endgroup$ – user2974951 Jan 9 at 8:23
  • $\begingroup$ @StatsStudent, in the mini-example I gave above on the table, if you average everything (which is what essentially happens when you don't break up by model/version) you get n=35, p=0.83. But if you check just the first line (model A version 2) after breaking up by model/version, you will probably be able to infer there's an issue with that model/version. $\endgroup$ – Lem0n Jan 9 at 10:57
  • $\begingroup$ Lem0n, I don't understand why you think it's a bad idea the data into groups by model and version. Are you, in fact, interested in detecting changes between model/versions? It seemed to me like you were only breaking your data out because of some other concern that I didn't quite understand from your explanation. I'm trying to understand that. Also, there are usually more powerful methods than having to resort to p-value adjustments as suggested by @user2974951. $\endgroup$ – StatsStudent Jan 9 at 12:29
  • $\begingroup$ I don't think it's a bad idea to break the data into groups by model and version, on the contrary. My concern is that doing that will increase the false-positive rates. It looks like using Benjamini-Hochberg as @user2974951 proposed might solve that. $\endgroup$ – Lem0n Jan 9 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.