A have a set of machines running a particular software. I want to do something that's called "canary deployment". This consists of deploying the new version of the software to a subset of machines and monitor error statistics. And then, if the new version doesn't seem to be causing more errors than the old version, progressively increase the number of machines with the new version until all of them are using it. The idea is that if the new version has any problem, it'll be detected before it affects all the users.
So, during this process, I have the time series of the number of events and number of errors for the old and new versions.
My idea is to build a rule that'll allow me to stop this process at any time if I see a significant increase in the number of errors being reported for the new version. It'll check this condition every X seconds.
More explicitly, I was framing the condition to stop as: the average error per event of the new version cannot be bigger than the average for the old version. I could set some thresholds but I'd like something with more theoretical foundation, other than just empirical knowledge or guesses. I'm also open to suggestions on better approaches to this problem.
What I need then is to figure out what type of test I could use to detect if the means are different and the best procedure to be able to test it several times and not incur in any over/underestimation. From what I've been reading, it seems that the Mann-Whitney U test is a good fit. Especially because it is non-parametric and my data is not normally distributed. Does it make sense?
Some of the characteristics of the problem and the data available:
- The idea is to do this check every X seconds, so I imagine I'll have to do some correction for multiple comparisons
- The same machine will be used for both versions during this rollout process, so I imagine there's some dependency among the instances
- I don't have data on an event level, only aggregated per second
- A false negative is more problematic than a false positive (saying that a new version is not worse when in fact it is)
Well, the question turned out quite big, sorry about that. Thanks in advance to those that got this far ;)