Calculating an SLA and considering statistical confidence When calculating a Service Level Agreement (SLA) one might consider the %age of successful transactions with the system. E.g. if 99 out of 100 transactions fail, the 99.9% SLA was not met.
However, this introduces a lot of 'noise' where some clients only have two transactions and one can fail - meaning only a 50% SLA was achieved.
How could we factor in statistical methods here to create a more concrete rule. For example, something like: 'where 99.9% of transactions succeed with a statistical confidence of 80%'.
Forgive the poor nomenclature but the idea would be that there would need to be statistically significant number of failures, with 80% certainty that the service was below 99.9%. 
Be great to hear some approaches on how we might model this statistically.
 A: $\DeclareMathOperator{\Beta}{Beta}$How about some sort of Bayesian analysis of the probability of failure?
The first step is to pick your prior distribution for the probability of success $s$. That is, before you see the results of any transactions, how probable do you think it is that $s$ is equal to a given value? Because we're looking at the results of independent trials, we should probably use a prior from the Beta family, because it makes it easy to update your beliefs. (If your prior for $f$ is a $\Beta(a, b)$ and you observe that a request succeeds or fails, your posterior belief about $f$ after you update on this evidence is a $\Beta(a+1, b)$ or $\Beta(a, b+1)$ distribution, respectively.) I'll talk more about picking the prior later, but suppose for now that it's a $Beta(a, b)$ for some $a$ and $b$.
Suppose your client makes $N$ requests and observes $F$ failures. Then your posterior distribution for $s$ is $\Beta(a+F, b+N-F)$. Then you can compute your subjective probability that $s < 0.99$ (or whatever your SLA is) using the CDF of the Beta distribution: $$p(s < 0.99) = \frac{B(0.99; a+N-F, b+F)}{B(a+N-F, b+F)}$$ where $B$ is the beta function (see Wiki on the link for details).
So how to choose a prior? A relatively uncontroversial choice would be the uniform prior $a = b = 1$, but this would make things quite tough for you, since you start out with 99% confidence that you will fail to make the SLA! In fact it would be pretty much impossible for you to overcome this with less than 100 trials (even if you succeeded in all 100 of them, you would still have 36% credence that you were failing your SLA). There are a couple other options:


*

*You could pool the results from similar clients as @fg nu commented.

*You could use the results from previous iterations of service. If you wanted to be fancy, you could weight them by time, so a success from 1 day ago counted as a full success, but a success from a year ago only counted as 1/2 success for your prior. If you make the weights decay exponentially, then this puts a limit on the amount that your prior can matter compared to the data you observe (although it's somewhat ad-hoc).

*You could pull something like $\Beta(101, 1)$ out of thin air. (This corresponds to saying that your client is as confident that you will uphold their SLA as they would be if they started from no preconceptions and then saw 100 successes in a row. They may or may not agree to this, but if their expectations are much lower than that, perhaps you need different clients!)
Of course, whether you could get your clients to understand this--let alone agree to it--is another matter entirely! Negotiating over priors is a notoriously tricky and subjective business, even for statisticians, let alone people who just want to buy software.
