For binomial success / fail data, how to calculate if the time since last success is out of the normal range

Question

Struggling to figure this out.

I have data sets that are essentially

ObjectId, Date, Pass/Fail (1/0)

The data is normally daily.

What I want to figure out is IF the number of days since the last pass is normal, or, out of the ordinary

For some of the data, it might show that we, on average, it passes every day.. In this case, a fail in the last 1 or 2 days is out of the ordinary

For other data, we might only have a pass on average every 10 days. For this data, what is out of the ordinary?

I tried just averaging the pass/fails into an average number. But in this case, roughly 50% of cases end up showing on my report

So - Any ideas for a formula to calculate some confidence that a particular object has had too many days since it's last pass? (based on the average number of days that it usually passes)

Let's say, for sake of argument that I look at a 90 day window.

The current things I have tried to calculate is:

Number of of passes divided by total tries (pass rate) Number of total tries divided by passes (avg days between passes)

I assume some standard deviation calculation comes into play?

Answer 1

If the ordinary probability of Pass is $p,$ then you are right that on average you should expect a Pass every $1/p$ days. So roughly speaking, if $p = 1/10$ you might start to get suspicious something is wrong if you haven't seen a pass in 'many' more than 10 days.

Here is one way to quantify 'many'. The probability of a Fail on any one day is $1-p$ and the probability of a run of $x$ consecutive failures is $(1-p)^x,$ so maybe you should get suspicious if $(1 - p)^x < .05.$

For $p = 1/10 = 0.1,$ that's about 29 days, because $0.9^{28} = .052$ and $0.9^{29} = 0.047 < .05.$ [Computations in R.]

.9^28;  .9^29
## 0.05233476
## 0.04710129

For $p = 1/5 = 0.2,$ you should sound the alarm at about 14 successive Fails: $0.8^{14} = 0.044.$ Using the same criterion, you should suspect that $p$ must be below $1/2$ if you see five fails in a row.

Of course, you might choose an 'error' probability other than $0.05$ and then results would differ accordingly.

Note: More formally, this is a test the hull hypothesis $H_0: p = p_0$ against the alternative $H_1: p < p_0$ at the 5% level.

The relevant probability distribution is the geometric distribution. Notice that there are two forms of the distribution, one counting the number of tosses until the first Pass appears, and the other counting the number of Fails before the first Pass. (In R, the grometric distribution uses the latter formulation.)