I'm building a system to alert when the proportion of successes to attempts goes out of control. To visualize the data, I've produced some rolling control charts to get an idea of when things would have alerted.
A sample of one such chart is here, roughly labeled in photoshop for some clarification:
Since each interval (day) has a widely varying amount of attempts, I'm concerned that I am going to have outliers occur due to sparse data (and low confidence) and that trigger an alarm. I'd like to only trigger an alarm when I have enough data to be confident in the generated proportions, but at the same time, leverage as much data as is available.
Is it a good idea to incorporate confidence intervals in such cases, and only trigger if I am, say, 75% confident that the point is out of the lower control limit?
It seems like if I have several points in a row that are out of the lower control limit, each with 50% confidence then it's a 75% chance that one of them is and I should alert.
Am I off on a wild track here? Has this ever been suggested as a best practice?
I am new to control charts and have just ordered Donald Wheeler's book. On a side note, I am beginning to question my decision to use geometric means and standard deviation as the chart doesn't ever appear to swing up as dramatically as it does down (geometrically speaking).
