# How to calculate CI of success rate when subject changes?

I'd like to get the 95% confidence interval for whether or not a subject will brush their teeth. Basically it is a success or failure (doing it or not doing it). The measurements are continuously repeated and I'd like to update the CI twice a day (after the morning and after the evening) to always have the most accurate CI as new data becomes available. I think I can use the Wilson score interval to calculate the CI.

However, in my particular case, the subject can change. Imagine getting a new tenant. Since the new tenant is the same species, the same sex, and the environment is the same as well, I could (as a very rough guess) assume that the CI will be similar to the last tenant. However, if they have different brushing habits (more or less diligent), I need to update the CI as I get more data.

My problem is that I would like to get the best estimate possible, so using the previous tenants data (when there is little to no data from the new tenant) as some kind of prior might be a good idea. But how do I update as I get more data from the new tenant? At which point can I say that his brushing behavior is so different that the previous data should no longer be included in CI calculation etc.

I'm not familiar with working with priors or this kind of problem in general. So if someone has an idea how best to estimate the CI, it would be great if you could dumb down the math a bit.

• I'm having trouble following this example, it looks like you're mixing metaphors. Would it be easier to describe your specific analysis? Anyway, it seems like it's a time series problem and either you have changepoints, or you need to detect them, then restart your sequence periodically. Is that ballpark the problem? Jun 17, 2021 at 5:37
• Sorry, but the actual problem is even more odd. Basically I want to create a sorting order. I want to rank "things" based on their number of successes and failures (hence I started out with lower bound of Wilson score interval). Like with online reviews, 10 of 10 successes is not as trustworthy (and should not rank as high) as 995 of 1000. I'm fairly satisfied with the order that Wilson gives me, but there is one problem. There is an event that can change the average success rate of the thing I'm ranking. In that case, I don't wanna start Wilson anew with 1 of 1, but include prior knowledge. Jun 17, 2021 at 6:17
• But since the success rate might drastically change (or not at all) due to the event, I have no idea how to weigh the data prior and after the change. Continuing with regularly updating Wilson (basically ignoring that the event might have changed things), the score would only very slowly change. Especially if I already recorded a huge n. So even if the average success rate changes to 0, with a prior n of 1000, I would only very slowly get changes in the ranking after many unsuccessful trials. Jun 17, 2021 at 6:25
• Basically if thing "A" had a bad success rate of 0.5 and after the event, it goes up to 1.0, I don't want the old data to "punish" A's ranking in the long term, because it will now continue to perform well. But If another thing ("B") already had a high success rate, I don't want to decrease the score I use for ranking by resetting B's n to 0, because even after the change, it might continue to have a high success rate. Jun 17, 2021 at 6:41

If you work in a Bayesian framework, updating to take account of additional information can be easy. [Note: Bayesians tend to use terminology probability interval or credible interval instead of confidence interval.]

Prior distribution. Take the probability $$\theta$$ of brushing as a beta random variable with values in $$(0,1).$$ Without much information about a fresh subject, if you feel their probability $$\theta$$ of brushing has $$P(1/3 <\theta < 2/3) = 0.60,$$ you may decide on a prior distribution $$\mathsf{Beta}(3,3),$$ with density

$$f(\theta)\propto \theta^{3-1}(1-\theta)^{3-1},$$ where the symbol $$\propto$$ (pronounced 'proportional to') indicates that, for simplicity, we have omitted the constant factor that makes $$f(\theta)$$ integrate to unity over $$(0,1).$$ If you want to start with a 95% probability interval that matches this prior distribution, then it would be $$(.15,\, .85).$$

qbeta(c(.2, .8), 3,3)
[1] 0.3265979 0.6734021
qbeta(c(.025,.975), 3,3)
[1] 0.1466328 0.8533672


[Instead of guessing you might use an interval based on behavior of people you consider to be somehow 'similar'.]

Likelihood information. Each brushing or non-brushing is a Bernoulli trial with $$1$$ for brushing and $$0$$ for not. Maybe over the first two days, this person brushes 3 out of the four times of interest. Then their binomial likelihood is $$g(x|\theta) \propto \theta^3(1-\theta).$$

Posterior distribution. Then by Bayes theorem the updated interval estimate is based on the posterior distribution $$h(\theta|x)$$ found by multiplying the prior distribution by the likelihood, as follows:

$$h(\theta|x) = f(\theta)\times g(x|\theta) \\ \propto \theta^{3-1}(1-\theta)^{3-1} \times \theta^3(1-\theta)\\ \propto \theta^{6-1}(1-\theta)^{4-1},$$

which we recognize as the 'kernel' (density without norming constant) of the distribution $$\mathsf{Beta}(6,4),$$ so that the new 95% interval estimate is $$(.30,\,.86).$$

qbeta(c(.025, .975), 6,4)
[1] 0.2992951 0.8630043


Subsequent updating. Then this posterior distribution becomes the prior distribution for the next update. Each updated distribution adds $$1$$ to the first beta shape parameter for a brushing or adds $$1$$ to the second shape parameter for a non-brushing. For example, if the next event is a brushing, then the next updated 95% interval is changed slightly to $$(.35,\, .88).$$

qbeta(c(.025,.975), 7,4)
[1] 0.3475471 0.8784477


Moving window updates. If you continue increasing the beta shape parameters over a long period of time, then the updated 95% probability intervals will get ever narrower. At some point, you may feel that behavior in the distant past may no longer be of current interest. Then you could base updates on some recent window of time (say, the last three months), dropping earlier counts as later ones become available.

Someone with a three-month history of brushing $$3/4$$ of the time (on your twice daily schedule) would have an updated 95% probability interval of about $$(.65,.83).$$

qbeta(c(.025,.975), 70, 24)
[1] 0.6523714 0.8271333


Notes: (1) If you need point estimates at each step, you can use the mean of of the current beta posterior distribution. The distribution $$\mathsf{Beta}(\alpha,\beta)$$ has mean $$\mu = \frac{\alpha}{\alpha+\beta}.$$

(2) One Bayesian probability interval, the Jeffreys Interval, is often used as a frequentist confidence interval. Based on a non-informative prior distribution, it uses appropriate quantiles of $$\mathsf{Beta}(x + .5, n-x + .5)$$ when there are $$x$$ successes in $$n$$ trials. You can see the Wikipedia article on 'binomial confidence intervals' for more about this interval estimate.

(3) Depending on your interests, you may want to Google topics such as 'sequential Bayesian updating'.

• Thank you for the explanation. I did look up some stuff related to your answer but I fear it does not help me much. It won't let me chose 0 as the third parameter and it goes to a probability of 1.0 too fast. With the Wilson score interval I can compare the observation pairs 10 to 0 and 100 to 0 without issues. Also it seems that I would update the probability interval the same way, whether the subject changes or not. I'm looking for a way to sensibly update the probability when the subject changes. The old subject's data should not have the same impact as the new subject's data. Jun 17, 2021 at 8:29
• This is beginning to sound like a different situation than the one you described in your Question. Consequently, I am beginning to wonder if you should be be investigating control chars instead of interval estimates. Jun 17, 2021 at 14:33
• Sorry, I thought my comment is on point with paragraph 3 of my question. At least that was the intention. I actually found a solution to my particular problem that is outside the scope of the question, so I'm accepting your answer. It was easy to follow and well written. Thank you! Jun 17, 2021 at 16:18