Beta distribution for uncertain binary trials

Question

I have a larger problem but have presented what I believe is a minimal example. Imagine that you are trying to determine the true probability of a potentially-biased coin landing on heads, and want to take a bayesian perspective. Our prior is hence that the probability of heads is beta(1,1) distributed. Say that we flip the coin once and we get a heads. Our posterior is now beta(2,1).

Then we flip once more, but the coin lands crooked against an object on the table. It looks like it would have landed tails, but say that we are only 70% sure that it would have landed tails (so 30% sure that it would have been heads).

Obviously the 'best' solution is to ignore and retest, but if these coin flips are limited/expensive that might not be ideal. So is there anyway to include this result even with the uncertainty? Possibilities I've considered are

1. Ignore result, p ~ beta(2,1)
2. Include and pretend we are certain, p ~ beta(2,2)
3. Include with uncertainty, p ~ beta(2, 1.7)
4. Include with uncertainty for both, p ~ beta(2.3 1.7)

Option 4 seems reasonable, but I'm worried this is a statistical golem and I'm missing something obvious. I'm trying to stay in the bayesian setting so the answer here Distribution of partially observable binominal parameter isn't sufficient. Cheers!

Answer 1

If the coin toss going wrong is just a random thing that has nothing to do with what the result of the coin toss would have been (had nothing gone wrong), then ignoring the result of this particular toss is the easiest.

Pretending that we are completely certain or that we only add a downscaled weight to the second shape parameter (options 2 and 3) ignores the possibility of the toss could have ended up as heads (i.e. it's not quite right).

Adding 0.3 and 0.7 is the right thing to do, if you truly believe that there was a 30:70 probability that the coin would have come up heads vs. tails. However, note you need to believe this no matter how unfair the coin would be in truth. Perhaps, it only looks like that conditional on the coin being fair? Let's look at an extreme example:

• You have observed 99 heads and 0 tails
• A coin toss goes wrong and you feel like that was 30% likely to be heads and 70% tails.

With option 4, your belief about the proportion of heads before this 100th toss was a 95% credible interval from 0.994 to 0.9997. After this toss it's 95% CrI from 0.95 to 0.998. Before this toss, what the probability that the proportion is below what is now your lower CrI limit was less than $2 \times 10^{-22}$, but now it's 0.025. You may question whether that seems quite right, but it's indeed the right update to your belief, if you really think that toss would have landed tails up with 70% probability and heads-up with 30% probability.

Another issue with option 4 is that if you keep having such "failed" coin tosses and they all favor tails over heads (in your judgement) by 70:30, then you eventually converge to believing in a probability of the coin coming up tails being 70%. Again, as above this may be the right update.

An alternative model of what is going on is that you think that if this is a fair coin, then what you saw was increasing the probability of this toss ending up tails from 50% to 70% (=increasing the log-odds from 0 to log(0.7)-log(0.3)=0.8472979). So, in that case your belief about the coin overall influences what you believe the outcome of this coin toss was. In fact, the more you learn, your opinion on this toss will change as more data comes in in the future. In that case, some simple conjugate updating rule will not work. I feat you'll have to write down an explicit model and do MCMC sampling for it, I fear. That could look like this:

• Observed coin tosses follow $Y_i \sim \text{Binary}(\pi)$
• Messed up tosses also follow $Z_i \sim \text{Binary}(\pi)$ for the latent (but unobserved) outcome they would have had
• We only observe that $ P(Z_i = 1)$ is $\text{logit}^{-1}(\text{logit}(\pi)-0.847)$.

That's actually surprisingly hard to code up in Stan (my normal preferred MCMC sampler) due to the discrete latent variable, but presumably this is possible to deal with, but it's definitely a bit messy.

Answer 2

I think you are overthinking the exact prior distribution to use. If you have examined the coin and it seems symmetrical, that information alone might push you closer to $\mathsf{Beta}(2,2)$ than to $\mathsf{Beta}(1,1).$ Also, there is accumulating experimental evidence that the method of tossing may have more to do with results than do minor imperfections in the coin.

An analysis after you have the data you can afford could show how sensitive your prior distribution is to your choice of prior.

Suppose you can afford $n = 10$ of your expensive tosses of the coin and you get seven heads and three tails. Then let's look at the practical difference between priors $\mathsf{Beta}(2,1)$ and $\mathsf{Beta}(2,1.7).$ Posteriors $\mathsf{Beta}(9,4)$ [solid blue] and $\mathsf{Beta}(9,4.7)$ plotted below.

hdr = "Posterior Distn's BETA(9,4) [blue] and BETA(9,4.7)"
curve(dbeta(x, 9,4), 0, 1, col="blue", lwd=2, main=hdr)
 curve(dbeta(x, 9,4.7), add=T, col="brown", lwd=2, lty="dotted")
 abline(h=0, col="green2")

Respective 95% posterior credible intervals are $(0.43, 0.90)$ and $(0.40, 0.87).$ Is the difference between these two posteriors going to make a practical difference in your course of action?

qbeta(c(.025,.975), 9,4)
[1] 0.4281415 0.9007539
qbeta(c(.025,.975), 9,4.7)
[1] 0.3974857 0.8731326