# Can I use p-values in probability estimates?

Pretend I have a barrel of jellybeans. I have eaten thousands of them and 90% were delicious. Then I ate a green jellybean with a dent in it. It was terrible. It is the only one I tried. If I find another green jellybean with a dent what is the best way to estimate its probability of being delicious? The p-value for there being something special about green jellybeans with dents is 0.1 (as there is a 10% chance of getting a non-delicious jellybean given the null hypothesis). So, could I estimate the probability of the green jellybean with dents being delicious as: (p-value)*(fraction of delicious jellybeans) + (1 - p-value)*(fraction of delicious green jellybeans with dents)? And if I know the number of delicious jellybeans that are just green or dented independently, how do I include that information in my formula?

• you need to make a distinction between a proportion and significance test value.
– user10619
Nov 15, 2016 at 3:10
• You seem to be muddling together different ideas. This sounds like a Bayesian probability question, and in that case you shouldn't be talking about p-values. You might have some conditional probability. Nov 15, 2016 at 4:50
• @Glen_b Wouldn't a standard conditional probability give a 100% chance that green dented jellybeans are not delicious? Intuitively, I would expect a small chance that if I tried another green dented jellybean it would be delicious because the one I tried was a fluke. Nov 17, 2016 at 17:32

Your use of the term p-value in the question is misplaced. It has a particular meaning in statistics that doesn't apply in the way you're asking.

I'm going to discuss this problem from a number of points of view, but first let's sort out some notation. Let $D$ be the event "jellybean is delicious". Let "G" be the event "jellybean is green and dented". We're interested in $P(D|G)$, which I'll call $p$ for short.

If we look at a direct sample estimate for $P(D|G)$ (i.e. $\hat{p}=0/1=0$), because we have so few observations under $G$ (exactly 1), it's not very informative. After all, the observation is quite consistent with much higher values for $p$. Consider if $p$ had been $0.5$ for example -- it's easily consistent with observing a single trial with a non-delicious one. Even if $p=0.8$ it shouldn't astound us to get a non-delicious one in a single trial.

There's an ancient approach, called Laplace's rule of succession that would estimate $p$ to be $\frac{s+1}{n+2}$ (as if we'd observed one success and one failure before we started). In this case $s$ is the number of delicious green dented jellybeans we've encountered ($0$) and $n$ is the total number of trials involving a green dented jellybean, which is $1$, making the rule of succession estimate $1/3$. I wouldn't put much weight on it though and I think we could take issue with some of its assumptions.

If we look at various approaches for calculating binomial proportion confidence intervals (some of which won't be suitable for such small sample sizes), we see that several of them correspond to using "additional observations" -- e.g. an Agresti-Coull interval with $1-\alpha$ just above $0.95$ ($0.9545$) corresponds to taking two additional successes and two additional failures and applying the usual large sample normal-approximation for an interval, which interval is then is centered on $\frac{s+2}{n+4}$ A Wilson score interval at $1-\alpha\approx 0.95$ corresponds approximately to taking two additional successes and failures and applying the usual large sample normal-approximation, so the center of the Wilson interval is approximately $\frac{s+2}{n+4}$ as well. These would suggest estimating $p$ at $\approx \frac15$.

And if I know the number of delicious jellybeans that are just green or dented independently, how do I include that information in my formula?

This would suggest we apply Bayes' rule

$P(D|G) = \frac{P(G|D)P(D)}{P(G)}$

We have $P(D)\approx 0.9$ and we know $P(G)$ is very small (though an estimate of it is again quite uncertain because we've only seen one $G$ event). So you can see how those quantities would scale an estimate of $p$, but it's difficult to apply Bayes' rule directly here because we don't really have any more information on $P(G|D)$ than we do on $P(G)$; we saw 1 out of 90% of thousands; it's about equally as uncertain a quantity.

However we could calculate an upper bound of a Bayesian credible interval by using say Jeffreys' prior (note the issue with the lower bound in the $s=0$ case mentioned in the Wikipedia article link on binomial intervals above, under Jeffreys interval).

If we apply Jeffreys interval in the form suggested there, a 95% interval for $p$ would be in the vicinity of (0,0.85) and it would suggest a posterior mean of $\frac14$ (they don't say how to get that there, but it's the mean of a Beta$(\frac12,\frac{3}{2})$).

So far we've seen estimates of $p$ of $0$, $\frac13$, $\frac15$ and $\frac14$; however, as we see if we calculate any of the mentioned intervals, a wider range of values is consistent with the information we have. To say what would constitute a best estimate depends on your criteria for "best".

• If 99.9% of jellybeans are delicious I would estimate the chance of a green dented jellybean being delicious as being lower because there is less of a chance the one I tried arose randomly from that distribution. It seems like this idea is only captured by the null hypothesis from p-values. My criteria for best is the estimate with the minimal loss when tested on new observations. Nov 18, 2016 at 5:17
• Again, you can't use p-values like this; this isn't hypothesis testing so please don't use that term in this context. Can you frame your argument in terms of conditional probability? .... You mention minimal loss, but don't state a loss function. That loss should be minimized is generally taken as given so you didn't really add usable information there. Nov 18, 2016 at 5:23
• The null hypothesis is that X percent of all jellybeans are delicious. The hypothesis test is eating the green dented jellybean. I'm not seeing the problem. Nov 18, 2016 at 6:07
• The original question contains no null hypothesis that I recognize and "X percent of all jellybeans are delicious" is not a null hypothesis in the absence of a value for $X$. How would such an $X$ arise? Where does it come from in the original problem? Nov 18, 2016 at 6:15
• In the original problem X comes from the 90% in "I have eaten thousands of them and 90% were delicious." Nov 18, 2016 at 6:41