# Invert P-Value to get Confidence Interval

This is a weird problem, but please bear with me. I could use some insight.

Imagine you have a black box that spits out p-values for $H_0: \theta = 0$. We'll call this black box $g(\cdot)$. Assume this black box is properly calibrated to make Type 1 errors only $\alpha$% of the time.

Examples: I put $\hat\theta = 0.2$ into my black box and I get $g(\hat\theta) = g(0.2) = 0.6$ and I fail to reject $H_0: \theta = 0$. Next I put $\hat\theta$ = 1.5 into my black box and get $g(1.5) = 0.01$ and I reject my null hypothesis.

Given that I have a method for generating p-values with a closed form expression, can I invert this procedure to generate confidence intervals with $1-\alpha$ level coverage?

It seems that I should, but I'm getting lost in the details. A confidence interval can be found by inverting a test statistic. I do have a test statistic, $\hat \theta$, but I do not have a sampling distribution for this test statistic.

• What are $\hat\theta$ and $g(\hat\theta)$ if not test statistics?
– whuber
Aug 3, 2018 at 17:06
• You're right. I will change my question a bit.
– Eli
Aug 3, 2018 at 17:10
• Now that I've thought about it a bit, I do have a test statistic. I do not have a sampling distribution for this test statistic.
– Eli
Aug 3, 2018 at 17:13

What you actually need is a function $g$ such that $g(\theta_0, D)$ returns a p-value for the test $H_0: \theta = \theta_0$. Then the collection $\{\theta : g(\theta, D) > \alpha\}$ is a confidence set with coverage level $1-\alpha$. In this case $D$ denotes your data.

If all you can do is test for $\theta = 0$ then I don’t think this is possible unless you make some strong assumptions about your $g$.

• Could you explain why $D$ is needed? If $\hat \theta$ is a sufficient statistic, then by definition of sufficiency $D$ would be superfluous.
– whuber
Aug 3, 2018 at 17:22
• @whuber when you have a sufficient statistic then you can use that in place of $D$, but this works even if you don't have access to a useful sufficient statistic. If you have a sufficient statistic, you can write $g(\theta, \widehat \theta)$, but the point is moreso that you need to be plugging in both some function of the observed data and some values $\theta$ (as opposed to just setting $\theta = 0$ as OP wants). Note also that this confidence region is valid with coverage level $1 - \alpha$ even if you don't base it on a minimal sufficient statistic.
– guy
Aug 3, 2018 at 18:14
• It is redundant to use $\theta$ as an input--that's already incorporated in how the black box works.
– whuber
Aug 3, 2018 at 18:39
• @whuber In the OP it was assumed that $g$ gives a $P$-value for the test $H_0: \theta = 0$. You need to be able to test $H_0: \theta = \theta_0$ for every value of $\theta_0$, i.e. you need to be able to compute something like $g(\theta_0, D)$ for many different values of $\theta_0$.
– guy
Aug 3, 2018 at 19:03
• @whuber $\theta$ cannot be "incorporated into how the black box works" because you don't know the true value of $\theta$.
– guy
Aug 3, 2018 at 19:06