This is a weird problem, but please bear with me. I could use some insight.

Imagine you have a black box that spits out p-values for $H_0: \theta = 0$. We'll call this black box $g(\cdot)$. Assume this black box is properly calibrated to make Type 1 errors only $\alpha$% of the time.

Examples: I put $\hat\theta = 0.2$ into my black box and I get $g(\hat\theta) = g(0.2) = 0.6$ and I fail to reject $H_0: \theta = 0$. Next I put $\hat\theta$ = 1.5 into my black box and get $g(1.5) = 0.01$ and I reject my null hypothesis.

Given that I have a method for generating p-values with a closed form expression, can I invert this procedure to generate confidence intervals with $1-\alpha$ level coverage?

It seems that I should, but I'm getting lost in the details. A confidence interval can be found by inverting a test statistic. I do have a test statistic, $\hat \theta$, but I do not have a sampling distribution for this test statistic.

  • $\begingroup$ What are $\hat\theta$ and $g(\hat\theta)$ if not test statistics? $\endgroup$ – whuber Aug 3 '18 at 17:06
  • $\begingroup$ You're right. I will change my question a bit. $\endgroup$ – Eli Aug 3 '18 at 17:10
  • $\begingroup$ Now that I've thought about it a bit, I do have a test statistic. I do not have a sampling distribution for this test statistic. $\endgroup$ – Eli Aug 3 '18 at 17:13

What you actually need is a function $g$ such that $g(\theta_0, D)$ returns a p-value for the test $H_0: \theta = \theta_0$. Then the collection $\{\theta : g(\theta, D) > \alpha\}$ is a confidence set with coverage level $1-\alpha$. In this case $D$ denotes your data.

If all you can do is test for $\theta = 0$ then I don’t think this is possible unless you make some strong assumptions about your $g$.

| cite | improve this answer | |
  • $\begingroup$ Could you explain why $D$ is needed? If $\hat \theta$ is a sufficient statistic, then by definition of sufficiency $D$ would be superfluous. $\endgroup$ – whuber Aug 3 '18 at 17:22
  • $\begingroup$ @whuber when you have a sufficient statistic then you can use that in place of $D$, but this works even if you don't have access to a useful sufficient statistic. If you have a sufficient statistic, you can write $g(\theta, \widehat \theta)$, but the point is moreso that you need to be plugging in both some function of the observed data and some values $\theta$ (as opposed to just setting $\theta = 0$ as OP wants). Note also that this confidence region is valid with coverage level $1 - \alpha$ even if you don't base it on a minimal sufficient statistic. $\endgroup$ – guy Aug 3 '18 at 18:14
  • $\begingroup$ It is redundant to use $\theta$ as an input--that's already incorporated in how the black box works. $\endgroup$ – whuber Aug 3 '18 at 18:39
  • $\begingroup$ @whuber In the OP it was assumed that $g$ gives a $P$-value for the test $H_0: \theta = 0$. You need to be able to test $H_0: \theta = \theta_0$ for every value of $\theta_0$, i.e. you need to be able to compute something like $g(\theta_0, D)$ for many different values of $\theta_0$. $\endgroup$ – guy Aug 3 '18 at 19:03
  • $\begingroup$ @whuber $\theta$ cannot be "incorporated into how the black box works" because you don't know the true value of $\theta$. $\endgroup$ – guy Aug 3 '18 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.