This is a weird problem, but please bear with me. I could use some insight.
Imagine you have a black box that spits out p-values for $H_0: \theta = 0$. We'll call this black box $g(\cdot)$. Assume this black box is properly calibrated to make Type 1 errors only $\alpha$% of the time.
Examples: I put $\hat\theta = 0.2$ into my black box and I get $g(\hat\theta) = g(0.2) = 0.6$ and I fail to reject $H_0: \theta = 0$. Next I put $\hat\theta$ = 1.5 into my black box and get $g(1.5) = 0.01$ and I reject my null hypothesis.
Given that I have a method for generating p-values with a closed form expression, can I invert this procedure to generate confidence intervals with $1-\alpha$ level coverage?
It seems that I should, but I'm getting lost in the details. A confidence interval can be found by inverting a test statistic. I do have a test statistic, $\hat \theta$, but I do not have a sampling distribution for this test statistic.