3
$\begingroup$

I am running a discontinuous regression to see the effect of a cash transfer on an outcome using a poverty index as the running variable. The problem is that the score is not a very good predictor of the treatment, with a compliance of about 40%. Moreover, the literature suggests that I should look for a minimum detectable effect size of about 0.1 standard deviations. Therefore, when doing the power calculations, I get that, although my sample is not small (~ 16 000 obs.), the power is still insufficient.

Reading the paper of Cattaneo et al. (2024; A practical introduction to regression discontinuity designs: Extensions), I notice that, in cases where there are not many observations, one can assume a random assignment just above and below the cutoff, and use Fisherian inference (which assumes a non stochastic potential outcome and uses a sharp null hypothesis) to still get robust results (although you have to give up point estimation).

Can the same logic be applied when you the results are underpowered not because of a small sample size, but due to imperfect compliance?

That is, if I get an non significant result with the common regression discontinuity approach and then I also get a non significant result when running the same regression but with Fisherian inference, is this last result reassuring? Can I be fairly confident that the non significance is not due to chance, but due to the absence of an effect?

$\endgroup$
7
  • 1
    $\begingroup$ Where in your link is it mentioned that one can Fisher's Exact test? $\endgroup$ Commented May 24 at 0:13
  • $\begingroup$ Page 15 recommends its use when sample size is small. In page 46 talks about its potential use when the instrument is weak. $\endgroup$ Commented May 24 at 8:42
  • $\begingroup$ I see "For example, in the context of local randomization methods, Fisherian inference methods continue to be valid...". Is this what you're referring to? I don't think this references FIsher's exact test. Rather, I think it refers to randomization inference, of which I believe Fisher was also involved. $\endgroup$ Commented May 24 at 14:27
  • $\begingroup$ Yes, you are totally right! For some reason both concepts where mixed up in my mind. I edited the question, thank you! $\endgroup$ Commented May 24 at 15:22
  • 1
    $\begingroup$ I don't know anything about discontinuous regression, but I do know that ideas like "the literature suggests that I should look for a minimum detectable effect size of about 0.1 standard deviations" are unhelpful. Is an effect of 0.1xSD large enough to matter to the trial participants? Is it a realistically achievable effect size to design a trial around? $\endgroup$ Commented May 24 at 22:02

1 Answer 1

2
+50
$\begingroup$

Your core question is "Can the same logic be applied when you the results are underpowered not because of a small sample size, but due to imperfect compliance?"

The "logic" of using randomization ("Fisherian") inference in the case of small samples is that asymptotic approximations are often poor approximations for small samples. Young (2018) provides a recent empirical demonstration where he finds between 11 and 49 percent fewer significant results when he reanalyzes a set of published papers using randomization inference. (It turns out that using an appropriate method for computing standard errors produces similar results.)

Ding (2017) also demonstrated that, paradoxically, asymptotic tests of no average treatment effect have more power than the Fisherian test of no treatment effect whatsoever (sharp null) in large samples even though the latter implies the former. Ding isn't looking at randomization tests using regressions, but I suspect his result would hold there too.

So, no, randomization inference is unlikely to help you if you have low power. You'll likely have less power with randomization inference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.