0
$\begingroup$

I'm using an online tool to test a hypothesis. I did the original analysis with FDR set at 0.01 but I'm not seeing the expected result. Perhaps it got filtered out in the statistical test. What FDR value do I set to essentially eliminate the statistical test (so that everything is statistically significant)?

As a follow up question, what p-value do I set to essentially eliminate the statistical test? I'm guessing set p-value to 1.

$\endgroup$
2
  • $\begingroup$ You want to always reject the null hypotheses? Set $\alpha = 1$. That's, of course, useless. $\endgroup$
    – Firebug
    Commented Nov 16, 2017 at 18:14
  • $\begingroup$ That's what I thought. I just wanted to double check. I'll set the FDR to 0.05 (less stringent) and check the results. Setting alpha = 1 will give false positives. $\endgroup$ Commented Nov 16, 2017 at 18:18

1 Answer 1

3
$\begingroup$

FDR does not eliminate statistical tests. If you wish to eliminate tests, you remove them from the hypotheses considered by-hand.

FDR will reclassify tests which are naively classified as significant at the $\alpha$ level as non-significant, which reduces both the familywise false positive error rate as well as the number of false discoveries.

You do not iteratively set the FDR to see an expected result. That is p-hacking. You instead specify a priori what is an acceptable rate of false discoveries and report all hypothesis tests which are classified as statistically significant. If you wished to test a particular inferential result, then FDR (or multiple corrections in general) is not justified since it treats all hypotheses with equal weight.

For an exploratory aim, you can inspect what the naive inference was for an outcome of interest. But for reporting your primary results, you should not list it among the significant findings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.