I have a very basic stats about significance values and p-value.

Say I perform a test using 0.05 as the significnace level. Using this p < 0.05, I obtain say 20 000 significant points. Is it valid to say that 5% of these points, so 1000 points, could be false positives?

Thanks in advance for your help

P.s I am familiar with multiple test correction

  • 2
    $\begingroup$ That doesn't sound right; if you have 20,000 "significant points", wouldn't each of these be subject to the 0.05 error? Though perhaps I misunderstand what the "significant points" are or how they might be obtained. If you meant to say that if you repeated the experiment 20,000 times, and the null hypothesis really was correct, you could expect to see ~1000 test statistics as or more extreme than the value observed in the one test (the one where you got a $p$ value < 0.05.) $\endgroup$ – Gavin Simpson Feb 18 '15 at 1:15
  • 2
    $\begingroup$ Can you clarify your question? Eg, what are the "significant points" you are referring to? $\endgroup$ – gung Feb 18 '15 at 1:25
  • $\begingroup$ @gung, My interpretation is that she's talking about the classification based on significance levels, like what is used in biostatistics. Say you run a blood test, and p-value is used to classify it as positive or negative for a disease. $\endgroup$ – Aksakal Feb 18 '15 at 18:10

If the null hypothesis is true, then yes, you'd have 1,000 false positive. However, if your alternative is true, then no, moreover, you won't have any false positives.

Note, that your significance level $\alpha$ defines your Type I error rate. False positive rate needs one more input: the rate at which null is true. In your case, in order to tell what's the number false positives you need to know what is rate of true nulls. Let's say 10% of nulls are true, in this case you get: 20,000x0.05x0.1=100 false positives are expected.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.