3
$\begingroup$

I constructed a statistical test and compared its performances to a test that already exists in the literature.

I found, after doing some simulations, that my test performs better when it comes to significance level ($\alpha$), but when I simulated the power ($1-\beta$), I found out that my test has less power than the other test.

Which test is better than the other?

$\endgroup$
4
  • 2
    $\begingroup$ If power falls below the significant level then your test is biased. $\endgroup$
    – Deep North
    May 5, 2017 at 10:49
  • $\begingroup$ @DeepNorth, I simulated the power of different alternatives and the minimum power I got was under the alternative and it was 0.495 (0.05), which is my significance level $\endgroup$
    – noob
    May 5, 2017 at 11:00
  • $\begingroup$ @DeepNorth, what exactly is a biased test? got any references? $\endgroup$
    – noob
    May 5, 2017 at 11:28
  • $\begingroup$ You can read this turing.une.edu.au/~stat354/notes/node80.html $\endgroup$
    – Deep North
    May 5, 2017 at 11:51

2 Answers 2

3
$\begingroup$

The most powerful test is always the best test provided it is of correct size. I do not think you have evaluated the size of your test under simulation. Generate data according to the null hypothesis and evaluate the frequency with which the null hypothesis rejects the null. It should be exactly 0.05. If it is under 0.05 your test is conservative and may have a correction.

Fisher's Exact Test is an example of a conservative test: it does not reject the null as often as we think it does. This results in a lower power test. The Pearson Chi-square test achieves the correct 0.05 level and thus has better power.

$\endgroup$
2
  • $\begingroup$ I simulated my test under the null and it gives about 0.05, somtimes it's a little bit more (like 0.0526), somtimes it's a little bit less (like 0.048), but the second test always gives 0.09. So my test has the correct size, but less power $\endgroup$
    – noob
    May 12, 2017 at 14:54
  • $\begingroup$ @Enthusiastic then the second test is said to be anti-conservative. It rejects the null hypothesis too frequently. To think of extreme examples: 3 tests: one reject the null never, one reject the null all the time, and one reject the null with 5% prob randomly. These tests all suck because they don't use the data. The second test is an example of a poorly calibrated test, it has 100% power but a 100% type-1 error rate as well. If you claim a 0.05 nominal type 1 error rate but it's actually 0.10, I'd distrust those test results. $\endgroup$
    – AdamO
    May 12, 2017 at 15:22
2
$\begingroup$

In the absence of a specific loss function, that is hard to say. If the other test has higher type I error rate and lower type II error rate (i.e., higher power), the answer boils down to whether you consider a type I or a type II error to be more consequential.

$\endgroup$
4
  • $\begingroup$ So it's up to the user and the scenario he's in. Isn't there any corrections to make a biased test unbiased? is a unbiased always better than a biased test? $\endgroup$
    – noob
    May 12, 2017 at 14:57
  • 1
    $\begingroup$ As for corrections, I would not know of any general recommendations. All else equal, an unbiased test evidently seems preferable, but I would again not make that a general statement. Consider a test that is biased only because it happens to have power lower than size for some very specific alternative, but that has excellent power against all other alternatives. I would not rule out such a test. $\endgroup$ May 12, 2017 at 15:49
  • $\begingroup$ So there's no general way of comparing the two tests, it's all up to the user to decide which error type he wants to control, right? $\endgroup$
    – noob
    May 12, 2017 at 16:24
  • 1
    $\begingroup$ In your particular case, that may be the answer, although in general, there are of course cases where one can clearly rank tests, e.g., as AdamO points out, you'll always prefer a uniformly most powerful test if it exists. $\endgroup$ May 12, 2017 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.