I'd like to know if there is a method for defining significant level based on a pilot study e.g. 73% successful rate. The background problem of my current study is at a very small sample size (N=9), which always yields non significant results when testing against the standard 0.10, 0.05 or 0.01 alpha values.

Any pointer is much appreciated.

Edited: In the pilot study where 11 left-handed people were recruited to test a new mouse invention. The test outcome resulted in 8/11 people said the new mouse was helpful to their computing tasks and the rest (3/11) said it was not helpful. I roughly calculated the success rate as (8/11)*100 = 72.72% or ~73%.

In the second study for which the improved version of the mouse was tested, 9 people were asked to repeat some computing tasks twice, one time using the new mouse invention and another time using an ordinary mouse decided for right-handed people. I have done some hypothesis tests (against 0.05 significance level) that resulted in Type II error as mentioned in ssdecontrol's answer below and would like to know how to assign a new significance level based on the pilot study result, instead of using 0.05 significance level.

  • $\begingroup$ You may need to provide more information. For example, what do you mean by a 73% success rate? $\endgroup$ – Ian_Fin Nov 11 '16 at 15:09
  • $\begingroup$ My apology for insufficient information. To explain 73% success rate result from the pilot study, this was calculated from 8 out of 11 test participants giving "yes" responses and 3 out of 11 giving "no". $\endgroup$ – Eye Ant Nov 11 '16 at 15:13
  • $\begingroup$ @EyeAnt be careful. That is not the same thing as $\alpha$ in a statistical hypothesis test. $\endgroup$ – shadowtalker Nov 11 '16 at 15:19
  • $\begingroup$ In addition of the answer of @ssdecontrol, to give test-specific answers, maybe you could say some more about the type of tests you are doing. From your comments it seems that you have binary data of 8 'yes' and 3 'no', but what are you trying to do with this? $\endgroup$ – JAD Nov 11 '16 at 15:21
  • 1
    $\begingroup$ Perhaps I'm missing something, but how do you know that study 2 was a type 2 error? It seems to me that you've decided that the results of study 1, which itself had a small N, represent the truth. How are you so sure that the results of study 1 weren't a type 1 error? $\endgroup$ – Ian_Fin Nov 11 '16 at 16:02

Recall that errors in hypothesis testing can be broken down into "type I errors" and "type II errors":

Let's say we are attempting to falsify the hypothesis that "blue-eyed people live longer than brown-eyed people."

  • Type I ("false rejection of a true null hypothesis"):
    • Test result: blue-eyed people can be expected to live longer than brown-eyed people
    • Reality: blue-eyed people and brown-eyed people have the same expected lifespans
  • Type II ("failure to reject a false null hypothesis"):
    • Test result: I cannot tell from the data whether blue-eyed people live longer than brown-eyed people
    • Reality: blue-eyed people do live slightly longer than brown-eyed people

In the standard statistical framework for conducting hypothesis tests, we must specify in advance the maximum Type I error rate we are willing to accept. This is what people commonly call the $\alpha$ level (or "size") of the test.

There is usually a trade-off involved in selecting $\alpha$: if it is set very low (i.e. you are very stringent about Type I errors -- false rejection of a true null) then the test will also tend to lack power (i.e. you are very susceptible to Type II errors -- failing to reject a false null).

The level 0.05 is arbitrary. If your data is very small, it may be the case that to obtain the desired statistical power you must set $\alpha$ much higher than this.

This is easier to do when there is a quantifiable cost of each error type. In that case, you can actually compute the expected cost of any hypothesis testing procedure in advance, and then choose an $\alpha$ that optimizes this cost.

For example: suppose you work for a life insurance company. Therefore the question of "do blue-eyed people live longer than brown-eyed people?" has quantifiable business implications. In this case, you will be able to (at least roughly) estimate the actual monetary cost of each result, and from there it becomes a straightforward optimization problem to find the cost-minimizing $\alpha$.

  • $\begingroup$ "The level 0.05 is arbitrary. If your data is very small, it may be the case that to obtain the desired statistical power you must set αα much higher than this." Please could you explain more on how to define a higher α value based on my pilot study result. If the success rate of 73% information is insufficient, what other information is required to do so. $\endgroup$ – Eye Ant Nov 11 '16 at 16:05
  • $\begingroup$ @EyeAnt sample size. You might find this page helpful: effectsizefaq.com/2010/05/31/… $\endgroup$ – shadowtalker Nov 11 '16 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.