Significance test: how to define alpha levels other than the standard 0.10, 0.05 or 0.01

Question

I'd like to know if there is a method for defining significant level based on a pilot study e.g. 73% successful rate. The background problem of my current study is at a very small sample size (N=9), which always yields non significant results when testing against the standard 0.10, 0.05 or 0.01 alpha values.

Any pointer is much appreciated.

Edited: In the pilot study where 11 left-handed people were recruited to test a new mouse invention. The test outcome resulted in 8/11 people said the new mouse was helpful to their computing tasks and the rest (3/11) said it was not helpful. I roughly calculated the success rate as (8/11)*100 = 72.72% or ~73%.

In the second study for which the improved version of the mouse was tested, 9 people were asked to repeat some computing tasks twice, one time using the new mouse invention and another time using an ordinary mouse decided for right-handed people. I have done some hypothesis tests (against 0.05 significance level) that resulted in Type II error as mentioned in ssdecontrol's answer below and would like to know how to assign a new significance level based on the pilot study result, instead of using 0.05 significance level.

Answer 1

Recall that errors in hypothesis testing can be broken down into "type I errors" and "type II errors":

Let's say we are attempting to falsify the hypothesis that "blue-eyed people live longer than brown-eyed people."

• Type I ("false rejection of a true null hypothesis"):
  • Test result: blue-eyed people can be expected to live longer than brown-eyed people
  • Reality: blue-eyed people and brown-eyed people have the same expected lifespans
• Type II ("failure to reject a false null hypothesis"):
  • Test result: I cannot tell from the data whether blue-eyed people live longer than brown-eyed people
  • Reality: blue-eyed people do live slightly longer than brown-eyed people

In the standard statistical framework for conducting hypothesis tests, we must specify in advance the maximum Type I error rate we are willing to accept. This is what people commonly call the $\alpha$ level (or "size") of the test.

There is usually a trade-off involved in selecting $\alpha$: if it is set very low (i.e. you are very stringent about Type I errors -- false rejection of a true null) then the test will also tend to lack power (i.e. you are very susceptible to Type II errors -- failing to reject a false null).

The level 0.05 is arbitrary. If your data is very small, it may be the case that to obtain the desired statistical power you must set $\alpha$ much higher than this.

This is easier to do when there is a quantifiable cost of each error type. In that case, you can actually compute the expected cost of any hypothesis testing procedure in advance, and then choose an $\alpha$ that optimizes this cost.

For example: suppose you work for a life insurance company. Therefore the question of "do blue-eyed people live longer than brown-eyed people?" has quantifiable business implications. In this case, you will be able to (at least roughly) estimate the actual monetary cost of each result, and from there it becomes a straightforward optimization problem to find the cost-minimizing $\alpha$.