Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Wikipedia defines size of a test as the maximum probability of committing a type 1 error. And I'm very confused about how this is different than the significance level alpha. It seems to me the definition for both is P(rejecting null | null is true).
Is the size of a hypothesis test dependent on the type of test (z-test vs t-test), and does it depend on other parameters such as power, effect size?
Could you please explain this in the context of a t-test?
Thanks,
Suppose, you've $X_1, X_2, \cdots , X_n \stackrel{\text{iid}}{\sim} N(\mu,1)$ , and you are testing $$H_0 : \mu \geqslant 0 \quad\text{against}\quad H_a : \mu < 0$$ Observe that the null hypothesis $H_0$ is a composite hypothesis, i.e. if $H_0$ is true, then for different values of $\mu~(\geqslant 0)$ , we shall obtain normal distributions with different parameters every time.
Therefore, for each possible value of $\mu~(\geqslant 0)$ , you get a different value of the type-I error (Size). For example, in this specific problem, there are uncountably many possible values of $\mu$ when $H_0$ is true. Now, the level of a test is defined to be the supremum of the set of all possible type-I errors. So, there may be infinitely many possible values of type-I errors. But the level will be a single real number, viz. the supremum of the set of all type-I errors.
Now, for the cases where $H_0$ is a simple null hypothesis, then the set of all possible type-I errors is just a singleton set. So, for that test, the level and type-I error are equal. e.g. suppose you're testing $$H_0 : \mu = 0 \quad\text{against}\quad H_a : \mu \neq 0$$ Then, the level and type-I error will indeed be equal.
Also, remember that the level or type-I errors of a test doesn't depend on its power, nor they are something special for different tests. They solely depend on the the type (Simple or Composite) of the null hypothesis $H_0$ , and also on the possible values that the concerned parameter may take if $H_0$ is true.
Hope this helps.
