1
$\begingroup$

Let's say I have two hypotheses for a coin with probability $p$ for heads:

$H_0$ - the null hypothesis - the coin is fair $p = 0.5$.

$H_1$ - coin is unfair $p \neq 0.5$.

Say the test is $|X-n/2| > r$ for some $r$ where $n$ is the number of tosses.

It is possible to choose $r$ for a desired significance level quite easily (theoretically, all the details are there for computation).

But what if I want to design a test with a certain type II error? It seems odd to me that one cannot compute $p(\textit{accept } H_0 | H_1)$ because $H_1$ doesn't specify what is the value of $p$.

When I looked around, I noticed that for this simple example, it is requested to calculate type II error for a certain value of $p \neq 0.5$ (such as $p = 0.7$).

Is there a meaning to type II error when $p$ is unknown?

Also, how is that related to the idea of "frequentist repeated experiments?"

Thanks.

$\endgroup$
  • $\begingroup$ Is your $H_1$ that p>0.5 or that it is not equal to 0.5 (as implied by your test statistic)? $\endgroup$ – Peter Ellis Jan 3 '13 at 22:49
  • $\begingroup$ I guess I misunderstand something - aren't the test statistic and the hypothesis decoupled from each other? I can set $H_1$ to whatever I want, $H_0$ to whatever I want, and the test statistics to whatever I want, and calculate the errors of both kinds. Or maybe you are saying out of common sense, I should change $H_1$. $\endgroup$ – hypothizer Jan 3 '13 at 22:54
  • $\begingroup$ I changed it to be $\neq 0.5$. $\endgroup$ – hypothizer Jan 3 '13 at 22:54
  • $\begingroup$ You don't need $H_1$ at all in some approaches (where $H_1$ is just NOT $H_0$), but if you have one it needs to be consistent with your statistic you are using to choose between the two. The question is clearer now. $\endgroup$ – Peter Ellis Jan 3 '13 at 23:09
  • $\begingroup$ Re: "Is there a meaning to type II error when p is unknown?" - well, sure, it has a meaning, but that it carries meaning doesn't imply you can compute its value. $\endgroup$ – Glen_b Jan 3 '13 at 23:38
1
$\begingroup$

The meaning of a type II error is not dependent on knowledge of the true parameter value, so the direct answer to your first question is "yes". If you accept the null hypothesis when the alternative is true then you've made a type II error (although you won't know it).

However, I suspect that you really want to know about the rate of making type II errors, or the power. The answer to questions about them are more complicated. To calculate the power of an experimental design (and analysis) you have to specify an alternative value for the parameter of interest. Just saying that p differs from 0.5 is not enough because if the true value of p is 0.51 then the experiment will have low power unless the sample is very large, but if the true p is 0.99 then the experiment will have high power with a relatively small sample. The reason that the experimental design aspects relating to type I error rates and type II error rates are so different is that type I errors can only be made when the null is true, and you've specified that condition exactly, whereas type II errors can only occur when the null is false, and you have not specified that condition except as anything except p = 0.5.

Note that the power of the experiment relates to the experimental design, not to the experimental results. Thus you can calculate the power for any parameter value, you don't need knowledge of the true value of the parameter.

The longer answer to your first question (the version that I think you wanted to ask) is that you cannot calculate the power because you have not specified a value of the parameter in the alternative hypothesis. Nonetheless you can still make a type II error.

Your second question about "frequentist repeated experiments" is bigger and my answer may be controversial. The word "frequentist" has different meanings to different people. Some (like myself) find it to be a useful label for statisticians who default to a definition of probability as being a long-run frequency of events. Others prefer to think of frequentists as being those who focus on the frequency of errors of the first type and the second type (i.e. those who work within the Neyman-Pearson error-decision framework). The distinction is important because the first type of frequentist can treat experimental data as evidence that can be used for inductive reasoning, whereas the latter type is constrained to practice inductive behaviour without (or with much less) regard to issues of evidence.

Fisher championed the notion of scientific induction and Neyman championed the notion that experimental results do not allow direct inductive reasoning about the state of the world. (Fisher and Neyman were at war until Fisher died and both played up their disagreements at the expense of their agreements. Much confusion in statistics stems from attempts to amagamate their essentially incopatible approaches.) While they agreed that the meaning of an experimental result related to notional results expected from notional replications of the experiment (your "frequentist repeated experiments"), their conceptions of the type of things that could be considered as repeated experiments differed substantially. Fisher thought of replication of the particular experiment and so interpreted the results of an experiment as evidence relating to the state of the world in that experiment, whereas Neyman preferred to consider replication in a more global manner. To him the repeated experiments were more like all of the experiments than an experimenter might conduct in his or her career. Thus for Neyman the results of a particular experiment are interpreted as being part of a global series of experiments and thus they relate to long term error rates without being evidence about the state of the world pertaining to the particular experiment in question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.