# “The statistician should set the probability of Type I error as low as possible”

Do you agree with the following statement? Why or why not?

I disagree with this statement but i have no reasons to back up. Anyone can help?

• Should this be community wiki? – Glen_b -Reinstate Monica Mar 24 '13 at 10:53
• It depends in part on the potential consequences of either a type 1 or 2 error, particularly in a safety context – Robert Jones Mar 24 '13 at 14:30

It's possible to set it to zero.

In general it would make for a test with no power.

So I disagree with the statement because - except in a few very special instances - setting it as low as possible makes for a pointless test.*

* [In fact, in the case of point nulls, given that point nulls are almost never exactly true, it may arguably make more sense to set it as high as possible.]

A sensible approach would look at the tradeoff between type I error rate and power - you can get more power (correct rejections) by accepting a higher rate of type I error (false rejections) -- and conversely you can lower type I error by giving up some power. The choice of significance level more properly relates to how you weigh the relative importance of the two in a particular situation.

Here's the rejection rate for the two-sample t-test (i.e. power everywhere but at $\delta=0$) at n=14:

the black curve shows $\alpha=0.10$, the red curve shows $\alpha=0.01$. When the true means are $\sigma$ apart ($\delta=1$ above), the power is $82.4\%$ for $\alpha=0.10$ and $46.1\%$ for $\alpha=0.01$. Obviously you'd prefer the higher power in this case; but** you can only get it at the cost of accepting an increase in type I errors. Of course there's a whole range of intermediate values of $\alpha$ that will give power in between. Your exercise is to choose which curve you want, which is always a tradeoff.

** (assuming your $n$ is fixed -- and if your budget is finite there will be pretty strong limits on how far you can play with $n$)

As low as possible is ill defined. You can always make a test with the probability of a Type 1 error 0 but then this test has no power at all. (Think what happens if you take a p-value of 0). You will have to make a trade off between Type 1 and Type 2 errors and what the impact is of each of these errors.

In an ideal world, the probability of type I error would be adjusted on a case by case basis, rather than being set to .05 or .01 or whatever by fiat.

Less type I error means more type II error. In many fields, defaults are 0.05 and 0.20 for these two errors (that is, power of 0.8 is sought with p 0.05). Are these choices reasonable? It depends.

Suppose you are testing a drug that may cure a quickly terminal disease. Then type I error would mean people with the disease die when they could be cured, while type II error means dying people take a drug that does not work.

I'd be willing to have a higher type I error in such a case.

Over and above this, I always prefer looking at effect sizes rather than significance levels. What people care about is how much of an effect a (treatment, drug, condition) is likely to have; not whether a test statistic as extreme as the one we got could have easily arisen from a population where there was no effect.

• In an ideal frequentist world. There are other worlds that are worthy of consideration in the context of statistical inference. – Michael Lew Mar 25 '13 at 0:25
• In your example, the null hypothesis is "Drug has no effect" and alternative is "Drug has effect on disease". Type 1 should be dying people taking a drug that doesn't have any effect. Your example for Type 1 and 2 are reversed. – HelloWorld Dec 31 '14 at 0:27

Usually the statement is false.

The statistician should make sure that the chance of false inference is as low as possible. Sometimes that would involve minimising the type I error rate, but mostly is involves looking at the problem differently. In other words, there are many circumstances where the type I error rate should be of no interest, and in those circumstances the statement is wrong.

The type I error rate relates to the global false positive error rate in a notional series of experiments, and it is useful in planning experiments and for testing hypotheses with an automatic decision rule. In many cases (most scientific cases, in my opinion) the statistician should be concerned with the more local question of what the data allow him to conclude about the state of the world in the context of this experiment. That may involve consideration of the actual p-value (as opposed to determining if it is above or below a threshold in a manner that is consistent with quantitation of long term type I errors) or inspection of a likelihood function or a Bayesian posterior probability distribution.

Even if one is determined to work within a strictly frequentist framework it is sensible to examine the trade-of between type I and type II errors. It is unfortunate that we describe tests in terms that privilege type I errors over all other types of error.

I disagree. Consider a petty crime trial. A prosecutor's null-hypothesis is that you committed a crime. The court considers this hypothesis, otherwise there wouldn't be a trial to start with.

The easiest way to minimize Type I error is to simply accept prosecutor's case, i.e. find you guilty without deliberations. In this case probability of Type I error would be minimal: you will not be acquitted if you did it.

Unfortunately, this will increase the probability of Type II error: you'll be be found guilty even if you didn't do it.

That's why we have presumption of innocence and a trial with competing prosecutor and defense, because sometimes Type II errors cost too much. You have to consider the costs of both errors when deciding on the balance between them.

UPDATE: this is somewhat off-topic, but still interesting.

whuber thinks that presumption of innocence applies to a prosecutor. I'm not a lawyer myself, but I know that it's clearly not the case in USA.

Consider the following quotes from Laudan, L. (2006). Truth, Error, and Criminal Law: An Essay in Legal Epistemology. Cambridge: Cambridge University Press, pp.93-96:

One might have supposed, naively as it turns out, that a person is presumed innocent from the moment police first suspect him until his trial ends and a verdict has been pronounced (supposing things progress that far). One might have supposed, equally erroneously, that all those charged to make decisions about the defendant (judges, grand juries, prosecutors, and so on) are under some duty to presume his innocence.

While such a broad construal of the meaning of PI is common in several countries (at least in theory), the Supreme Court has made it very clear that in American criminal procedure, the PI applies only to the trial itself and, within the trial, only to the jury or other trier of fact.

The Louisiana Supreme Court put the same argument in blunter terms:

The presumption of innocence is a guide to the jury. If it were absolute and operative at every stage of a prosecution, the defendant could never be jailed until conviction. It does not prevent arrest. ... There is little relationship between the right to bail and the presumption of innocence. The presumption of innocence is operative and protects against conviction, not against arrest (which is taking into custody).

State v. Green, 275 So.2d 184, at 186 (La., 1973).

Apparently, Lord doesn't like both Type I and II errors:

Proverbs 17:15

He that justifieth the wicked, and he that condemneth the just, even they both are abomination to the LORD.

• This definitely is not how things work in the United States, where everybody's "null hypothesis" is the presumption of innocence. What body of jurisprudence or culture are you assuming for context? – whuber Dec 26 '14 at 0:05
• Prosecutor brings the case because she thinks you're guilty, and tries to convince the jury or a judge. Prosecutor doesn't presume that you're innocent, otherwise she wouldn't bring the case to a trial. – Aksakal Dec 26 '14 at 0:08
• Right: But in order to prove guilt, the prosecutor is obliged by law to presume innocence and present evidence causing this presumption to be rejected. The null hypothesis is not what the prosecutor believes! – whuber Dec 26 '14 at 0:09
• @whuber, thanks for bringing this up. You're right that PI works differently in other countries. I was describing US-centric process. – Aksakal Dec 26 '14 at 0:31
• The issue lies partly in "The court considers this hypothesis" - but not as its null hypothesis. What the prosecutor decides is relevant for whether the trial occurs at all but the judgment of the prosecutor in that decision is based not so much on whether they suspect the accused is guilty, but how likely a court would find the admissible evidence sufficiently convincing to secure a conviction. That is, the prosecutor and court see different evidence and are trying trying to form different conclusions. – Silverfish Dec 26 '14 at 0:50

In general, there is no consensus on how large one should set the type 1 error. In slightly high-powered language, it doesn't matter when the set of best tests forms a complete class. In such cases, all best tests are admissible, one may choose the type 1 error (size of the test) freely and the choice of type 1 error has no effect on the credibility of inference.

It is traditionally set at lower values, 0.01, 0.05, 0.1 for good reason: in science, rejection is more pronounced than acceptance because it supports the falsification. Rejecting the true is fatal, accepting the false is undesirable, but tolerable. Setting at too low leads to, as demonstrated by @Glen_b, frequent acceptance of the false, so we never know what is true or false.