# Is a 'Type 1 error' an indication of error, a probability of error, a measure of error or all of the above?

I have come across two definitions of 'Type 1 error' in dictionaries published by Oxford University Press:

1. In hypothesis testing, the incorrect rejection of the null hypothesis when it is true. Alternatively termed false-positive.
A Dictionary of Business Research Methods (2016)

2. The probability of being wrong in rejecting a null hypothesis about a parameter in a model. A Dictionary of Social Research Methods (2016)

I am not a statistician, but these seem to me to be two different definitions. After further reading, I'm led to believe that there are two different approaches to hypothesis testing:

1. A 'classical' method of conducting hypothesis testing which produces one of two results: either True or False.

2. A Bayesian factor approach which produces a probability as its result, rather than returning either True or False.

I suppose I have two related questions:

1. Am I correct in thinking that the two different approaches to hypothesis testing I note above exist?

2. If so, is the Type 1 error as probability a measure of how erroneous the error is, or how probable it is that there is an error in the first place?

The reason I ask my second question is that in my language we use a different word for an error (='mistake') and error (='measure of error', or literally 'astrayness'). In other words, senses 1.1 and 1.2 here use different words in my language, and if the probability-based Type 1 error is a measure of error, we would have to use a different term for it than that used for the true/false Type 1 error.

Thanks for taking the time to read. Any help will be gratefully received!

You are extremely perceptive in the way you've asked the question. To begin with, the second definition is completely wrong. For it to be in a dictionary of social research methods is quite astounding. First of all, a type I error is a result or an action, not a probability (the dictionary should have said "type I error probability or type I error rate (but see below)" if they wanted to address a risk. But a more fundamental problem with definition 2. is that null hypothesis statistical tests (NHST) assume the null hypothesis is true. The probability of being wrong about rejecting the null hypothesis is the prob. the null is true when you reject. Frequentists can't know that but have already assumed the null is true. You are also very perceptive in using words like "measure of error" or "astrayness". Below I try to use slightly better wording.

"Errors" in NHST are not really errors in the traditional sense. The p-value is "the degree to which the data are embarrassed by the null hypothesis" and is a measure of "surprise" given by the data if H0 is true. Type I error probability is the probability of making an assertion of a nonzero effect if the effect is truly zero. Once you understand the difference between assertions and conclusions, things get more clear. These issues have been written about by William Briggs, Steve Goodman, and many others, and I am putting together training materials that goes into details and gives a lot of references. I go into detail about some of the concepts in some of my blog articles here.

William Briggs has decoded Ronald Fisher's statement "either an event of low probability has been observed or the null hypothesis is false" to be logically equivalence to "an event of low probability has been observed". So p-values tell us less than we need. Cohen has shown that proof by contradiction doesn't really work when uncertainties are involved; it only works with logical certainties.

The shortest definition of p-values and type I error probabilities I can come up with are as follows. The p-value is the probability that someone else's data will be more impressive than yours if their data were generated with the null hypothesis in force. If you set a rejection levels of $\alpha=0.05$ then type I error is the probability that the p-value will be less than 0.05. This is by definition 0.05 if the p-value is computed exactly correctly, there was only one look at the data, and the sample size is a constant. When multiplicities exist as when doing sequential testing, and if each test is done at a nominal $\alpha$ level, the type I error will be $> \alpha$.

If you use a hard-and-fast rule that you'll make an assertion of an effect if p < 0.05 (which is seen as quite silly nowadays) then type I error probability is the probability of making an assertion of an effect when there is no effect. With the single test fixed sample size situation, this is 0.05 by definition, independent of data. As Jeffrey Blume has pointed out, this results in all kinds of craziness, because most type I errors occur when the observed effects are small. Not to condition on the observed data when computing probabilities really misses this point, and fixing type I error at, say, 0.05 no matter how large the sample size becomes will insure that your type I error never vanishes no matter what is going on.

Pure subjective Bayesian statistics could be said to not require complex notions of inference and counterfactuals and use only observed quantities in calculations. Bayesian posterior probabilities allow you to carry along probabilities that assertions hold, to make optimum decisions. Once you think probabilistically, notions of assertions, conclusions, inductive vs. deductive reasoning, etc. can be somewhat ignored. Posterior probabilities are fully conditional on the data observed so far, and quantities such as P(unknown effect > 0 | data) will be low when the data are tilted away from a positive effect. One minus this probability is the probability that the effect is zero or is in the wrong direction. So Bayesian posterior probabilities have a dual purpose - providing both evidence for and evidence against unknown effects. With NHST you can only amass evidence against H0, never evidence in favor of H0.

• Thank you very much for taking the time to write such an in-depth reply, and for your kind words. It's taking me quite some time to process all the information that you've shared! I find it strange to think of a Type 1 error as a probability rather than an error in the traditional sense. If I understand you correctly, the classical Type 1 error is "the assertion of a nonzero effect if the effect is truly zero", whilst in NHST Type 1 error is "the probability assertion of a nonzero effect if the effect is truly zero". Or am I guilty of oversimplification? – PrettyHands Jun 22 '18 at 15:43
• Again you're thinking clearly. I failed to distinguish a type I error and the probability (often incorrectly called a "rate") of a type I error. I'll try to edit the post in a bit. – Frank Harrell Jun 22 '18 at 16:31
• Just a quick prod as I'd love to be able to accept your answer! – PrettyHands Jun 27 '18 at 12:49
• I made those edits a few days ago – Frank Harrell Jun 27 '18 at 16:17
• Apologies, I had assumed that the website would notify me of your edits. As a friend of mine says: "Never assume!". I see now where you edited the text. Thanks again for your invaluable help. – PrettyHands Jun 28 '18 at 9:39

The classic one produces " Accept or Reject" the HO: ( Null hypothesis) The type one error is committed when you Reject the null hypothesis instead of Accepting it

• type I error is the rejection of a true null hypothesis – Dr. Eldard Mukasa Jun 22 '18 at 12:50
• No, type I error is the probability of asserting a nonzero effect if the effect is in fact zero. But this does not deal with the original question and doesn't provide the deep meaning of 'rejection of H0'. – Frank Harrell Jun 22 '18 at 14:14
• Not sure what example is needed. – Frank Harrell Jun 22 '18 at 15:35
• I need an empirical example to support the argument – Dr. Eldard Mukasa Jun 22 '18 at 15:50
• I'm using definitions so no empirical data needed. – Frank Harrell Jun 22 '18 at 16:29