# In an experiment, why can you select your own alpha level for significance testing?

From what I have learned, the alpha level tells you that if you repeat the experiment a bunch of times, then (alpha x 100)% of the time you will have a false positive. I'm confused though on if you're the one who gets to pick the alpha level, then how is this always the case? Like if my alpha is 0.05, 5% of the time I will have a false positive (FP), but then if I want to use alpha = 0.5, then 50% of the time I will get a FP. Why is this?

• You can also pick $\alpha = 100\%$, but then would anyone trust your significant results? :) Commented Feb 24, 2023 at 12:57

When you conduct a statistical test, basically it outputs a p-value that you do not choose.

This p-value ($$p$$) tells you that if the null hypothesis is true and the test assumptions are met, then you'd see a result at least as extreme as the one you're seeing $$p \times 100$$% of times, if you were to repeat the experiment a lot of times.

The alpha level ($$a$$) you choose before conducting the test is not the p-value that the test outputs. Your question seems to assume that $$p$$ and $$a$$ are the same, but they are two separate things.

Alpha is a threshold that you choose, and use to decide if the p-value returned by the test is significant or not. Ideally, this threshold should make sense and be relevant.

An alpha level that makes sense depends on the context of the study you're conducting. In particular, you should think about the consequences of choosing an alpha level that is too conservative or too liberal (e.g. would some people get hurt if you incorrectly reject the null hypothesis and then take some decision based on that? Would the study cost too much to conduct if you set a too small alpha level? etc.).

Here is a simple example that hopefully should make things clearer if your confusion came from the difference between $$p$$ and $$a$$:

1. You decide to use an alpha level of 0.05 before conducting your experiment and statistical analysis, based on your knowledge of the consequences that this alpha level would have in your specific situation.
2. Then, you conduct your experiment, and after that you conduct a statistical test on the data you collected.
3. The statistical test returns a p-value of 0.00001: the result is significant according to the threshold you pre-specified, as $$0.00001 < 0.05$$. Had the test returned a p-value of 0.3, the result would have not been significant according to the criteria you pre-specified, as $$0.3 > 0.05$$.

The premise of your question is not quite right. Alpha is the fraction of the time you'll have a false positive only if:

• The null hypothesis is assumed to be true. Usually, this means assuming (for the sake of argument while interpreting alpha or p) that the two populations have the same mean (or proportion, or survival curves...).
• All the assumptions of the analysis are correct. Often this means assuming the population follows a normal distribution and the deviation of each value from the mean is due to independent random reasons.

Your overall question needs clarifying. Alpha is a cutoff for making a conclusion, so it has to be set to a value. If not chosen by the experimenter, then by who?

Addendum Feb. 25, 2023: Maybe this will help. If you choose a large value for alpha, you'll have a larger chance of obtaining a false positive but a smaller chance of a false negative. If you choose a small value for alpha, you'll have a smaller chance of obtaining a false positive but a larger chance of false negative. It's a tradeoff. That is why there is a choice.

• Could be set by a regulator, reviewer, funder…
– Dave
Commented Feb 23, 2023 at 16:56
• @Dave "By a reviewer" assumes that $a$ would be set post-hoc (assuming we're talking about someone reviewing a paper submission, i.e. after the test has been conducted). Isn't it a problem, in particular relatively to publication bias? Genuine question. I'm wondering if I'm missing something here out of ignorance, or if you were just describing how things are (regardless of how they should or should not be). Commented Feb 23, 2023 at 18:05
• I meant in anticipation of what a reviewer might say to a paper claiming something like "significance at the $\alpha = 0.9$-level". // That could make for a good posted question, though I suspect we already have one.
– Dave
Commented Feb 23, 2023 at 18:07
• Oh yes I meant if you assume the null is true, and I was just wondering because sometime's in problems I've been able to select alpha myself, and other times it's given but I know when I can select alpha, I can choose any value like 0.05 or 0.1. So then I was wondering why do you get to randomly choose what value will be the fraction of false positives Commented Feb 23, 2023 at 19:59
• February 25, 2023?
– Dave
Commented Feb 28, 2023 at 2:32

UNDER THE NULL HYPOTHESIS, THE P-VALUE HAS A UNIFORM(0,1) DISTRIBUTION.

That's really all there is to it, but I believe that a simulation and a picture can help unpack that sentence.

UNDER THE NULL HYPOTHESIS means when the null hypothesis is totally, 100% true, along with all other assumptions (such as normality for a t-test). Consequently, what ever p-values we simulate should happen for tests of true null hypothesis. In this case, we will use a one-sample t-test to test if the mean is zero, simulating independent draws from a $$N(0,1)$$ distribution. Thus, all standard assumptions of the t-test are satisfied: independence, equal variance, and normality. The simulation repeats this $$10000$$ times to get an entire distribution of p-values.

library(ggplot2)
set.seed(2023)
N <- 1000
R <- 10000
p <- rep(NA, R)
for (i in 1:R){
x <- rnorm(N) # Simulate the distribution
p[i] <- t.test(x, mu = 0)\$p.value # Calculate the t-test p-value
}
d0 <- data.frame(
p_value = p,
CDF = ecdf(p)(p),
Distribution = "p-value Distribution"
)
d1 <- data.frame(
p_value = p,
CDF = qunif(p, 0, 1),
Distribution = "U(0,1) Distribution"
)
d <- rbind(d0, d1)
ggplot(d, aes(x = p_value, y = CDF, col = Distribution)) +
geom_line() +
theme(legend.position="bottom")


As you can see from the lines overlapping so nicely, when the null hypothesis is true and the other test assumptions are met, the p-values generated by a t-test do generate a $$U(0,1)$$ distribution.

What this means is that, if you reject the null hypothesis when $$p\le0.05$$, you will reject in $$5\%$$ of these cases where the null hypothesis and other test assumptions are true, since $$P(p\le0.05)=0.05$$. If you reject the null hypothesis when $$p=0.5$$, as is suggested in the OP, you will reject in $$50\%$$ of these cases where the null hypothesis and other test assumptions are true, since $$P(p\le0.5)=0.5$$.

In general, for a $$U(0,1)$$ distribution, if $$0\le\alpha\le 1$$, $$P(p\le\alpha)=\alpha$$. That's really the defining characteristic of the $$U(0,1)$$ distribution. In that sense, by picking the $$\alpha$$-level for the test, you pick your tolerance for rejecting when the null hypothesis is true.

However, note that rejecting in this case is a mistake. By the simulation, the null hypothesis is correct, and rejecting the null hypothesis is a so-called type I error. Errors are bad, and we like to minimize them. Consequently, it is typical to set $$\alpha$$ low, often around $$0.05$$ or lower, in order to keep from having a bunch of false positives.

You can take it it to the extreme to see why we don't want a high $$\alpha$$-level: if you set $$\alpha = 1$$, your test always rejects. Is a rejection by a statistical hypothesis test that always rejects, no matter what, any kind of evidence of a discovery? I say it is not. That is equivalent to the function below that makes absolutely no use of the data.

worthless.test <- function(input_data){
print("Reject the null hypothesis!")
}

• worthless.test reads like conspiracy_theory.test.
– Yakk
Commented Feb 24, 2023 at 15:21
• Commented May 1, 2023 at 22:23

Alpha is a decision threshold of how "sure" you need to be that an observed result is incompatible with the null before you conclude that the null is indeed false. For a typical alpha like 0.05, only 5% of the time will you reject the null when it's actually true - you'll get very few false positives, since you require pretty strong evidence to reject the null. For an uncommonly permissive alpha like 0.5, you'll incorrectly reject the null 50% of the time - you'll make many mistakes by rejecting the null since little evidence is required at that threshold.

Alpha = 0.05 is a typical threshold where most people agree that "statistically significant" is generally useful and not a meaningless statistical fluctuation. But each researcher is free to choose their alpha, which in a way represents how much evidence they require to be convinced - in other words, how skeptical they are. A high-alpha individual will spuriously reject many nulls but will rarely miss a true effect, while a low-alpha individual will find fewer true effects overall, but at a higher rate.

Imagine the Bigfoot enthusiast who accepts a grainy photo as sufficient proof of cryptozoology. He has a high alpha threshold, willing to reject the null with little evidence. He will not, however, ever be proven wrong by claiming that cryptids do not exist when they actually do. In contrast, a scientifically rigorous biologist may have a much lower alpha threshold, requiring more evidence to reject the default state of knowledge that there is no Bigfoot. The scientist's skepticism may lead him to disregard evidence of cryptids that truly do exist, but when the scientist claims that one exists, he's more often correct.

There is not one "optimal" alpha that is universally agreed upon for all scenarios. At the one end, a person who uses a very high alpha threshold regularly makes false findings based on little evidence, but at the other end, a person who uses a very low alpha regularly fails to acknowledge real effects because they deem even strong evidence insufficient. What alpha to use depends on your goals and how you perceive the relative costs of false positives versus false negatives.

Alpha represents a decision threshold for whether you deem an observation to be a result of random chance or a true effect. The question could be rephrased as, why am I allowed to decide for myself how much evidence will be convincing? Although there are common standards, it's a personal choice.

The optimal alpha value depends on many factors of the specific research area. It can't be fixed a priori by mathematics alone, and there's no universal consensus on what the "best" value is, nor can there be. Here's a recent paper in PLoS One that tries to grapple with this issue:

Miller J, Ulrich R. The quest for an optimal alpha. PLoS One. 2019 Jan 2;14(1):e0208631. doi: 10.1371/journal.pone.0208631. PMID: 30601826; PMCID: PMC6314595.

The abstract:

Researchers who analyze data within the framework of null hypothesis significance testing must choose a critical “alpha” level, α, to use as a cutoff for deciding whether a given set of data demonstrates the presence of a particular effect. In most fields, α = 0.05 has traditionally been used as the standard cutoff. Many researchers have recently argued for a change to a more stringent evidence cutoff such as α = 0.01, 0.005, or 0.001, noting that this change would tend to reduce the rate of false positives, which are of growing concern in many research areas. Other researchers oppose this proposed change, however, because it would correspondingly tend to increase the rate of false negatives. We show how a simple statistical model can be used to explore the quantitative tradeoff between reducing false positives and increasing false negatives. In particular, the model shows how the optimal α level depends on numerous characteristics of the research area, and it reveals that although α = 0.05 would indeed be approximately the optimal value in some realistic situations, the optimal α could actually be substantially larger or smaller in other situations. The importance of the model lies in making it clear what characteristics of the research area have to be specified to make a principled argument for using one α level rather than another, and the model thereby provides a blueprint for researchers seeking to justify a particular α level.

There are some excellent answers, but none discuss risk.

Often, stats are used for practical reasons. We can talk about the probability that two people in a particular group weigh within a kilogram of each other, or we can talk about the probability that a vaccine has a potentially deadly side effect. We may use similar tools, but it should be apparent that making a poor decision concerning the latter has ramifications above and beyond messing up on the former. If you tell the FDA that you tested the RSV vaccine for the side effect of Guillan Barre Syndrome at $$\alpha=0.05$$, the FDA may well turn your application down (as soon as they finish laughing).

Thus, we can choose alpha based upon how certain we need to be. In fact, when you present a reviewer with the results of your test and the alpha you used, you tell the reviewer what they need to know to make a decision about whether your statistical methods have an appropriate level of conservatism for a given situation.

When you do a statistical test (any kind of yes/no test, really), you can have two kinds of errors (wrong results): false positives (test says yes but real answer is no), and false negatives (test says no but real answer is yes).

The catch is that it's very easy to skew (non-technical term) your test one way or the other simply by making the test more likely to give one or the other answer - make a "yes" answer more likely, and then you get less false negatives, but more false positives, or make a "no" answer more likely, and then you get more false negatives, but less false positives.

So, how do you know that your test has a good yes/no balance? There's no "default" or "neutral" setting - there's nothing we can look at and definitively say "this is a fair test". Setting alpha=0.05 is setting your test's skew so that if the real answer is no, your test gives a false positive result 5% of the time. That seems fairly reasonable, so they do it. I don't think there's any reason why 5% is so common, rather than say 2% or 10%, other than gut feelings.

You only get to set one knob - all the other "skew parameters" are determined based on the one you control, and the quality of your data. You can't say you want 1% false positives and also 1% false negatives. If you want very few false positives, and your data is crap, you're going to get very few true positives as well.

Why do we choose to set the percentage of the time we get a false positive when the real answer is no, and not some other statistic, like the percentage of the time we get a false negative when the real answer is yes? Well that just happens to be something that's easy to control.