In an experiment, why can you select your own alpha level for significance testing?

Question

From what I have learned, the alpha level tells you that if you repeat the experiment a bunch of times, then (alpha x 100)% of the time you will have a false positive. I'm confused though on if you're the one who gets to pick the alpha level, then how is this always the case? Like if my alpha is 0.05, 5% of the time I will have a false positive (FP), but then if I want to use alpha = 0.5, then 50% of the time I will get a FP. Why is this?

Answer 1

The premise of your question is not quite right. Alpha is the fraction of the time you'll have a false positive only if:

• The null hypothesis is assumed to be true. Usually, this means assuming (for the sake of argument while interpreting alpha or p) that the two populations have the same mean (or proportion, or survival curves...).
• All the assumptions of the analysis are correct. Often this means assuming the population follows a normal distribution and the deviation of each value from the mean is due to independent random reasons.

Your overall question needs clarifying. Alpha is a cutoff for making a conclusion, so it has to be set to a value. If not chosen by the experimenter, then by who?

Addendum Feb. 25, 2023: Maybe this will help. If you choose a large value for alpha, you'll have a larger chance of obtaining a false positive but a smaller chance of a false negative. If you choose a small value for alpha, you'll have a smaller chance of obtaining a false positive but a larger chance of false negative. It's a tradeoff. That is why there is a choice.

Answer 2

UNDER THE NULL HYPOTHESIS, THE P-VALUE HAS A UNIFORM(0,1) DISTRIBUTION.

That's really all there is to it, but I believe that a simulation and a picture can help unpack that sentence.

UNDER THE NULL HYPOTHESIS means when the null hypothesis is totally, 100% true, along with all other assumptions (such as normality for a t-test). Consequently, what ever p-values we simulate should happen for tests of true null hypothesis. In this case, we will use a one-sample t-test to test if the mean is zero, simulating independent draws from a $N(0,1)$ distribution. Thus, all standard assumptions of the t-test are satisfied: independence, equal variance, and normality. The simulation repeats this $10000$ times to get an entire distribution of p-values.

library(ggplot2)
set.seed(2023)
N <- 1000
R <- 10000
p <- rep(NA, R)
for (i in 1:R){
  x <- rnorm(N) # Simulate the distribution
  p[i] <- t.test(x, mu = 0)$p.value # Calculate the t-test p-value
}
d0 <- data.frame(
  p_value = p,
  CDF = ecdf(p)(p),
  Distribution = "p-value Distribution"
)
d1 <- data.frame(
  p_value = p,
  CDF = qunif(p, 0, 1),
  Distribution = "U(0,1) Distribution"
)
d <- rbind(d0, d1)
ggplot(d, aes(x = p_value, y = CDF, col = Distribution)) +
  geom_line() +
  theme(legend.position="bottom")

As you can see from the lines overlapping so nicely, when the null hypothesis is true and the other test assumptions are met, the p-values generated by a t-test do generate a $U(0,1)$ distribution.

What this means is that, if you reject the null hypothesis when $p\le0.05$, you will reject in $5\%$ of these cases where the null hypothesis and other test assumptions are true, since $P(p\le0.05)=0.05$. If you reject the null hypothesis when $p=0.5$, as is suggested in the OP, you will reject in $50\%$ of these cases where the null hypothesis and other test assumptions are true, since $P(p\le0.5)=0.5$.

In general, for a $U(0,1)$ distribution, if $0\le\alpha\le 1$, $P(p\le\alpha)=\alpha$. That's really the defining characteristic of the $U(0,1)$ distribution. In that sense, by picking the $\alpha$-level for the test, you pick your tolerance for rejecting when the null hypothesis is true.

However, note that rejecting in this case is a mistake. By the simulation, the null hypothesis is correct, and rejecting the null hypothesis is a so-called type I error. Errors are bad, and we like to minimize them. Consequently, it is typical to set $\alpha$ low, often around $0.05$ or lower, in order to keep from having a bunch of false positives.

You can take it it to the extreme to see why we don't want a high $\alpha$-level: if you set $\alpha = 1$, your test always rejects. Is a rejection by a statistical hypothesis test that always rejects, no matter what, any kind of evidence of a discovery? I say it is not. That is equivalent to the function below that makes absolutely no use of the data.

worthless.test <- function(input_data){
    print("Reject the null hypothesis!")
}

Answer 3

Alpha is a decision threshold of how "sure" you need to be that an observed result is incompatible with the null before you conclude that the null is indeed false. For a typical alpha like 0.05, only 5% of the time will you reject the null when it's actually true - you'll get very few false positives, since you require pretty strong evidence to reject the null. For an uncommonly permissive alpha like 0.5, you'll incorrectly reject the null 50% of the time - you'll make many mistakes by rejecting the null since little evidence is required at that threshold.

Alpha = 0.05 is a typical threshold where most people agree that "statistically significant" is generally useful and not a meaningless statistical fluctuation. But each researcher is free to choose their alpha, which in a way represents how much evidence they require to be convinced - in other words, how skeptical they are. A high-alpha individual will spuriously reject many nulls but will rarely miss a true effect, while a low-alpha individual will find fewer true effects overall, but at a higher rate.

Imagine the Bigfoot enthusiast who accepts a grainy photo as sufficient proof of cryptozoology. He has a high alpha threshold, willing to reject the null with little evidence. He will not, however, ever be proven wrong by claiming that cryptids do not exist when they actually do. In contrast, a scientifically rigorous biologist may have a much lower alpha threshold, requiring more evidence to reject the default state of knowledge that there is no Bigfoot. The scientist's skepticism may lead him to disregard evidence of cryptids that truly do exist, but when the scientist claims that one exists, he's more often correct.

There is not one "optimal" alpha that is universally agreed upon for all scenarios. At the one end, a person who uses a very high alpha threshold regularly makes false findings based on little evidence, but at the other end, a person who uses a very low alpha regularly fails to acknowledge real effects because they deem even strong evidence insufficient. What alpha to use depends on your goals and how you perceive the relative costs of false positives versus false negatives.

Alpha represents a decision threshold for whether you deem an observation to be a result of random chance or a true effect. The question could be rephrased as, why am I allowed to decide for myself how much evidence will be convincing? Although there are common standards, it's a personal choice.

Answer 4

The optimal alpha value depends on many factors of the specific research area. It can't be fixed a priori by mathematics alone, and there's no universal consensus on what the "best" value is, nor can there be. Here's a recent paper in PLoS One that tries to grapple with this issue:

Miller J, Ulrich R. The quest for an optimal alpha. PLoS One. 2019 Jan 2;14(1):e0208631. doi: 10.1371/journal.pone.0208631. PMID: 30601826; PMCID: PMC6314595.

The abstract:

Researchers who analyze data within the framework of null hypothesis significance testing must choose a critical “alpha” level, α, to use as a cutoff for deciding whether a given set of data demonstrates the presence of a particular effect. In most fields, α = 0.05 has traditionally been used as the standard cutoff. Many researchers have recently argued for a change to a more stringent evidence cutoff such as α = 0.01, 0.005, or 0.001, noting that this change would tend to reduce the rate of false positives, which are of growing concern in many research areas. Other researchers oppose this proposed change, however, because it would correspondingly tend to increase the rate of false negatives. We show how a simple statistical model can be used to explore the quantitative tradeoff between reducing false positives and increasing false negatives. In particular, the model shows how the optimal α level depends on numerous characteristics of the research area, and it reveals that although α = 0.05 would indeed be approximately the optimal value in some realistic situations, the optimal α could actually be substantially larger or smaller in other situations. The importance of the model lies in making it clear what characteristics of the research area have to be specified to make a principled argument for using one α level rather than another, and the model thereby provides a blueprint for researchers seeking to justify a particular α level.

Link to the article.

Answer 5

There are some excellent answers, but none discuss risk.

Often, stats are used for practical reasons. We can talk about the probability that two people in a particular group weigh within a kilogram of each other, or we can talk about the probability that a vaccine has a potentially deadly side effect. We may use similar tools, but it should be apparent that making a poor decision concerning the latter has ramifications above and beyond messing up on the former. If you tell the FDA that you tested the RSV vaccine for the side effect of Guillan Barre Syndrome at $\alpha=0.05$, the FDA may well turn your application down (as soon as they finish laughing).

Thus, we can choose alpha based upon how certain we need to be. In fact, when you present a reviewer with the results of your test and the alpha you used, you tell the reviewer what they need to know to make a decision about whether your statistical methods have an appropriate level of conservatism for a given situation.

Answer 6

When you do a statistical test (any kind of yes/no test, really), you can have two kinds of errors (wrong results): false positives (test says yes but real answer is no), and false negatives (test says no but real answer is yes).

The catch is that it's very easy to skew (non-technical term) your test one way or the other simply by making the test more likely to give one or the other answer - make a "yes" answer more likely, and then you get less false negatives, but more false positives, or make a "no" answer more likely, and then you get more false negatives, but less false positives.

So, how do you know that your test has a good yes/no balance? There's no "default" or "neutral" setting - there's nothing we can look at and definitively say "this is a fair test". Setting alpha=0.05 is setting your test's skew so that if the real answer is no, your test gives a false positive result 5% of the time. That seems fairly reasonable, so they do it. I don't think there's any reason why 5% is so common, rather than say 2% or 10%, other than gut feelings.

You only get to set one knob - all the other "skew parameters" are determined based on the one you control, and the quality of your data. You can't say you want 1% false positives and also 1% false negatives. If you want very few false positives, and your data is crap, you're going to get very few true positives as well.

Why do we choose to set the percentage of the time we get a false positive when the real answer is no, and not some other statistic, like the percentage of the time we get a false negative when the real answer is yes? Well that just happens to be something that's easy to control.