Never understood the concept of the p-value, it should be higher than 0.05

Question

In multiple places, p-value is defined as something that quantifies the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct. A p-value < 0.05 is usually considered in favour of the alternate hypothesis.
If I understand correctly 'more extreme' are results in favour of the null hypothesis, because then a high p-value is in favour of it.
My actual question is that, I feel a p-value of 0.05 is way too low to reject the null hypothesis. To have conclusive favour in support of the null hypothesis, it should be more than that.. maybe 0.7 or 0.9? Perhaps I don't understand the p-value well enough intuitively. Any help would be appreciated. Thanks.

Answer 1

'More extreme results are in favor of the null hypothesis' --> This is not true!

The point of the p-value is to quantify how extreme your observation is if you assume the null hypothesis to be true. Assume you have a null hypothesis that states that some mean is equal to 3 and the alternative hypothesis states it is larger than 3. You now collect data and build a test statistic and calculate a p-value. If you have a mean in your sample that is close to 3, the p-value is going to say that it is quite normal to find a sample that has a mean that is at least as extreme as the one you have. And thus the null hypothesis seems plausible (note: you NEVER accept it!).

However, if you have a sample with mean equal to 5, the p-value will be very low since, assuming the true mean is 3 (your null hypothesis), it will be very unlikely to find a sample with a mean at least equal to 5. A low p-value (let's say 0.01) will then tell you: there is 1% that your results will occur when the null hypothesis is actually true. So this is good evidence to say that the null hypothesis might not be true at all (Note again: you also don't accept the alternative since you can never know 100% sure it is true, here it is still 1% possible that you found this extreme result due to luck and that the null hypothesis is still true).

I hope this clarifies that extreme results do not favor the null hypothesis since they are exactly the prove against it that you need.

If you would set the significance level equal to 0.9 as you say for example (so you reject the null hypothesis for all p-values lower than 0.9), you are going to make a lot of wrong conclusions since a p-value of 0.85 says that there is 85% probability to find a result like you did under the null hypothesis (so it is very common to find it and really gives no strong evidence against the null hypothesis), but since it is below your significance level of 0.9, you would still reject the null hypothesis based on this result and probably make a type I error. Note that also your significance level is in fact your type I error probability, so controlling it on 90% means that you still allow results that have 90% chance to be a type I error.

In conclusion: you really want a low p-value to avoid type I errors as much as possible.

Answer 2

You wrote:

If I understand correctly 'more extreme' are results in favour of the null hypothesis, because then a high p-value is in favour of it.

this is just the opposite of the case. More extreme result give a lower p. Let's demonstrate:

set.seed(1234)

x1 <- rnorm(100, 1, 5) #Random normal, mean 1, sd 5
x2 <- rnorm(100, 5, 5)  #Random normal, mean 5, sd 5

t.test(x1) #p = .67
t.test(x2) #p = 0.0000000000000002

Of course, p is also affected by sample size, and the effect size is affected not just by the difference from 0, but by the sd.

However, you are on to something when you complain about p < 0.05 being so generally accepted. If you lower the threshold for p, you decrease the type 1 error rate, but you increase the type 2 error rate. Often we accept power of 0.8, which is \beta of 0.2 implying that type 2 errors are four times worse than type 1 errors. This may be reasonable, or it may be nonsensical. Suppose, for example, that you have come up with a new treatment for a terminal disease that has no known treatments. Then a type 1 error means you give an ineffective treatment to someone who is dying, while a type 2 error means you fail to cure people.

Even Ronald Fisher once said that researchers should vary the threshold to suit the case, but, while I remember reading this, I cannot find the citation.

However, a threshold of 0.9 seems preposterous; it's hard to think of a case where that could be useful.

Answer 3

Sometimes it is easier to look at a visual representation to better understand the as extreme or more part of the definition of the p-value, or the p-value itself.

Suppose you repeat an experiment over and over again under the exact same conditions. You measure 10 samples and calculate the sample means, standard errors, and t-statistics for each experiment. The t-statistic in its simplest form is just the difference of the sample mean and some hypothesized parameter value (let's just for simplicity say it's zero) divided by the standard error of the mean of the respective sample:

$$t=\frac{\bar{x}-\mu_{0}}{s/\sqrt{n}}$$

Then you plot the t-statistics in a histogram or density plot (see purple line in the plot below). This distribution is also known as a sampling distribution, which fully characterizes your experiments with respect to the underlying unknown parameter. The sampling distribution will eventually converge to Student's t-distribution whose area under the curve will sum up to 1 (shaded blue and red areas in the plot below).

Now let's forget about the sampling distribution, which we never directly observe anyway, and instead focus on Student's t-distribution which serves as a very good approximation of the sampling distribution (provided the assumption of independent and random sampling has been satisfied). The shape of the distribution only depends on the degrees of freedom of your sample ($n-1$). Now you can ask for example which values will I observe on average 95% of the time given this experiment. This would be the blue-shaded area and all values that fall within this area would be in agreement with the null hypothesis that those values were generated from the same data generating process (experimental condition). But we still have to account for the remaining 5% under the curve. We can shove those into the tails, i.e. 2.5% left and 2.5% right, which are the red-shaded areas. Any values that would fall into those extremes (extreme in the sense of far away from the mean of the distribution), would not be in agreement with the null hypothesis and we would reject the null hypothesis, meaning that we believe that those values have likely originated from a different experimental population. This would be a significant result and the probability of observing these values is 0.05 or smaller than that. Remember areas under the curves represent probabilities, so if you fall into the red extremes, the area is small and hence the probability or p-value is small.

From that you can also see that the p-value is always a probability statment about the observed data and not a probability about the null or alternative hypothesis being true. A small p-value indicates that the observed data is unlikely (but not improbable) under the null hypothesis provided the null hypothesis is true.

Here's the code I used, just in case you were wondering. You can change those values and play around with it. Simulation is very helpful in understanding these concepts.

# required package
library(tidyverse)

# create a simulation function for 95% confidence intervals and p-values
simulation <- function(n, mu, stdev) {
  s <- rnorm(n, mu, stdev)
  tibble(
    N = length(s),
    sample_mean = mean(s),
    sample_sd = sd(s),
    sample_se = sample_sd / sqrt(N),
    confint_95 = sample_se * qt(0.975, N - 1),
    t_statistic = (sample_mean - mu) / sample_se,
    p_value = (1 - pt(abs(t_statistic), N - 1)) * 2
  )
}

# specify parameters and sample size for each draw or experiment
n <- 10 # sample size
mu <- 0 # population mean
stdev <- 5 # population standard deviation


############ SIMULATION ##############

# choose number of repetitions
sim <- 1:1000

# rerun experiment under exact same conditions
map(sim, ~ simulation(n, mu, stdev)) %>%
  bind_rows() %>%
  mutate(experiment_id = 1:length(sim)) -> draws

ggplot(data = draws_colored, aes(x = t_statistic)) + 
  geom_area(fun = dt, args = list(df = n-1), stat = "function", fill = "red") +
  geom_area(fun = dt, args = list(df = n-1), stat = "function", fill = "steelblue", xlim = c(qt(p = 0.025, df = n-1), qt(p = 0.975, df = n-1))) +
  stat_function(fun = dt, args = list(df = n-1), color = "steelblue", linewidth = 1) +
  geom_density(color = "purple", linewidth = 1.5) +
  scale_x_continuous(limits = c(-5,5)) +
  labs(x = "t-values", y = "Probability density", title = paste("t-distribution with",n-1,"degrees of freedom"))