# Questions tagged [p-value]

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

496 questions
Filter by
Sorted by
Tagged with
554k views

### What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
• 4,414
101k views

Recently, I have found in a paper by Klammer, et al. a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand why it is so. Klammer, A. A., Park, C. Y.,...
• 1,703
72k views

### Computing p-value using bootstrap with R

I use "boot" package to compute an approximated 2-sided bootstrapped p-value but the result is too far away from p-value of using t.test. I can't figure out what I did wrong in my R code. Can someone ...
• 2,947
30k views

### How should tiny $p$-values be reported? (and why does R put a minimum on 2.22e-16?)

For some tests in R, there is a lower limit on the p-value calculations of $2.22 \cdot 10^{-16}$. I'm not sure why it's this number, if there is a good reason for ...
• 1,422
36k views

### When to use Fisher versus Neyman-Pearson framework?

I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought. My question is, ignoring philosophical objections, when ...
• 1,892
273k views

### How to obtain the p-value (check significance) of an effect in a lme4 mixed model?

I use lme4 in R to fit the mixed model lmer(value~status+(1|experiment))) where value is continuous, status and experiment are factors, and I get ...
• 2,201
11k views

### Misunderstanding a P-value?

So I've been reading a lot about how to correctly interpret a P-value, and from what I've read, the p-value says NOTHING about the probability that the null hypothesis is true or false. However, when ...
• 716
76k views

### What does the notation like 8.6e-28 mean? What is the 'e' for?

I have a problem with the interpretation of a test result in which the p-value is 8.6e-28. How should it be interpreted? What is the ...
134k views

### What is the effect of having correlated predictors in a multiple regression model?

I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
• 3,233
10k views

### Interpretation of p-value in hypothesis testing

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
19k views

### Comparing and contrasting, p-values, significance levels and type I error

I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error. I understand that p-values are defined as "the probability of ...
• 1,505
14k views

### How much do we know about p-hacking "in the wild"?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
• 23.7k
5k views

### Understanding p-value

I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification. Here is the definition of p-value from Wikipedia: The p-...
• 501
24k views

### Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%? Is ...
• 1,489
8k views

### ASA discusses limitations of $p$-values - what are the alternatives?

We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
• 140k
7k views

### Why is "statistically significant" not enough?

I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is ...
• 621
5k views

### Why does collecting data until finding a significant result increase Type I error rate?

I was wondering exactly why collecting data until a significant result (e.g., $p \lt .05$) is obtained (i.e., p-hacking) increases the Type I error rate? I would also highly appreciate an ...
• 936
3k views

### Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?

While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed. Thus the question: are they formally equivalent? I....
• 1,359
2k views

### Does p-value ever depend on the alternative?

Our tag definition of the $p$-value says In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the ...
• 68.5k
30k views

### Explain the xkcd jelly bean comic: What makes it funny?

I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$). xkcd jelly bean ...
• 713
696 views

### Can we think of a probability in both the classical and subjective sense simultaneously?

I'm a statistics student. I am trying to understand the classical and objective definitions of probability and how they are related to frequentist and Bayesian inference. It's not obvious to me why ...
• 364
4k views

### Equivalence of AIC and p-values in model selection

In a comment to the answer of this question, it was stated that using AIC in model selection was equivalent to using a p-value of 0.154. I tried it in R, where I used a "backward" subset selection ...
• 233
129k views

### "Accept null hypothesis" or "fail to reject the null hypothesis"? [duplicate]

I'm trying to conduct a Student's t-test for a table of values while trying to follow the explanation and details found on this website. I understand that if the p-value is <.01 then it's really ...
• 223
4k views

### Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
• 106k
29k views

### When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that -2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
• 2,243
2k views

### Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?

I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
• 106k
3k views

### Effect size as the hypothesis for significance testing

Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked: Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
• 5,025
2k views

### Non-uniform distribution of p-values when simulating binomial tests under the null hypothesis

I heard that under the null hypothesis the p-value distribution should be uniform. However, simulations of binomial test in MATLAB return very different-from-uniform distributions with mean larger ...
• 425
17k views

### Are smaller p-values more convincing?

I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
• 1,846
11k views

### P-value in a two-tail test with asymmetric null distribution

My situation is as follows: I want, through a Monte-Carlo study, to compare $p$-values of two different tests for statistical significance of an estimated parameter (null is "no effect - parameter is ...
• 59.7k
3k views

### Why is mean ± 2*SEM (95% confidence interval) overlapping, but the p-value is 0.05?

I have data as two lists: ...
• 10.1k
5k views

### Origin of "5$\sigma$" threshold for accepting evidence in particle physics?

News reports say that CERN will announce tomorrow that the Higgs boson has been experimentally detected with 5$\sigma$ evidence. According to that article: 5$\sigma$ equates to a 99.99994% chance ...
• 20.9k
1k views

### Accommodating entrenched views of p-values

Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided. I say that since the sample wasn't random, then such statistics would not strictly apply....
• 12.8k
7k views

### Explaining two-tailed tests

I am looking for various ways of explaining to my students (in an elementary statistics course) what is a two tailed test, and how its P value is calculated. How do you explain to your students the ...
• 21.8k
45k views

### Stepwise regression in R – Critical p-value

What is the critical p-value used by the step() function in R for stepwise regression? I assume it is 0.15, but is my assumption correct? How can I change the ...
• 1,091
2k views

### High p-values for logistic regression variable that perfectly separates?

I'm using R to run some logistic regression. My variables were continuous, but I used cut to bucket the data. Some particular buckets for these variables always result in dependent variable being ...
• 771
7k views

### Is the exact value of a 'p-value' meaningless?

I had a discussion with a statistician back in 2009 where he stated that the exact value of a p-value is irrelevant: the only thing that is important is whether it is significant or not. I.e. one ...
28k views

### What's the formula for the Benjamini-Hochberg adjusted p-value?

I understand the procedure and what it controls. So what's the formula for the adjusted p-value in the BH procedure for multiple comparisons? Just now I realized the original BH didn't produce ...
• 19.3k
59k views

### Sample size too large? [duplicate]

I always thought larger sample sizes were better. Then I read something somewhere about how when sample sizes are larger, it's easier to find significant p-values when they're not really there (i.e., ...
11k views

### G-test vs Pearson's $\chi^2$ test

I'm testing independence in an $N \times M$ contingency table. I don't know whether the G-test or Pearson's $\chi^2$ test is better. The sample size is in the hundreds but there are some low cell ...
• 121
633 views

### $p$-value: Fisherian vs. contemporary frequentist definitions

I am trying to see if I understand the definition of $p$-value as used by Sir Ronald A. Fisher and the one used today by frequentist statisticians (not sure how to call it better). $p$-value according ...
• 68.5k
4k views

### Why are the number of false positives independent of sample size, if we use p-values to compare two independent datasets?

If I run the R code below, it will generate two independent vectors then test them to see if they are related in some way (i.e. p-value < 0.05). If I repeat this 1,000 times, then 50 of these (5%) ...
• 1,489
4k views

### Why are lower p-values not more evidence against the null? Arguments from Johansson 2011

Johansson (2011) in "Hail the impossible: p-values, evidence, and likelihood" (here is also link to the journal) states that lower $p$-values are often considered as stronger evidence against the null....
• 14.4k
20k views

### LASSO Regression - p-values and coefficients

I've run a LASSO in R using cv.glmnet. I would like to generate p-values for the coefficients that are selected. I found the boot.lass.proj to produce bootstrapped ...
• 265
17k views

### Ridge regression in R with p values and goodness of fit [closed]

Doing ridge regression in R I have discovered linearRidge in the ridge package - which fits a model, reports coefficients and p ...
• 1,485
833 views

### Defining extremeness of test statistic and defining $p$-value for a two-sided test

Our tag definition of the $p$-value says In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the ...
• 68.5k
9k views

### Explaining p-value to a sophisticated layman [duplicate]

I think I understand the concept of p-value but unfortunately I still have to exert a lot of brain cycles to get my arms around it. I would like to get an explanation of the p-value that is rigorous ...
• 2,045
6k views

### References containing arguments against null hypothesis significance testing?

In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...