Questions tagged [p-value]
In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.
501 questions
302
votes
15
answers
556k
views
What is the meaning of p values and t values in statistical tests?
After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
- 4,454
170
votes
5
answers
104k
views
Why are p-values uniformly distributed under the null hypothesis?
Recently, I have found in a paper by Klammer, et al. a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand why it is so.
Klammer, A. A., Park, C. Y.,...
- 1,723
44
votes
3
answers
74k
views
Computing p-value using bootstrap with R
I use "boot" package to compute an approximated 2-sided bootstrapped p-value but the result is too far away from p-value of using t.test. I can't figure out what I did wrong in my R code. Can someone ...
- 2,977
97
votes
5
answers
36k
views
When to use Fisher versus Neyman-Pearson framework?
I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought.
My question is, ignoring philosophical objections, when ...
- 1,922
82
votes
4
answers
30k
views
How should tiny $p$-values be reported? (and why does R put a minimum on 2.22e-16?)
For some tests in R, there is a lower limit on the p-value calculations of $2.22 \cdot 10^{-16}$. I'm not sure why it's this number, if there is a good reason for ...
- 1,432
100
votes
11
answers
280k
views
How to obtain the p-value (check significance) of an effect in a lme4 mixed model?
I use lme4 in R to fit the mixed model
lmer(value~status+(1|experiment)))
where value is continuous, status and experiment are factors, and I get
...
- 2,201
24
votes
4
answers
11k
views
Misunderstanding a P-value?
So I've been reading a lot about how to correctly interpret a P-value, and from what I've read, the p-value says NOTHING about the probability that the null hypothesis is true or false. However, when ...
- 726
9
votes
2
answers
78k
views
What does the notation like 8.6e-28 mean? What is the 'e' for?
I have a problem with the interpretation of a test result in which the p-value is 8.6e-28. How should it be interpreted? What is the ...
- 121
68
votes
3
answers
136k
views
What is the effect of having correlated predictors in a multiple regression model?
I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
- 3,263
45
votes
4
answers
10k
views
Interpretation of p-value in hypothesis testing
I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
user13587
27
votes
4
answers
19k
views
Comparing and contrasting, p-values, significance levels and type I error
I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error.
I understand that p-values are defined as "the probability of ...
- 1,505
100
votes
2
answers
14k
views
How much do we know about p-hacking "in the wild"?
The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
- 23.9k
39
votes
2
answers
5k
views
Understanding p-value
I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification.
Here is the definition of p-value from Wikipedia:
The p-...
- 501
87
votes
10
answers
25k
views
Regarding p-values, why 1% and 5%? Why not 6% or 10%?
Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%?
Is ...
- 1,499
16
votes
5
answers
2k
views
Does p-value ever depend on the alternative?
Our tag definition of the $p$-value says
In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the ...
- 69.5k
117
votes
10
answers
8k
views
ASA discusses limitations of $p$-values - what are the alternatives?
We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
- 141k
68
votes
5
answers
6k
views
Why does collecting data until finding a significant result increase Type I error rate?
I was wondering exactly why collecting data until a significant result (e.g., $p \lt .05$) is obtained (i.e., p-hacking) increases the Type I error rate?
I would also highly appreciate an ...
- 976
51
votes
7
answers
7k
views
Why is "statistically significant" not enough?
I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is ...
- 621
32
votes
2
answers
3k
views
Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?
While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed.
Thus the question: are they formally equivalent? I....
- 1,369
65
votes
3
answers
30k
views
Explain the xkcd jelly bean comic: What makes it funny?
I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$).
xkcd jelly bean ...
- 713
60
votes
3
answers
30k
views
When combining p-values, why not just averaging?
I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
- 2,253
12
votes
1
answer
751
views
Can we think of a probability in both the classical and subjective sense simultaneously?
I'm a statistics student. I am trying to understand the classical and objective definitions of probability and how they are related to frequentist and Bayesian inference. It's not obvious to me why ...
- 364
73
votes
6
answers
5k
views
Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?
There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-...
- 107k
13
votes
2
answers
4k
views
Equivalence of AIC and p-values in model selection
In a comment to the answer of this question, it was stated that using AIC in model selection was equivalent to using a p-value of 0.154.
I tried it in R, where I used a "backward" subset selection ...
- 233
12
votes
1
answer
129k
views
"Accept null hypothesis" or "fail to reject the null hypothesis"? [duplicate]
I'm trying to conduct a Student's t-test for a table of values while trying to follow the explanation and details found on this website. I understand that if the p-value is
<.01 then it's really ...
- 223
52
votes
4
answers
2k
views
Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?
I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
- 107k
40
votes
6
answers
3k
views
Effect size as the hypothesis for significance testing
Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked:
Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
- 5,065
17
votes
1
answer
2k
views
Non-uniform distribution of p-values when simulating binomial tests under the null hypothesis
I heard that under the null hypothesis the p-value distribution should be uniform. However, simulations of binomial test in MATLAB return very different-from-uniform distributions with mean larger ...
- 425
13
votes
5
answers
737
views
$p$-value: Fisherian vs. contemporary frequentist definitions
I am trying to see if I understand the definition of $p$-value as used by Sir Ronald A. Fisher and the one used today by frequentist statisticians (not sure how to call it better).
$p$-value according ...
- 69.5k
9
votes
3
answers
835
views
How many p-value observations do you think are required before doing FDR correction
I have two sets of correlation data and my variable of interest and I am wondering if/when o apply FDR to the obtained P-Values.
On one hand, I have correlations with gene expression data, so a great ...
- 161
47
votes
4
answers
18k
views
Are smaller p-values more convincing?
I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
- 1,876
28
votes
2
answers
11k
views
P-value in a two-tail test with asymmetric null distribution
My situation is as follows: I want, through a Monte-Carlo study, to compare $p$-values of two different tests for statistical significance of an estimated parameter (null is "no effect - parameter is ...
- 60.8k
11
votes
3
answers
3k
views
Why is mean ± 2*SEM (95% confidence interval) overlapping, but the p-value is 0.05?
I have data as two lists:
...
- 10.2k
41
votes
4
answers
5k
views
Origin of "5$\sigma$" threshold for accepting evidence in particle physics?
News reports say that CERN will announce tomorrow that the Higgs boson has been experimentally detected with 5$\sigma$ evidence. According to that article:
5$\sigma$ equates to a 99.99994% chance ...
- 21.2k
33
votes
3
answers
1k
views
Accommodating entrenched views of p-values
Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided. I say that since the sample wasn't random, then such statistics would not strictly apply....
- 12.9k
24
votes
1
answer
60k
views
Sample size too large? [duplicate]
I always thought larger sample sizes were better.
Then I read something somewhere about how when sample sizes are larger, it's easier to find significant p-values when they're not really there (i.e., ...
17
votes
2
answers
7k
views
Explaining two-tailed tests
I am looking for various ways of explaining to my students (in an elementary statistics course) what is a two tailed test, and how its P value is calculated.
How do you explain to your students the ...
- 21.9k
14
votes
2
answers
45k
views
Stepwise regression in R – Critical p-value
What is the critical p-value used by the step() function in R for stepwise regression? I assume it is 0.15, but is my assumption correct? How can I change the ...
- 1,091
6
votes
3
answers
644
views
p value correction in multiple outcomes study
I am performing an analysis on different values of complement (in total 18, 9 in blood and 9 in CSF). We aim to analyze if any of these complement values:
Differentiate between 3 diseases (Disease 1, ...
6
votes
1
answer
2k
views
High p-values for logistic regression variable that perfectly separates?
I'm using R to run some logistic regression. My variables were continuous, but I used cut to bucket the data. Some particular buckets for these variables always result in dependent variable being ...
- 771
32
votes
3
answers
7k
views
Is the exact value of a 'p-value' meaningless?
I had a discussion with a statistician back in 2009 where he stated that the exact value of a p-value is irrelevant: the only thing that is important is whether it is significant or not. I.e. one ...
- 539
29
votes
2
answers
29k
views
What's the formula for the Benjamini-Hochberg adjusted p-value?
I understand the procedure and what it controls. So what's the formula for the adjusted p-value in the BH procedure for multiple comparisons?
Just now I realized the original BH didn't produce ...
- 19.5k
12
votes
3
answers
11k
views
G-test vs Pearson's $\chi^2$ test
I'm testing independence in an $N \times M$ contingency table. I don't know whether the G-test or Pearson's $\chi^2$ test is better. The sample size is in the hundreds but there are some low cell ...
- 121
9
votes
2
answers
4k
views
Why are the number of false positives independent of sample size, if we use p-values to compare two independent datasets?
If I run the R code below, it will generate two independent vectors then test them to see if they are related in some way (i.e. p-value < 0.05).
If I repeat this 1,000 times, then 50 of these (5%) ...
- 1,499
38
votes
4
answers
4k
views
Why are lower p-values not more evidence against the null? Arguments from Johansson 2011
Johansson (2011) in "Hail the impossible: p-values, evidence, and likelihood" (here is also link to the journal) states that lower $p$-values are often considered as stronger evidence against the null....
- 14.6k
15
votes
2
answers
21k
views
LASSO Regression - p-values and coefficients
I've run a LASSO in R using cv.glmnet. I would like to generate p-values for the coefficients that are selected.
I found the boot.lass.proj to produce bootstrapped ...
- 275
13
votes
2
answers
17k
views
Ridge regression in R with p values and goodness of fit [closed]
Doing ridge regression in R I have discovered
linearRidge in the ridge package - which fits a model, reports coefficients and p ...
- 1,485
8
votes
4
answers
874
views
Defining extremeness of test statistic and defining $p$-value for a two-sided test
Our tag definition of the $p$-value says
In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the ...
- 69.5k
2
votes
1
answer
9k
views
Explaining p-value to a sophisticated layman [duplicate]
I think I understand the concept of p-value but unfortunately I still have to exert a lot of brain cycles to get my arms around it.
I would like to get an explanation of the p-value that is rigorous ...
- 2,055
69
votes
3
answers
6k
views
References containing arguments against null hypothesis significance testing?
In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...