Questions tagged [p-value]
In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.
2,950
questions
300
votes
15
answers
555k
views
What is the meaning of p values and t values in statistical tests?
After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
- 4,424
169
votes
5
answers
102k
views
Why are p-values uniformly distributed under the null hypothesis?
Recently, I have found in a paper by Klammer, et al. a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand why it is so.
Klammer, A. A., Park, C. Y.,...
- 1,713
116
votes
10
answers
8k
views
ASA discusses limitations of $p$-values - what are the alternatives?
We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
- 140k
104
votes
9
answers
15k
views
Is this really how p-values work? Can a million research papers per year be based on pure randomness?
I'm very new to statistics, and I'm just learning to understand the basics, including $p$-values. But there is a huge question mark in my mind right now, and I kind of hope my understanding is wrong. ...
- 1,101
100
votes
11
answers
275k
views
How to obtain the p-value (check significance) of an effect in a lme4 mixed model?
I use lme4 in R to fit the mixed model
lmer(value~status+(1|experiment)))
where value is continuous, status and experiment are factors, and I get
...
- 2,201
100
votes
2
answers
14k
views
How much do we know about p-hacking "in the wild"?
The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
- 23.8k
96
votes
5
answers
36k
views
When to use Fisher versus Neyman-Pearson framework?
I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought.
My question is, ignoring philosophical objections, when ...
- 1,892
86
votes
10
answers
24k
views
Regarding p-values, why 1% and 5%? Why not 6% or 10%?
Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%?
Is ...
- 1,489
82
votes
4
answers
30k
views
How should tiny $p$-values be reported? (and why does R put a minimum on 2.22e-16?)
For some tests in R, there is a lower limit on the p-value calculations of $2.22 \cdot 10^{-16}$. I'm not sure why it's this number, if there is a good reason for ...
- 1,432
75
votes
4
answers
6k
views
A psychology journal banned p-values and confidence intervals; is it indeed wise to stop using them?
On 25 February 2015, the journal Basic and Applied Social Psychology issued an editorial banning $p$-values and confidence intervals from all future papers.
Specifically, they say (formatting and ...
73
votes
3
answers
4k
views
Is this the solution to the p-value problem?
In February 2016, the American Statistical Association released a formal statement on statistical significance and p-values. Our thread about it discusses these issues extensively. However, no ...
70
votes
6
answers
4k
views
Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?
There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
- 106k
69
votes
3
answers
6k
views
References containing arguments against null hypothesis significance testing?
In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
68
votes
5
answers
5k
views
Why does collecting data until finding a significant result increase Type I error rate?
I was wondering exactly why collecting data until a significant result (e.g., $p \lt .05$) is obtained (i.e., p-hacking) increases the Type I error rate?
I would also highly appreciate an ...
- 976
68
votes
8
answers
8k
views
What is a good, convincing example in which p-values are useful?
My question in the title is self explanatory, but I would like to give it some context.
The ASA released a statement earlier this week “on p-values: context, process, and purpose”, outlining various ...
- 21.8k
68
votes
3
answers
135k
views
What is the effect of having correlated predictors in a multiple regression model?
I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
- 3,243
65
votes
3
answers
30k
views
Explain the xkcd jelly bean comic: What makes it funny?
I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$).
xkcd jelly bean ...
- 713
62
votes
10
answers
11k
views
What does "Scientists rise up against statistical significance" mean? (Comment in Nature)
The title of the Comment in Nature Scientists rise up against statistical significance begins with:
Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to ...
- 695
61
votes
2
answers
76k
views
A/B tests: z-test vs t-test vs chi square vs fisher exact test
I'm trying to understand the reasoning by choosing a specific test approach when dealing with a simple A/B test - (i.e. two variations/groups with a binary respone (converted or not). As an example I ...
- 1,239
60
votes
3
answers
29k
views
When combining p-values, why not just averaging?
I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
- 2,243
59
votes
4
answers
155k
views
Manually Calculating P value from t-value in t-test
I have a sample dataset with 31 values. I ran a two-tailed t-test using R to test if the true mean is equal to 10:
t.test(x=data, mu=10, conf.level=0.95)
Output:
...
- 693
56
votes
5
answers
44k
views
Correct spelling (capitalization, italicization, hyphenation) of "p-value"?
I realize this is pedantic and trite, but as a researcher in a field outside of statistics, with limited formal education in statistics, I always wonder if I'm writing "p-value" correctly. ...
- 953
52
votes
4
answers
2k
views
Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?
I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
- 106k
51
votes
7
answers
7k
views
Why is "statistically significant" not enough?
I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is ...
- 621
48
votes
10
answers
7k
views
Why do people use p-values instead of computing probability of the model given data?
Roughly speaking a p-value gives a probability of the observed outcome of an experiment given the hypothesis (model). Having this probability (p-value) we want to judge our hypothesis (how likely it ...
- 664
46
votes
4
answers
17k
views
Are smaller p-values more convincing?
I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
- 1,856
45
votes
4
answers
10k
views
Interpretation of p-value in hypothesis testing
I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
user13587
44
votes
3
answers
73k
views
Computing p-value using bootstrap with R
I use "boot" package to compute an approximated 2-sided bootstrapped p-value but the result is too far away from p-value of using t.test. I can't figure out what I did wrong in my R code. Can someone ...
- 2,947
43
votes
2
answers
2k
views
Evidence for man-made global warming hits 'gold standard': how did they do this?
This message in a Reuter's article from 25.02.2019 is currently all over the news:
Evidence for man-made global warming hits 'gold standard'
[Scientists] said confidence that human activities ...
- 82.8k
41
votes
4
answers
5k
views
Origin of "5$\sigma$" threshold for accepting evidence in particle physics?
News reports say that CERN will announce tomorrow that the Higgs boson has been experimentally detected with 5$\sigma$ evidence. According to that article:
5$\sigma$ equates to a 99.99994% chance ...
- 20.9k
40
votes
6
answers
3k
views
Effect size as the hypothesis for significance testing
Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked:
Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
- 5,045
39
votes
2
answers
5k
views
Understanding p-value
I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification.
Here is the definition of p-value from Wikipedia:
The p-...
- 501
38
votes
9
answers
4k
views
Why do we use hypothesis tests instead of just letting people do Bayesian updates?
Why do we need discretize our judgements using hypothesis tests?
Why can't we just have people report the data every time a study is done, and the p-values and effect size, and then report how the ...
- 491
37
votes
5
answers
8k
views
Is p-value essentially useless and dangerous to use?
This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that
[Bayesian statistics] is proving especially useful in approaching complex ...
- 2,372
37
votes
4
answers
4k
views
Why are lower p-values not more evidence against the null? Arguments from Johansson 2011
Johansson (2011) in "Hail the impossible: p-values, evidence, and likelihood" (here is also link to the journal) states that lower $p$-values are often considered as stronger evidence against the null....
- 14.5k
36
votes
2
answers
2k
views
Performing a statistical test after visualizing data - data dredging?
I'll propose this question by means of an example.
Suppose I have a data set, such as the boston housing price data set, in which I have continuous and categorical variables. Here, we have a "quality"...
- 1,410
35
votes
7
answers
3k
views
Why is it bad to teach students that p-values are the probability that findings are due to chance?
Can someone please offer a nice succinct explanation why it is not a good idea to teach students that a p-value is the prob(their findings are due to [random] chance). My understanding is that a p-...
- 1,581
34
votes
3
answers
137k
views
How to compute a p-value for the Pearson correlation in R?
Is it possible to find the p-value in pearson correlation in R?
To find the pearson correlation, I usually do this
...
- 623
34
votes
3
answers
7k
views
Is p-value a point estimate?
Since one can calculate confidence intervals for p-values and since the opposite of interval estimation is point estimation: Is p-value a point estimate?
- 1,342
33
votes
4
answers
41k
views
Why do linear regression and ANOVA give different $p$-value in case of considering interaction between variable?
I was trying to fit one time-series data (without replicates) using regression model.
The data looks like follows:
...
- 667
33
votes
3
answers
1k
views
Accommodating entrenched views of p-values
Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided. I say that since the sample wasn't random, then such statistics would not strictly apply....
- 12.8k
33
votes
2
answers
6k
views
Who first used/invented p-values?
I am attempting to write a series of blog posts on p-values and I thought it would be interesting to go back to where it all started - which appears to be Pearson's 1900 paper. If you are familiar ...
- 3,900
32
votes
2
answers
3k
views
Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?
While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed.
Thus the question: are they formally equivalent? I....
- 1,359
32
votes
3
answers
7k
views
Is the exact value of a 'p-value' meaningless?
I had a discussion with a statistician back in 2009 where he stated that the exact value of a p-value is irrelevant: the only thing that is important is whether it is significant or not. I.e. one ...
- 539
32
votes
5
answers
40k
views
Getting p-values for "multinom" in R (nnet package)
How do I get p-values using the multinom function of nnet package in R?
I have a dataset ...
- 629
32
votes
5
answers
3k
views
How should an individual researcher think about the false discovery rate?
I've been trying to wrap my head around how the False Discovery Rate (FDR) should inform the conclusions of the individual researcher. For example, if your study is underpowered, should you discount ...
- 1,209
31
votes
8
answers
3k
views
The abundance of P values in absence of a hypothesis
I'm into epidemiology. I'm not a statistician but I try to perform the analyses myself, although I often encounter difficulties. I did my first analysis some 2 years ago. P values were included ...
- 2,403
30
votes
4
answers
4k
views
Does a uniform distribution of many p-values give statistical evidence that H0 is true?
A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
30
votes
3
answers
3k
views
What causes the discontinuity in the distribution of published p-values at p = 5%?
In a recent paper , Masicampo and Lalande (M-L) collected a large number of p-values published in many different studies. They observed a curious jump in the histogram of the p-values right at the ...
29
votes
3
answers
26k
views
Sanity check: how low can a p-value go?
I am using a ranksum test to compare the median of two samples ($n=120000$) and have found that they are significantly different with: p = 1.12E-207. Should I be ...
- 1,975