Questions tagged [p-value]

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

Filter by
Sorted by
Tagged with
283 votes
16 answers
530k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
user avatar
  • 4,186
157 votes
5 answers
81k views

Why are p-values uniformly distributed under the null hypothesis?

Recently, I have found in a paper by Klammer, et al. a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand why it is so. Klammer, A. A., Park, C. Y.,...
user avatar
  • 1,593
116 votes
10 answers
7k views

ASA discusses limitations of $p$-values - what are the alternatives?

We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
user avatar
  • 114k
102 votes
9 answers
14k views

Is this really how p-values work? Can a million research papers per year be based on pure randomness?

I'm very new to statistics, and I'm just learning to understand the basics, including $p$-values. But there is a huge question mark in my mind right now, and I kind of hope my understanding is wrong. ...
user avatar
  • 1,071
95 votes
2 answers
13k views

How much do we know about p-hacking "in the wild"?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
user avatar
  • 20.9k
91 votes
4 answers
32k views

When to use Fisher and Neyman-Pearson framework?

I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought. My question is, ignoring philosophical objections for a ...
user avatar
  • 1,560
86 votes
11 answers
206k views

How to obtain the p-value (check significance) of an effect in a lme4 mixed model?

I use lme4 in R to fit the mixed model lmer(value~status+(1|experiment))) where value is continuous, status and experiment are factors, and I get ...
user avatar
  • 1,931
83 votes
10 answers
22k views

Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%? Is ...
user avatar
  • 1,407
75 votes
4 answers
24k views

How should tiny $p$-values be reported? (and why does R put a minimum on 2.22e-16?)

For some tests in R, there is a lower limit on the p-value calculations of $2.22 \cdot 10^{-16}$. I'm not sure why it's this number, if there is a good reason for ...
user avatar
  • 1,342
73 votes
4 answers
6k views

A psychology journal banned p-values and confidence intervals; is it indeed wise to stop using them?

On 25 February 2015, the journal Basic and Applied Social Psychology issued an editorial banning $p$-values and confidence intervals from all future papers. Specifically, they say (formatting and ...
71 votes
3 answers
3k views

Is this the solution to the p-value problem?

In February 2016, the American Statistical Association released a formal statement on statistical significance and p-values. Our thread about it discusses these issues extensively. However, no ...
69 votes
8 answers
8k views

What is a good, convincing example in which p-values are useful?

My question in the title is self explanatory, but I would like to give it some context. The ASA released a statement earlier this week “on p-values: context, process, and purpose”, outlining various ...
user avatar
  • 20.1k
67 votes
3 answers
5k views

References containing arguments against null hypothesis significance testing?

In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
63 votes
5 answers
4k views

Why does collecting data until finding a significant result increase Type I error rate?

I was wondering exactly why collecting data until a significant result (e.g., $p \lt .05$) is obtained (i.e., p-hacking) increases the Type I error rate? I would also highly appreciate an ...
user avatar
  • 896
63 votes
3 answers
26k views

Explain the xkcd jelly bean comic: What makes it funny?

I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$). xkcd jelly bean ...
user avatar
  • 693
63 votes
3 answers
119k views

What is the effect of having correlated predictors in a multiple regression model?

I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
user avatar
62 votes
6 answers
4k views

Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
user avatar
  • 94.9k
61 votes
10 answers
11k views

What does "Scientists rise up against statistical significance" mean? (Comment in Nature)

The title of the Comment in Nature Scientists rise up against statistical significance begins with: Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to ...
user avatar
  • 685
58 votes
3 answers
22k views

When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
user avatar
  • 2,133
58 votes
4 answers
147k views

Manually Calculating P value from t-value in t-test

I have a sample dataset with 31 values. I ran a two-tailed t-test using R to test if the true mean is equal to 10: t.test(x=data, mu=10, conf.level=0.95) Output: ...
user avatar
  • 683
57 votes
2 answers
62k views

A/B tests: z-test vs t-test vs chi square vs fisher exact test

I'm trying to understand the reasoning by choosing a specific test approach when dealing with a simple A/B test - (i.e. two variations/groups with a binary respone (converted or not). As an example I ...
user avatar
  • 1,129
53 votes
5 answers
38k views

Correct spelling (capitalization, italicization, hyphenation) of "p-value"?

I realize this is pedantic and trite, but as a researcher in a field outside of statistics, with limited formal education in statistics, I always wonder if I'm writing "p-value" correctly. ...
user avatar
  • 923
51 votes
4 answers
1k views

Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?

I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
user avatar
  • 94.9k
50 votes
7 answers
6k views

Why is "statistically significant" not enough?

I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is ...
user avatar
  • 611
46 votes
10 answers
7k views

Why do people use p-values instead of computing probability of the model given data?

Roughly speaking a p-value gives a probability of the observed outcome of an experiment given the hypothesis (model). Having this probability (p-value) we want to judge our hypothesis (how likely it ...
user avatar
  • 397
41 votes
3 answers
9k views

Interpretation of p-value in hypothesis testing

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
user avatar
41 votes
2 answers
1k views

Evidence for man-made global warming hits 'gold standard': how did they do this?

This message in a Reuter's article from 25.02.2019 is currently all over the news: Evidence for man-made global warming hits 'gold standard' [Scientists] said confidence that human activities ...
user avatar
40 votes
6 answers
2k views

Effect size as the hypothesis for significance testing

Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked: Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
user avatar
37 votes
3 answers
55k views

Computing p-value using bootstrap with R

I use "boot" package to compute an approximated 2-sided bootstrapped p-value but the result is too far away from p-value of using t.test. I can't figure out what I did wrong in my R code. Can someone ...
user avatar
  • 2,657
37 votes
5 answers
7k views

Is p-value essentially useless and dangerous to use?

This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that [Bayesian statistics] is proving especially useful in approaching complex ...
user avatar
  • 2,162
36 votes
4 answers
14k views

Are smaller p-values more convincing?

I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
user avatar
  • 1,596
36 votes
2 answers
4k views

Understanding p-value

I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification. Here is the definition of p-value from Wikipedia: The p-...
user avatar
  • 441
34 votes
7 answers
3k views

Why is it bad to teach students that p-values are the probability that findings are due to chance?

Can someone please offer a nice succinct explanation why it is not a good idea to teach students that a p-value is the prob(their findings are due to [random] chance). My understanding is that a p-...
user avatar
  • 1,411
34 votes
3 answers
6k views

Is p-value a point estimate?

Since one can calculate confidence intervals for p-values and since the opposite of interval estimation is point estimation: Is p-value a point estimate?
user avatar
  • 1,202
34 votes
2 answers
2k views

Performing a statistical test after visualizing data - data dredging?

I'll propose this question by means of an example. Suppose I have a data set, such as the boston housing price data set, in which I have continuous and categorical variables. Here, we have a "quality"...
user avatar
  • 1,260
34 votes
4 answers
4k views

Origin of "5$\sigma$" threshold for accepting evidence in particle physics?

News reports say that CERN will announce tomorrow that the Higgs boson has been experimentally detected with 5$\sigma$ evidence. According to that article: 5$\sigma$ equates to a 99.99994% chance ...
user avatar
33 votes
2 answers
6k views

Who first used/invented p-values?

I am attempting to write a series of blog posts on p-values and I thought it would be interesting to go back to where it all started - which appears to be Pearson's 1900 paper. If you are familiar ...
user avatar
  • 3,660
32 votes
4 answers
3k views

Why are lower p-values not more evidence against the null? Arguments from Johansson 2011

Johansson (2011) in "Hail the impossible: p-values, evidence, and likelihood" (here is also link to the journal) states that lower $p$-values are often considered as stronger evidence against the null....
user avatar
  • 12.5k
32 votes
3 answers
973 views

Accommodating entrenched views of p-values

Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided. I say that since the sample wasn't random, then such statistics would not strictly apply....
user avatar
  • 11.7k
32 votes
5 answers
2k views

How should an individual researcher think about the false discovery rate?

I've been trying to wrap my head around how the False Discovery Rate (FDR) should inform the conclusions of the individual researcher. For example, if your study is underpowered, should you discount ...
user avatar
31 votes
3 answers
121k views

finding p-value in pearson correlation in R

Is it possible to find the p-value in pearson correlation in R? To find the pearson correlation, I usually do this ...
user avatar
  • 593
31 votes
2 answers
6k views

Is the exact value of a 'p-value' meaningless?

I had a discussion with a statistician back in 2009 where he stated that the exact value of a p-value is irrelevant: the only thing that is important is whether it is significant or not. I.e. one ...
user avatar
30 votes
4 answers
3k views

Does a uniform distribution of many p-values give statistical evidence that H0 is true?

A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
user avatar
30 votes
2 answers
2k views

Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?

While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed. Thus the question: are they formally equivalent? I....
user avatar
  • 1,249
30 votes
5 answers
31k views

Getting p-values for "multinom" in R (nnet package)

How do I get p-values using the multinom function of nnet package in R? I have a dataset ...
user avatar
  • 609
29 votes
8 answers
3k views

The abundance of P values in absence of a hypothesis

I'm into epidemiology. I'm not a statistician but I try to perform the analyses myself, although I often encounter difficulties. I did my first analysis some 2 years ago. P values were included ...
user avatar
28 votes
3 answers
3k views

What causes the discontinuity in the distribution of published p-values at p < .05?

In a recent paper , Masicampo and Lalande (M-L) collected a large number of p-values published in many different studies. They observed a curious jump in the histogram of the p-values right at the ...
user avatar
  • 22.1k
26 votes
5 answers
6k views

When does Fisher's "go get more data" approach make sense?

Quoting gung's great answer Allegedly, a researcher once approached Fisher with 'non-significant' results, asking him what he should do, and Fisher said, 'go get more data'. From a Neyman-Pearson ...
user avatar
  • 1,427
26 votes
4 answers
35k views

Why do linear regression and ANOVA give different $p$-value in case of considering interaction between variable?

I was trying to fit one time-series data (without replicates) using regression model. The data looks like follows: ...
user avatar
  • 597
26 votes
4 answers
2k views

Ziliak (2011) opposes the use of p-values and mentions some alternatives; what are they?

In a recent article discussing the demerits of relying on the p-value for statistical inference, called "Matrixx v. Siracusano and Student v. Fisher Statistical significance on trial" (DOI: 10.1111/j....
user avatar
  • 508

1
2 3 4 5
52