Questions tagged [p-value]

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

2,455 questions
Filter by
Sorted by
Tagged with
521k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
75k views

Recently, I have found in a paper by Klammer, et al. a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand why it is so. Klammer, A. A., Park, C. Y.,...
7k views

ASA discusses limitations of $p$-values - what are the alternatives?

We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
14k views

Is this really how p-values work? Can a million research papers per year be based on pure randomness?

I'm very new to statistics, and I'm just learning to understand the basics, including $p$-values. But there is a huge question mark in my mind right now, and I kind of hope my understanding is wrong. ...
13k views

How much do we know about p-hacking "in the wild"?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
32k views

When to use Fisher and Neyman-Pearson framework?

I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought. My question is, ignoring philosophical objections for a ...
21k views

Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%? Is ...
191k views

How to obtain the p-value (check significance) of an effect in a lme4 mixed model?

I use lme4 in R to fit the mixed model lmer(value~status+(1|experiment))) where value is continuous, status and experiment are factors, and I get ...
23k views

How should tiny $p$-values be reported? (and why does R put a minimum on 2.22e-16?)

For some tests in R, there is a lower limit on the p-value calculations of $2.22 \cdot 10^{-16}$. I'm not sure why it's this number, if there is a good reason for ...
6k views

A psychology journal banned p-values and confidence intervals; is it indeed wise to stop using them?

On 25 February 2015, the journal Basic and Applied Social Psychology issued an editorial banning $p$-values and confidence intervals from all future papers. Specifically, they say (formatting and ...
3k views

Is this the solution to the p-value problem?

In February 2016, the American Statistical Association released a formal statement on statistical significance and p-values. Our thread about it discusses these issues extensively. However, no ...
8k views

What is a good, convincing example in which p-values are useful?

My question in the title is self explanatory, but I would like to give it some context. The ASA released a statement earlier this week “on p-values: context, process, and purpose”, outlining various ...
5k views

References containing arguments against null hypothesis significance testing?

In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
4k views

Why does collecting data until finding a significant result increase Type I error rate?

I was wondering exactly why collecting data until a significant result (e.g., $p \lt .05$) is obtained (i.e., p-hacking) increases the Type I error rate? I would also highly appreciate an ...
25k views

Explain the xkcd jelly bean comic: What makes it funny?

I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$). xkcd jelly bean ...
10k views

What does "Scientists rise up against statistical significance" mean? (Comment in Nature)

The title of the Comment in Nature Scientists rise up against statistical significance begins with: Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to ...
115k views

What is the effect of having correlated predictors in a multiple regression model?

I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
4k views

Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
143k views

Manually Calculating P value from t-value in t-test

I have a sample dataset with 31 values. I ran a two-tailed t-test using R to test if the true mean is equal to 10: t.test(x=data, mu=10, conf.level=0.95) Output: ...
21k views

When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that -2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
58k views

A/B tests: z-test vs t-test vs chi square vs fisher exact test

I'm trying to understand the reasoning by choosing a specific test approach when dealing with a simple A/B test - (i.e. two variations/groups with a binary respone (converted or not). As an example I ...
37k views

Correct spelling (capitalization, italicization, hyphenation) of "p-value"?

I realize this is pedantic and trite, but as a researcher in a field outside of statistics, with limited formal education in statistics, I always wonder if I'm writing "p-value" correctly. ...
1k views

Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?

I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
6k views

Why is "statistically significant" not enough?

I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is ...
7k views

Why do people use p-values instead of computing probability of the model given data?

Roughly speaking a p-value gives a probability of the observed outcome of an experiment given the hypothesis (model). Having this probability (p-value) we want to judge our hypothesis (how likely it ...
9k views

Interpretation of p-value in hypothesis testing

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
1k views

Evidence for man-made global warming hits 'gold standard': how did they do this?

This message in a Reuter's article from 25.02.2019 is currently all over the news: Evidence for man-made global warming hits 'gold standard' [Scientists] said confidence that human activities ...
2k views

Effect size as the hypothesis for significance testing

Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked: Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
7k views

Is p-value essentially useless and dangerous to use?

This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that [Bayesian statistics] is proving especially useful in approaching complex ...
52k views

Computing p-value using bootstrap with R

I use "boot" package to compute an approximated 2-sided bootstrapped p-value but the result is too far away from p-value of using t.test. I can't figure out what I did wrong in my R code. Can someone ...
13k views

Are smaller p-values more convincing?

I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
4k views

Understanding p-value

I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification. Here is the definition of p-value from Wikipedia: The p-...
3k views

Why is it bad to teach students that p-values are the probability that findings are due to chance?

Can someone please offer a nice succinct explanation why it is not a good idea to teach students that a p-value is the prob(their findings are due to [random] chance). My understanding is that a p-...
6k views

Is p-value a point estimate?

Since one can calculate confidence intervals for p-values and since the opposite of interval estimation is point estimation: Is p-value a point estimate?
2k views

Performing a statistical test after visualizing data - data dredging?

I'll propose this question by means of an example. Suppose I have a data set, such as the boston housing price data set, in which I have continuous and categorical variables. Here, we have a "quality"...
3k views

Origin of "5$\sigma$" threshold for accepting evidence in particle physics?

News reports say that CERN will announce tomorrow that the Higgs boson has been experimentally detected with 5$\sigma$ evidence. According to that article: 5$\sigma$ equates to a 99.99994% chance ...
3k views

Why are lower p-values not more evidence against the null? Arguments from Johansson 2011

Johansson (2011) in "Hail the impossible: p-values, evidence, and likelihood" (here is also link to the journal) states that lower $p$-values are often considered as stronger evidence against the null....
957 views

Accommodating entrenched views of p-values

Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided. I say that since the sample wasn't random, then such statistics would not strictly apply....
2k views

How should an individual researcher think about the false discovery rate?

I've been trying to wrap my head around how the False Discovery Rate (FDR) should inform the conclusions of the individual researcher. For example, if your study is underpowered, should you discount ...
117k views

finding p-value in pearson correlation in R

Is it possible to find the p-value in pearson correlation in R? To find the pearson correlation, I usually do this ...
6k views

Is the exact value of a 'p-value' meaningless?

I had a discussion with a statistician back in 2009 where he stated that the exact value of a p-value is irrelevant: the only thing that is important is whether it is significant or not. I.e. one ...
6k views

Who first used/invented p-values?

I am attempting to write a series of blog posts on p-values and I thought it would be interesting to go back to where it all started - which appears to be Pearson's 1900 paper. If you are familiar ...
3k views

Does a uniform distribution of many p-values give statistical evidence that H0 is true?

A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
2k views

Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?

While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed. Thus the question: are they formally equivalent? I....
2k views

The abundance of P values in absence of a hypothesis

I'm into epidemiology. I'm not a statistician but I try to perform the analyses myself, although I often encounter difficulties. I did my first analysis some 2 years ago. P values were included ...
2k views

What causes the discontinuity in the distribution of published p-values at p < .05?

In a recent paper , Masicampo and Lalande (M-L) collected a large number of p-values published in many different studies. They observed a curious jump in the histogram of the p-values right at the ...
29k views

Getting p-values for "multinom" in R (nnet package)

How do I get p-values using the multinom function of nnet package in R? I have a dataset ...
6k views

When does Fisher's "go get more data" approach make sense?

Quoting gung's great answer Allegedly, a researcher once approached Fisher with 'non-significant' results, asking him what he should do, and Fisher said, 'go get more data'. From a Neyman-Pearson ...