Questions tagged [p-value]

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

Filter by
Sorted by
Tagged with
253
votes
16answers
473k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
118
votes
6answers
53k views

Why are p-values uniformly distributed under the null hypothesis?

Recently, I have found in a paper by Klammer, et al. a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand why it is so. Klammer, A. A., Park, C. Y.,...
17
votes
4answers
6k views

Misunderstanding a P-value?

So I've been reading a lot about how to correctly interpret a P-value, and from what I've read, the p-value says NOTHING about the probability that the null hypothesis is true or false. However, when ...
66
votes
4answers
18k views

How should tiny $p$-values be reported? (and why does R put a minimum on 2.22e-16?)

For some tests in R, there is a lower limit on the p-value calculations of $2.22 \cdot 10^{-16}$. I'm not sure why it's this number, if there is a good reason for ...
75
votes
4answers
28k views

When to use Fisher and Neyman-Pearson framework?

I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought. My question is, ignoring philosophical objections for a ...
28
votes
3answers
34k views

Computing p-value using bootstrap with R

I use "boot" package to compute an approximated 2-sided bootstrapped p-value but the result is too far away from p-value of using t.test. I can't figure out what I did wrong in my R code. Can someone ...
22
votes
3answers
17k views

Comparing and contrasting, p-values, significance levels and type I error

I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error. I understand that p-values are defined as "the probability of ...
60
votes
9answers
128k views

How to obtain the p-value (check significance) of an effect in a lme4 mixed model?

I use lme4 in R to fit the mixed model lmer(value~status+(1|experiment))) where value is continuous, status and experiment are factors, and I get ...
36
votes
3answers
8k views

Interpretation of p-value in hypothesis testing

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
47
votes
3answers
92k views

What is the effect of having correlated predictors in a multiple regression model?

I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
33
votes
2answers
4k views

Understanding p-value

I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification. Here is the definition of p-value from Wikipedia: The p-...
7
votes
2answers
19k views

What does the notation like 8.6e-28 mean? What is the 'e' for?

I have a problem with the interpretation of a test result in which the p-value is 8.6e-28. How should it be interpreted? What is the ...
80
votes
9answers
18k views

Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%? Is ...
29
votes
2answers
2k views

Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?

While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed. Thus the question: are they formally equivalent? I....
105
votes
10answers
6k views

ASA discusses limitations of $p$-values - what are the alternatives?

We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
94
votes
2answers
12k views

How much do we know about p-hacking “in the wild”?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
60
votes
3answers
21k views

Explain the xkcd jelly bean comic: What makes it funny?

I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$). xkcd jelly bean ...
37
votes
6answers
2k views

Effect size as the hypothesis for significance testing

Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked: Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
47
votes
3answers
15k views

When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
18
votes
2answers
5k views

P-value in a two-tail test with asymmetric null distribution

My situation is as follows: I want, through a Monte-Carlo study, to compare $p$-values of two different tests for statistical significance of an estimated parameter (null is "no effect - parameter is ...
10
votes
1answer
122k views

“Accept null hypothesis” or “fail to reject the null hypothesis”? [duplicate]

I'm trying to conduct a Student's t-test for a table of values while trying to follow the explanation and details found on this website. I understand that if the p-value is <.01 then it's really ...
46
votes
7answers
5k views

Why is “statistically significant” not enough?

I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is ...
56
votes
6answers
3k views

Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
12
votes
2answers
30k views

Stepwise regression in R – Critical p-value

What is the critical p-value used by the step() function in R for stepwise regression? I assume it is 0.15, but is my assumption correct? How can I change the ...
16
votes
2answers
4k views

Explaining two-tailed tests

I am looking for various ways of explaining to my students (in an elementary statistics course) what is a two tailed test, and how its P value is calculated. How do you explain to your students the ...
31
votes
3answers
889 views

Accommodating entrenched views of p-values

Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided. I say that since the sample wasn't random, then such statistics would not strictly apply....
7
votes
2answers
2k views

Why are the number of false positives independent of sample size, if we use p-values to compare two independent datasets?

If I run the R code below, it will generate two independent vectors then test them to see if they are related in some way (i.e. p-value < 0.05). If I repeat this 1,000 times, then 50 of these (5%) ...
17
votes
1answer
1k views

Non-uniform distribution of p-values when simulating binomial tests under the null hypothesis

I heard that under the null hypothesis the p-value distribution should be uniform. However, simulations of binomial test in MATLAB return very different-from-uniform distributions with mean larger ...
52
votes
4answers
1k views

Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?

I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
31
votes
2answers
6k views

Is the exact value of a 'p-value' meaningless?

I had a discussion with a statistician back in 2009 where he stated that the exact value of a p-value is irrelevant: the only thing that is important is whether it is significant or not. I.e. one ...
16
votes
1answer
42k views

sample size too large? [duplicate]

I always thought larger sample sizes were better. Then I read something somewhere about how when sample sizes are larger, it's easier to find significant p-values when they're not really there (i.e., ...
17
votes
3answers
32k views

What is the relationship between R-squared and p-value in a regression?

tl;dr - for OLS regression, does a higher R-squared also imply a higher P-value? Specifically for a single explanatory variable (Y = a + bX + e) but would also be interested to know for n multiple ...
3
votes
3answers
10k views

Chi-Square-Test: Why is the chi-squared test a one-tailed test? [duplicate]

In my Masters thesis I conducted an experiment in that subjects should decide between two alternatives. I had a control and a treatment group. The hypothesis was that in the treatment group more ...
5
votes
1answer
1k views

High p-values for logistic regression variable that perfectly separates?

I'm using R to run some logistic regression. My variables were continuous, but I used cut to bucket the data. Some particular buckets for these variables always result in dependent variable being ...
63
votes
3answers
5k views

References containing arguments against null hypothesis significance testing?

In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
31
votes
4answers
10k views

Are smaller p-values more convincing?

I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
31
votes
4answers
3k views

Why are lower p-values not more evidence against the null? Arguments from Johansson 2011

Johansson (2011) in "Hail the impossible: p-values, evidence, and likelihood" (here is also link to the journal) states that lower $p$-values are often considered as stronger evidence against the null....
8
votes
2answers
8k views

Ridge regression in R with p values and goodness of fit

Doing ridge regression in R I have discovered linearRidge in the ridge package - which fits a model, reports coefficients and p ...
-2
votes
1answer
5k views

What test should I use with shannon diversity index values?

What sort of test should I use to get the p-value after getting a shannon diversity index? Also does anyone know if and how they can be done using PRIMER 5?
4
votes
2answers
2k views

Regression: Scatterplot with low R squared and high p-values

Based on three datasets, I have produced the scatterplot below in Python: I am trying to fit a line on each dataset, but when I check the metrics this is what I get: Set 1 (red): $R^2$=0.002, p-...
4
votes
1answer
8k views

Calculate p-value for a negative z

I'm trying to do a hypothesis testing and the result I'm getting is strange because the p value I get is greater than 1. I'm using $p_0= 0.65$ and $\hat{p} = 0.60$. To calculate $Z$ I'm using $(0.60-...
97
votes
9answers
14k views

Is this really how p-values work? Can a million research papers per year be based on pure randomness?

I'm very new to statistics, and I'm just learning to understand the basics, including $p$-values. But there is a huge question mark in my mind right now, and I kind of hope my understanding is wrong. ...
11
votes
2answers
13k views

How to get pooled p-values on tests done in multiple imputed datasets?

Using Amelia in R, I obtained multiple imputed datasets. After that, I performed a repeated measures test in SPSS. Now, I want to pool test results. I know that I can use Rubin's rules (implemented ...
19
votes
3answers
3k views

Confusion with false discovery rate and multiple testing (on Colquhoun 2014)

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can ...
18
votes
3answers
3k views

Is it wrong to refer to results as being “highly significant”?

Why do statisticians discourage us from referring to results as "highly significant" when the $p$-value is well below the conventional $\alpha$-level of $0.05$? Is it really wrong to trust a result ...
9
votes
2answers
4k views

Why are 0.05 < p < 0.95 results called false positives?

Edit: The basis of my question is flawed, and I need to spend some time figuring out whether it can even be made to make sense. Edit 2: Clarifying that I recognize that a p-value isn't a direct ...
6
votes
3answers
2k views

p-value with multimodal PDF of a test statistic

I have opened a thread about p-value under the title "Understanding p-value" and gotten two answers and some comments. I think my questions in the thread is somewhat diverse and want to clarify my ...
24
votes
3answers
13k views

Sanity check: how low can a p-value go?

I am using a ranksum test to compare the median of two samples ($n=120000$) and have found that they are significantly different with: p = 1.12E-207. Should I be ...
10
votes
3answers
26k views

glm in R - which pvalue represents the goodness of fit of entire model?

I am running glms in R (generalised linear models). I thought I knew pvalues - until I saw that calling up a summary for a glm does not give you an overriding pvalue representative of the model as a ...
8
votes
3answers
9k views

How to correct for small p-value due to very large sample size

I am running into a problem where an independent variable, which should have no predictive power on the dependent variable based on domain knowledge, comes out with very small p-value because the ...