# Questions tagged [p-value]

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

2,003 questions
Filter by
Sorted by
Tagged with
1k views

### Non-parametric bootstrap p-values vs confidence intervals

Context This is somewhat similar to this question, but I do not think it is an exact duplicate. When you look for how instructions on how to perform a bootstrap hypothesis test, it is usually stated ...
1k views

### Adjusting the p-value for adaptive sequential analysis (for chi square test)?

I wish to know what statistical literature is relevant for the following problem, and maybe even an idea on how to solve it. Imagine the following problem: We have 4 possible treatments for some ...
506 views

### How to rigorously justify chosen false-positive/false-negative error rates and underlying cost ratio?

Context A group of social scientists and statisticians (Benjamin et al., 2017) have recently suggested that the typical false-positive rate ($\alpha$ = .05) used as a threshold for determining "...
2k views

### Why we reject the null hypothesis at the 0.05 level and not the 0.5 level (as we do in the Classification)

Hypothesis testing is akin to a Classification problem. So say, we have 2 possible labels for an observation (subject) -- Guilty vs. Non-Guilty. Let Non-Guilty be the null Hypothesis. If we viewed the ...
33k views

### What is the difference between Z-scores and p-values?

In network motif algorithms, it seems quite common to return both a p-value and a Z-score for a statistic: "Input network contains X copies of subgraph G". A subgraph is deemed a motif if it ...
14k views

### How to get pooled p-values on tests done in multiple imputed datasets?

Using Amelia in R, I obtained multiple imputed datasets. After that, I performed a repeated measures test in SPSS. Now, I want to pool test results. I know that I can use Rubin's rules (implemented ...
2k views

### Why are the cut-offs used for Bayes factors and p-values so different?

I am trying to understand Bayes Factor (BF). I believe they are like likelihood ratio of 2 hypotheses. So if BF is 5, it means H1 is 5 times more likely than H0. And value of 3-10 indicates moderate ...
22k views

### Examples of studies using p < 0.001, p < 0.0001 or even lower p-values?

I come from the social sciences, where p < 0.05 is pretty much the norm, with p < 0.1 and p < 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a ...
123k views

### “Accept null hypothesis” or “fail to reject the null hypothesis”? [duplicate]

I'm trying to conduct a Student's t-test for a table of values while trying to follow the explanation and details found on this website. I understand that if the p-value is <.01 then it's really ...
507 views

### p-value subtlety: greater-equal vs. greater

As I am reading Wassermann's book All of Statistics, I notice a fine subtlety in the definition of p-values, which I cannot make sense of. Informally, the Wassermann defines the p-value as [..] ...
991 views

### Two definitions of p-value: how to prove their equivalence?

I'm reading through Larry Wasserman's book, All of Statistics, and currently about p-values (page 187). Let me first introduce some definitions (I quote): Definition 1 The power function of a test ...
2k views

### “Liberal” p-values?

My question is rather semantic. When a method routinely produces high p-values it is called conservative. Would you call the opposite, i.e. a method with a high type-II-error rate liberal?
12k views

### Two methods of bootstrap significance tests

Using bootstrap I calculate p values of significance tests using two methods: resampling under the null hypothesis and counting the outcomes at least as extreme as the outcome coming from the ...
250 views

### Paper on performing hypothesis tests based on outcome of another test

It is well known that it is problematic to choose a statistical test based on the outcome of another statistical test, as the p-values are difficult to impossible to interpret (e.g. Choosing a ...
14k views

### How to obtain p-values of coefficients from bootstrap regression?

From Robert Kabacoff's Quick-R I have ...
8k views

### What are Hommel Hochberg corrections?

I have recently been introduced to to Hommel Hochberg corrections. I am trying to find a simple explanation about what this actually is/does, but am having no luck. Can anyone please give a brief and ...
3k views

### What is the benefit of using permutation tests?

When testing some null versus alternative hypotheses by a test statistic $U(X)$, where $X = \{ x_i, ..., x_n\}$, apply the permutation test with the set $G$ of permutations on $X$ and we have a new ...
28k views

### glm in R - which pvalue represents the goodness of fit of entire model?

I am running glms in R (generalised linear models). I thought I knew pvalues - until I saw that calling up a summary for a glm does not give you an overriding pvalue representative of the model as a ...
8k views

### What are Bayesian p-values?

I'm looking for an answer that would satisfy a reader who understands frequentist p-values but only understands the rudiments of Bayesian approaches to statistics. At present google searches do not ...
7k views

### G-test vs Pearson's chi-squared test

I'm testing independence in an $N \times M$ contingency table. I don't know whether the G-test or Pearson's chi-squared test is better. The sample size is in the hundreds but there are some low cell ...
5k views

### How to test an interaction effect with a non-parametric test (e.g. a permutation test)?

I have two categorical / nominal variables. Each of them can take only two distinct values (so, I have 4 combinations in total). Each combination of values comes with a set of numerical values. So, I ...
6k views

### How to Interpret GAM P-Values?

My name is Hugh, and I'm a PhD student using generalised additive models to do some exploratory analysis. I'm not sure how to interpret the p-values that come from the MGCV package and wanted to ...
9k views

### P value for interaction term in mixed effects models using lme4

I'm analysing some behavioural data using lme4 in R, mostly following Bodo Winter's excellent tutorials, but I don't understand ...
385 views

### A ''significant variable'' that does not improve out-of-sample predictions - how to interpret?

I have a question that I think will be quite basic to a lot of users. Im using linear regression models to (i) investigate the relationship of several explanatory variables and my response variable ...
534 views

### Should I trust the $p$-value in statistical tests?

I got into a debate with my supervisor over a recent paper. The test of correlation in a sample of 77 participants yielded a p-value smaller than 0.05. After removing a few participants (because later ...
2k views

### p-values of Mann-Whitney U test identical for raw and log-transformed data

I am new to Stats and came across this problem while running my analyses on SPSS which I cannot explain. How is it that even after transforming my data by logging it, it still has the same p-value ...
4k views

### Why are 0.05 < p < 0.95 results called false positives?

Edit: The basis of my question is flawed, and I need to spend some time figuring out whether it can even be made to make sense. Edit 2: Clarifying that I recognize that a p-value isn't a direct ...
2k views

### Using p-value to compute the probability of hypothesis being true; what else is needed?

Question: One common misunderstanding of p-values is that they represent the probability of the null hypothesis being true. I know that's not correct and I know that p-values only represent the ...
2k views

### Hypothesis testing. Why center the sampling distribution on H0?

A p-value is the probability to obtain a statistic that is at least as extreme as the one observed in the sample data when assuming that the null-hypothesis ($H_0$) is true. Graphically this ...
9k views

### Ridge regression in R with p values and goodness of fit

Doing ridge regression in R I have discovered linearRidge in the ridge package - which fits a model, reports coefficients and p ...
680 views

### Unknown p-value calculation

I was recently debugging an R script and I found something very weird, the author defined their own p-value function ...
455 views

### Why is p-value termed as P(Data | Hypothesis/Model)?

As the title suggests, Why is p-value termed as P(Data | Hypothesis/Model) and not P(Hypothesis | Data)? Shouldn't both be the same? Why is P(Data | Hypothesis) != P(Hypothesis | Data)? Is there any ...
3k views

### Why is the confidence interval considered a random interval?

I've been reading a lot on confidence intervals lately and I keep seeing statements such as: "A 95% confidence interval is a random interval that contains the true parameter 95% of the time" or "A ...
3k views

### What is the relationship between $p$ values and Type I errors [duplicate]

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was ...
2k views

### Equivalence of AIC and p-values in model selection

In a comment to the answer of this question, it was stated that using AIC in model selection was equivalent to using a p-value of 0.154. I tried it in R, where I used a "backward" subset selection ...
4k views

### Can a -2 Log likelihood be calculated with only one model?

I am using the glmfit function in MATLAB. The function only returns the deviance and not the log likelihood. I understand that the deviance is basically twice the ...
16k views

### Meaning of p-value of logistic regression model variables

So I'm working with logistic regression models in R. Though I'm still new to statistics I feel like I got a bit of an understanding for regression models by now, but there's still something that ...
16k views

### How to calculate p-value for multivariate linear regression

Software packages that calculate regressions sometimes also return p-values. I want to understand how to calculate this p-value by hand. Here's what I think I understand: I want to calculate the ...
4k views

### Why does Stouffer's method work?

It seems like a fairly straightforward question, but when I really think about it, Stouffer's method doesn't make sense to me. This is why: Assume a two-tailed hypothesis. You first calculate $z_i$ ...
2k views

### Why is my combined p-value, obtained using the Fisher's method, so low?

I have multiple individuals, for whom I collected two time series of some parameter. For each individual, I calculated, whether these time series are correlated. So, if I have 20 individuals, as a ...
37k views

### Understanding the p-value in Spearman's rank correlation

I have some pairs of datasets (n=200 or thereabouts), of samples which are non-negative and not normally distributed. I think these pairs of variables are related, ...
7k views

### What is a good introduction to statistical hypothesis testing for computer scientists?

I was recently exposed to some statistical hypothesis testing methods (e.g. Friedman test) at work, and I would like to increase my knowledge on the topic. Can you suggest a good introduction to ...
35k views

### What does a p-value of exactly 1.0000 mean?

I have a table of pairwise comparisons using paired t-tests, which I produced using the following code in R: ...
4k views

### How is $\chi^2$ value converted to p-value?

I'm brand new to statistics and am studying the math behind split testing (A/B and multivariate). I've learned how to calculate $\chi^2$ with given test data, and I understand how to translate this ...
898 views

### Does an exact test always yield a higher P value then an approximated test?

I ran a simulation on that for mcnemar test, and the answer seemed to be yes. I was wondering if this can always be said to be the case that the exact P value is higher (or not smaller) then the p ...
2k views

### Why are the number of false positives independent of sample size, if we use p-values to compare two independent datasets?

If I run the R code below, it will generate two independent vectors then test them to see if they are related in some way (i.e. p-value < 0.05). If I repeat this 1,000 times, then 50 of these (5%) ...
9k views

### How to correct for small p-value due to very large sample size

I am running into a problem where an independent variable, which should have no predictive power on the dependent variable based on domain knowledge, comes out with very small p-value because the ...
1k views

### Is the type I error rate equal to alpha or at most alpha?

According the p-value Wikipedia page: When the p-value is calculated correctly, this test guarantees that the Type I error rate is at most $\alpha$. However further down the page this formula is ...
If I need to calculate a permutation-based $p$-value with significance level $\alpha$, how many permutations do I need ? From the article "Permutation Tests for Studying Classifier Performance", page ...