Questions tagged [p-value]

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

Filter by
Sorted by
Tagged with
12
votes
1answer
1k views

Non-parametric bootstrap p-values vs confidence intervals

Context This is somewhat similar to this question, but I do not think it is an exact duplicate. When you look for how instructions on how to perform a bootstrap hypothesis test, it is usually stated ...
12
votes
2answers
1k views

Adjusting the p-value for adaptive sequential analysis (for chi square test)?

I wish to know what statistical literature is relevant for the following problem, and maybe even an idea on how to solve it. Imagine the following problem: We have 4 possible treatments for some ...
12
votes
2answers
506 views

How to rigorously justify chosen false-positive/false-negative error rates and underlying cost ratio?

Context A group of social scientists and statisticians (Benjamin et al., 2017) have recently suggested that the typical false-positive rate ($\alpha$ = .05) used as a threshold for determining "...
11
votes
5answers
2k views

Why we reject the null hypothesis at the 0.05 level and not the 0.5 level (as we do in the Classification)

Hypothesis testing is akin to a Classification problem. So say, we have 2 possible labels for an observation (subject) -- Guilty vs. Non-Guilty. Let Non-Guilty be the null Hypothesis. If we viewed the ...
11
votes
3answers
33k views

What is the difference between Z-scores and p-values?

In network motif algorithms, it seems quite common to return both a p-value and a Z-score for a statistic: "Input network contains X copies of subgraph G". A subgraph is deemed a motif if it ...
11
votes
2answers
14k views

How to get pooled p-values on tests done in multiple imputed datasets?

Using Amelia in R, I obtained multiple imputed datasets. After that, I performed a repeated measures test in SPSS. Now, I want to pool test results. I know that I can use Rubin's rules (implemented ...
11
votes
3answers
2k views

Why are the cut-offs used for Bayes factors and p-values so different?

I am trying to understand Bayes Factor (BF). I believe they are like likelihood ratio of 2 hypotheses. So if BF is 5, it means H1 is 5 times more likely than H0. And value of 3-10 indicates moderate ...
11
votes
4answers
22k views

Examples of studies using p < 0.001, p < 0.0001 or even lower p-values?

I come from the social sciences, where p < 0.05 is pretty much the norm, with p < 0.1 and p < 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a ...
11
votes
1answer
123k views

“Accept null hypothesis” or “fail to reject the null hypothesis”? [duplicate]

I'm trying to conduct a Student's t-test for a table of values while trying to follow the explanation and details found on this website. I understand that if the p-value is <.01 then it's really ...
11
votes
2answers
507 views

p-value subtlety: greater-equal vs. greater

As I am reading Wassermann's book All of Statistics, I notice a fine subtlety in the definition of p-values, which I cannot make sense of. Informally, the Wassermann defines the p-value as [..] ...
11
votes
2answers
991 views

Two definitions of p-value: how to prove their equivalence?

I'm reading through Larry Wasserman's book, All of Statistics, and currently about p-values (page 187). Let me first introduce some definitions (I quote): Definition 1 The power function of a test ...
11
votes
2answers
2k views

“Liberal” p-values?

My question is rather semantic. When a method routinely produces high p-values it is called conservative. Would you call the opposite, i.e. a method with a high type-II-error rate liberal?
11
votes
1answer
12k views

Two methods of bootstrap significance tests

Using bootstrap I calculate p values of significance tests using two methods: resampling under the null hypothesis and counting the outcomes at least as extreme as the outcome coming from the ...
11
votes
1answer
250 views

Paper on performing hypothesis tests based on outcome of another test

It is well known that it is problematic to choose a statistical test based on the outcome of another statistical test, as the p-values are difficult to impossible to interpret (e.g. Choosing a ...
10
votes
3answers
14k views

How to obtain p-values of coefficients from bootstrap regression?

From Robert Kabacoff's Quick-R I have ...
10
votes
1answer
8k views

What are Hommel Hochberg corrections?

I have recently been introduced to to Hommel Hochberg corrections. I am trying to find a simple explanation about what this actually is/does, but am having no luck. Can anyone please give a brief and ...
10
votes
1answer
3k views

What is the benefit of using permutation tests?

When testing some null versus alternative hypotheses by a test statistic $U(X)$, where $X = \{ x_i, ..., x_n\}$, apply the permutation test with the set $G$ of permutations on $X$ and we have a new ...
10
votes
3answers
28k views

glm in R - which pvalue represents the goodness of fit of entire model?

I am running glms in R (generalised linear models). I thought I knew pvalues - until I saw that calling up a summary for a glm does not give you an overriding pvalue representative of the model as a ...
10
votes
1answer
8k views

What are Bayesian p-values?

I'm looking for an answer that would satisfy a reader who understands frequentist p-values but only understands the rudiments of Bayesian approaches to statistics. At present google searches do not ...
10
votes
3answers
7k views

G-test vs Pearson's chi-squared test

I'm testing independence in an $N \times M$ contingency table. I don't know whether the G-test or Pearson's chi-squared test is better. The sample size is in the hundreds but there are some low cell ...
10
votes
5answers
5k views

How to test an interaction effect with a non-parametric test (e.g. a permutation test)?

I have two categorical / nominal variables. Each of them can take only two distinct values (so, I have 4 combinations in total). Each combination of values comes with a set of numerical values. So, I ...
10
votes
1answer
6k views

How to Interpret GAM P-Values?

My name is Hugh, and I'm a PhD student using generalised additive models to do some exploratory analysis. I'm not sure how to interpret the p-values that come from the MGCV package and wanted to ...
10
votes
2answers
9k views

P value for interaction term in mixed effects models using lme4

I'm analysing some behavioural data using lme4 in R, mostly following Bodo Winter's excellent tutorials, but I don't understand ...
10
votes
2answers
385 views

A ''significant variable'' that does not improve out-of-sample predictions - how to interpret?

I have a question that I think will be quite basic to a lot of users. Im using linear regression models to (i) investigate the relationship of several explanatory variables and my response variable ...
9
votes
8answers
534 views

Should I trust the $p$-value in statistical tests?

I got into a debate with my supervisor over a recent paper. The test of correlation in a sample of 77 participants yielded a p-value smaller than 0.05. After removing a few participants (because later ...
9
votes
2answers
2k views

p-values of Mann-Whitney U test identical for raw and log-transformed data

I am new to Stats and came across this problem while running my analyses on SPSS which I cannot explain. How is it that even after transforming my data by logging it, it still has the same p-value ...
9
votes
2answers
4k views

Why are 0.05 < p < 0.95 results called false positives?

Edit: The basis of my question is flawed, and I need to spend some time figuring out whether it can even be made to make sense. Edit 2: Clarifying that I recognize that a p-value isn't a direct ...
9
votes
6answers
2k views

Using p-value to compute the probability of hypothesis being true; what else is needed?

Question: One common misunderstanding of p-values is that they represent the probability of the null hypothesis being true. I know that's not correct and I know that p-values only represent the ...
9
votes
3answers
2k views

Hypothesis testing. Why center the sampling distribution on H0?

A p-value is the probability to obtain a statistic that is at least as extreme as the one observed in the sample data when assuming that the null-hypothesis ($H_0$) is true. Graphically this ...
9
votes
2answers
9k views

Ridge regression in R with p values and goodness of fit

Doing ridge regression in R I have discovered linearRidge in the ridge package - which fits a model, reports coefficients and p ...
9
votes
1answer
680 views

Unknown p-value calculation

I was recently debugging an R script and I found something very weird, the author defined their own p-value function ...
9
votes
2answers
455 views

Why is p-value termed as P(Data | Hypothesis/Model)?

As the title suggests, Why is p-value termed as P(Data | Hypothesis/Model) and not P(Hypothesis | Data)? Shouldn't both be the same? Why is P(Data | Hypothesis) != P(Hypothesis | Data)? Is there any ...
9
votes
3answers
3k views

Why is the confidence interval considered a random interval?

I've been reading a lot on confidence intervals lately and I keep seeing statements such as: "A 95% confidence interval is a random interval that contains the true parameter 95% of the time" or "A ...
9
votes
5answers
3k views

What is the relationship between $p$ values and Type I errors [duplicate]

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was ...
9
votes
1answer
2k views

Equivalence of AIC and p-values in model selection

In a comment to the answer of this question, it was stated that using AIC in model selection was equivalent to using a p-value of 0.154. I tried it in R, where I used a "backward" subset selection ...
9
votes
1answer
4k views

Can a -2 Log likelihood be calculated with only one model?

I am using the glmfit function in MATLAB. The function only returns the deviance and not the log likelihood. I understand that the deviance is basically twice the ...
9
votes
1answer
16k views

Meaning of p-value of logistic regression model variables

So I'm working with logistic regression models in R. Though I'm still new to statistics I feel like I got a bit of an understanding for regression models by now, but there's still something that ...
9
votes
1answer
16k views

How to calculate p-value for multivariate linear regression

Software packages that calculate regressions sometimes also return p-values. I want to understand how to calculate this p-value by hand. Here's what I think I understand: I want to calculate the ...
9
votes
4answers
4k views

Why does Stouffer's method work?

It seems like a fairly straightforward question, but when I really think about it, Stouffer's method doesn't make sense to me. This is why: Assume a two-tailed hypothesis. You first calculate $z_i$ ...
9
votes
1answer
2k views

Why is my combined p-value, obtained using the Fisher's method, so low?

I have multiple individuals, for whom I collected two time series of some parameter. For each individual, I calculated, whether these time series are correlated. So, if I have 20 individuals, as a ...
9
votes
1answer
37k views

Understanding the p-value in Spearman's rank correlation

I have some pairs of datasets (n=200 or thereabouts), of samples which are non-negative and not normally distributed. I think these pairs of variables are related, ...
9
votes
3answers
7k views

What is a good introduction to statistical hypothesis testing for computer scientists?

I was recently exposed to some statistical hypothesis testing methods (e.g. Friedman test) at work, and I would like to increase my knowledge on the topic. Can you suggest a good introduction to ...
8
votes
3answers
35k views

What does a p-value of exactly 1.0000 mean?

I have a table of pairwise comparisons using paired t-tests, which I produced using the following code in R: ...
8
votes
1answer
4k views

How is $\chi^2$ value converted to p-value?

I'm brand new to statistics and am studying the math behind split testing (A/B and multivariate). I've learned how to calculate $\chi^2$ with given test data, and I understand how to translate this ...
8
votes
2answers
898 views

Does an exact test always yield a higher P value then an approximated test?

I ran a simulation on that for mcnemar test, and the answer seemed to be yes. I was wondering if this can always be said to be the case that the exact P value is higher (or not smaller) then the p ...
8
votes
2answers
2k views

Why are the number of false positives independent of sample size, if we use p-values to compare two independent datasets?

If I run the R code below, it will generate two independent vectors then test them to see if they are related in some way (i.e. p-value < 0.05). If I repeat this 1,000 times, then 50 of these (5%) ...
8
votes
3answers
9k views

How to correct for small p-value due to very large sample size

I am running into a problem where an independent variable, which should have no predictive power on the dependent variable based on domain knowledge, comes out with very small p-value because the ...
8
votes
2answers
1k views

Is the type I error rate equal to alpha or at most alpha?

According the p-value Wikipedia page: When the p-value is calculated correctly, this test guarantees that the Type I error rate is at most $\alpha$. However further down the page this formula is ...
8
votes
1answer
5k views

Required number of permutations for a permutation-based p-value

If I need to calculate a permutation-based $p$-value with significance level $\alpha$, how many permutations do I need ? From the article "Permutation Tests for Studying Classifier Performance", page ...
8
votes
3answers
202 views

Why are Type-II errors in hypothesis testing considered to be errors?

I am reading about decision errors in hypothesis testing. My question is why is a "Type-II error" considered to be an error at all? From what I understand, it arises when we fail to reject a false ...

1 2
3
4 5
41