Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [combining-p-values]

Methods for combining p-values from several hypothesis tests with the same null hypothesis into one aggregate p-value, such as e.g. Fisher's method and Stouffer's method.

1
vote
1answer
38 views

Interim analysis: probability of stopping

Say we do some time-varying study. We check yearly if early stopping is necessary, which is done via interim analysis (O'Brien-Fleming stopping boundaries). The 4 p-values for stopping in 4 ...
0
votes
0answers
18 views

compute AUC and statistical testing of stratified data

I have a dataset where samples are stratified in groups. That is, there are N samples per examination, and M examinations per subject. I would like to account for this when estimating the ...
1
vote
1answer
37 views

How to test the significant of multiply chi square tests

I know that if I have a model from which I have a very large sample, so that I can create a null distribution. And I also have an experimental sample, I can use the ChiSquare test (or FisherExact test)...
1
vote
1answer
27 views

Combining the conclusions from multiple Diebold-Mariano tests

Suppose I have two forecasting models, A and B. Moreover, suppose I have multiple datasets and I use each of these sets to perform Diebold-Mariano tests. My aim is to find out which of the two models ...
3
votes
0answers
91 views

How to combine multiple p-values into one? [closed]

I have the data of 841 genes along with the p-values which represents the association of that gene in a particular tissue (Aorta in this case) with some diseased phenotype. I want to combine the p-...
28
votes
4answers
2k views

Does a uniform distribution of many p-values give statistical evidence that H0 is true?

A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
1
vote
1answer
172 views

Bonferroni-type corrections and Fisher Method Meta-analysis

I'm not an expert on the field, but I need to know more about how to combined multiple significance test p-values after a two sample Kolmogorov Smirnov (KS)test on two distributions. But I have two ...
0
votes
0answers
87 views

p value for non-independent simulations of kendall tau correlation

I want to work out Kendall's tau b for some sets of variables. In each case, I am building my sample from two sub-populations, where one of the sub-populations is larger than the other. For the ...
3
votes
1answer
478 views

Problems with Fisher's Method for combining p-values

I am using Fisher's Method to combine p-values, and have noticed some strange behavior for large p-values and large $n.$ In my case I have a large number of not statistically significant results(e.g. ...
1
vote
1answer
71 views

Rolling sum of 2 sample KS test results

In order to compare two lists of samples from 'before-treatment' and 'after-treatment', I am doing a two sample KS test using the ks_2samp function from Python's scipy.stats package which gives me the ...
0
votes
0answers
453 views

GLMM with highly unbalanced data and statistical power

Although I have used R to explain my question, I think and hope this is of general interest and is not off-topic here. I want to test whether the continuous variable “varA” is different between the ...
1
vote
1answer
387 views

How to achieve a two-sided combined p-value using Fisher's method?

Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
0
votes
1answer
54 views

Comparing 2 p values of different sample size

I did 2 linear regression analyses. White woman (WW) N1= 50 Black woman (BW) N2= 49 Correlation btw WW's V1 & V2 P = 0.001 Correlation btw BW's V1 & V2 P = 0.013 How do i know if the 2 p ...
0
votes
1answer
128 views

Probability density Function for p-value product

I am running an experiment where I get 22,152,480 p-values in a given order, ranging from 0 to 1, but strongly tilted toward 1 (i.e. not uniformly distributed). I want to say that a given series of ...
0
votes
1answer
119 views

Combining p values: what can we say about population of studies

I have N p values (N is around 200) one for each of N independent experiments in my study. These p values range from 0.03 to 0.7 (mean is 0.3), where only 5% of p values are less than 0.05. If I look ...
29
votes
3answers
2k views

Can a meta-analysis of studies which are all “not statistically signficant” lead to a “significant” conclusion?

A meta-analysis includes a bunch of studies, all of which reported a P value greater than 0.05. Is it possible for the overall meta-analysis to report a P value less than 0.05? Under what ...
1
vote
1answer
187 views

Performing a meta-analysis of wilcoxon rank sum analyses

Suppose an experiment is done and the following is presented in a paper: median for experimental, median for control, and a WRS p-value, and (n) for EXP and CTRL arms. Now suppose experiment 2 is ...
3
votes
1answer
178 views

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say?

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say? What can I say about the underlying distribution's being a Student's-$t$...
2
votes
1answer
58 views

How would you compare subgroup Stouffer Zs?

p-curve analyses (http://www.p-curve.com/) use significant p-values to estimate evidential value and effect sizes. To estimate evidential value, p-values are computed for p-values (pp-values), and ...
6
votes
0answers
48 views

Can one use Stouffer's method to integrate t-values? [duplicate]

I would like to integrate T-statistics values from multiple, independent tests. How can this be done? Since my degrees of freedom are many ($n>1000$) I was thinking to use the Stouffer's method,...
5
votes
1answer
122 views

Using p-values reported as inequalities such as $p<0.05$ in meta-analysis; can I convert them to $0.05$?

I am currently undertaking a meta-analysis of the expression profile of specific biomarkers in a specific disease. During the extraction process, I extracted the name of the biomarkers and their ...
1
vote
1answer
224 views

Testing multiple comparisons of paired proportions

I want to know if a proportion increased significantly after an event, and then find out if that holds true over multiple events. I'm using a paired t-test to measure this with continuous data (e.g. ...
1
vote
1answer
256 views

Difference between Fisher's method and the sum of chi-square test statistics

I'm learning about combining $p$-values and I have a couple of (somewhat elementary, perhaps) questions. Suppose I have $n$ independent tests, each of these to test its own null hypothesis $H_0 (i)$, $...
0
votes
2answers
511 views

How to calculate probability of success based on prior p-value

The FDA often requires a sponsor to conduct multiple clinical trials prior to approval. Given the following observations in a ph2 and ph3 trial, how would you go about predicting the probability of ...
0
votes
1answer
100 views

Combine multiple independent permutation tests

I want to find out whether a neural measure measured at 12 time-points is correlated with a behavioral measure measured on the same time-points. I have data from 20 subjects. I'm hoping to perform a ...
2
votes
1answer
355 views

Fisher's combined test vs testing uniform distribution of p-values

In testing a global null hypothesis, with independent tests, the p-values are distributed as $U(0,1)$. There are many goodness-of-fit tests to check that the distribution is uniform. For example, this ...
4
votes
0answers
149 views

Reference for tail bound of chi square distribution [closed]

Fisher's method allows to combine a set of p values $p_1,\ldots,p_n$. In order to compute the p value associated with Fisher's method one needs to compute the tail of a chi square distribution with ...
1
vote
1answer
74 views

Combining different normality transformations for Fisher's method

I have 2 data sets that are applicable to Fisher's method. One has a p-value of 0.07 and the other 0.15. Neither sets were normal or homoscedastic. The first satisfied the ANOVA assumptions only with ...
1
vote
2answers
684 views

meta-analysis of p values

I am a bit confused about doing the meta analysis with my data. Given z-scores I calculate p values in a way: p.values = 2*pnorm(-abs(z)). Then I would like to ...
3
votes
0answers
80 views

Fisher's method for P-value more conservative than OLS?

I am currently observing an interesting phenomenon in my analysis. I have a simple logistic regression model for independent Inds. The model is as follows: $$\operatorname{logit}(Y) = \beta_0+\...
5
votes
2answers
231 views

Test for significant excess of significant p-values across multiple comparisons

I have what feels like a simple question, but was unable to find answers easily. The situation Let's say I have a gene microarray dataset with tens of thousands of genes and small (<100) number ...
9
votes
2answers
8k views

R package for combining p-values using Fisher's or Stouffer's method

Is there an R package (or even a base R function) that implements Fisher's or Stouffer's method for combining p-values? Coding this should be almost trivial, but I'd rather use (and cite) a package. ...
0
votes
0answers
50 views

Test of Equal Proportions with exact p-values [duplicate]

The question "Test of Equal Proportions with zero successes" showed me that using a test of equal proportions with very low or very high numbers of successes doesn't give an accurate p-value. Below is ...
3
votes
1answer
180 views

Combining very small p-values - Fisher's method gives 0

I have 30 p-values all very small (varying between $10^{-140}$ and $10^{-110}$), and I want to combine them in some way to get a single statistic. I learned about Fisher's method, but if I apply it to ...
7
votes
1answer
187 views

What are the reasonable properties of p-value combination methods when all p-values are the same?

Suppose we are testing a null hypothesis $H_0$, by combining evidence from $n$ independent tests, using a p-value combination method $M$. The combined p-value is $M(p_1,...,p_n)$. Assume the extreme ...
8
votes
4answers
2k views

Why does Stouffer's method work?

It seems like a fairly straightforward question, but when I really think about it, Stouffer's method doesn't make sense to me. This is why: Assume a two-tailed hypothesis. You first calculate $z_i$ ...
3
votes
0answers
100 views

Methods for meta-analysis: z-transform (Stouffer's approach) vs. Fisher's method vs… infinite others? [duplicate]

Fisher's method and Stouffer's approach (z-transform) for meta-analysis roughly follow the same scheme: take the p-values from some different experiments on the same hypotheses apply to them some ...
7
votes
1answer
2k views

Why is my combined p-value, obtained using the Fisher's method, so low?

I have multiple individuals, for whom I collected two time series of some parameter. For each individual, I calculated, whether these time series are correlated. So, if I have 20 individuals, as a ...
3
votes
1answer
152 views

Is there a t-test equivalent to Stouffer's z-test?

I have $k$ pairs of samples from different distributions. For each pair of samples I want to check if the samples are taken from a normal distribution with the same mean and variance. I assume the ...
2
votes
0answers
125 views

Alternatives to Fisher's method?

I'm using the Mann-Whitney U test to compare scores from a foreground (n ranging between 5-500) to a large background (up to 2000). Since my foreground is quite small I've been taking smaller random ...
2
votes
1answer
1k views

Combining multiple p-values

Is there any statistically relevant way to combine multiple p-values measuring different hypotheses but that are all trying to measure the same thing "in spirit"? Eg, if I wanted to measure some ...
5
votes
1answer
384 views

Independent replication experiments yielding contrasting results; how to combine them?

Imagine a simple experiment, trying to answer a simple question. For example, is body temperature the same in men and in women ? To answer this question, let's say you sample 10 men, and 10 women, ...
4
votes
1answer
677 views

Integrating p-values (or z-scores) with missing values

I would like to apply a Stouffer (or Fisher) integration on a list of p-values or z-scores. However, some of them contain missing values, e.g. (R snippet): ...
6
votes
2answers
576 views

Why does Fisher's method yield $p\gg 0.5$ when combining several p-values all equal to $0.5$?

W = 0 for p in xrange(10): W += -2*log(0.5) print "W:",W final_p = 1 - scipy.stats.chi2.cdf(W, 2*10) print "Final p-value",final_p Why does this result in a ...
42
votes
3answers
13k views

When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
0
votes
1answer
1k views

MULTIPLE TESTING: Probability of specific outcomes from MULTIPLE experiments using randomized data

This is a follow-up question to this one: Significance test across multiple simulated experiments There's one answer I'm leaning towards accepting, but I wanted to make sure I understood how ...
3
votes
1answer
399 views

Significance test across multiple simulated experiments

First time question on this site, so please bear with me, thank you: I have 6 coin-flip-type experiments for which I can calculate 6 binomial p-values. I would now like to calculate the significance ...
21
votes
4answers
7k views

Stouffer's Z-score method: what if we sum $z^2$ instead of $z$?

I am performing $N$ independent statistical tests with the same null hypothesis, and would like to combine the results into one $p$-value. It seems that there are two "accepted" methods: Fisher's ...
3
votes
1answer
531 views

Distribution of p-values from multiple experiments

My experimental setup is the following: compute an optimal solution $S$ to problem $A$ on instance $I$, evaluate test statistic $f(S)$, compute $f(S_i)$ for $S_i$ $(0 \leq i \leq n)$ drawn from a ...
5
votes
1answer
7k views

Fisher's method for combing p-values - what about the lower tail?

I have a bunch of independent p-values and now I want to combine them using the Fisher's method. Each of the individual p-values is coming from a one-sided test. I am just a little bit confused about ...