# Questions tagged [combining-p-values]

Methods for combining p-values from several hypothesis tests with the same null hypothesis into one aggregate p-value, such as e.g. Fisher's method and Stouffer's method.

93 questions
Filter by
Sorted by
Tagged with
25 views

### Algorithm for conducting Westfall & Young permutation test under Multiple Hypothesis Testing given a list of raw(unadjusted) p values

I am currently trying to set up a python program to implement Westfall & Young (1993)'s minP permutation procedure using Python. The computationally efficient algorithm I was able to find by ...
33 views

### Understaning Fisher’s combined probability test in my analysis

I'm trying to understand Fisher's combined probability test, especially how I used it in my analysis. I have 25 datasets, each one containing genomic data from cancer patients. I conducted a ...
139 views

### How to combine results from Wilcoxon Rank Sum Test for multiple imputed data sets from proc MI in SAS

Endpoint information: We have seizure count collected for every day and therefore there will be some missing for some days. We got average seizure frequency per 28-day, for an interval. That is, (...
44 views

### How can we infer from the combined p-value of Fisher's method whether the null hypotheses are all true?

Using the R package oolong (oolong), I used Fisher's method to combine the p-values of three binomial tests. I have difficulty understanding the combined p-value. I understood it to mean that the ...
288 views

### Measure of difference or distinctiveness that's comparable across both discrete and continuous variables

I am a product manager for a data analytics startup, and on our platform our clients end up with large tables of data about their customers which has many attributes, which are of mixed datatype: some ...
• 171
112 views

### How can the combined p-value be larger than one of the original two p-values?

I have two p-values from independent test which I wish to combine. I do this using fisher's method, but I get some results that I don't quite understand. If I have two p-values, and combine them, this ...
• 111
77 views

### difference between two sets of pvalues

I am performing a biological experiment, where I am trying to capture potential mediation effects from some genes. I have 150 significant results, containing 30 tissues with different sample size. For ...
• 11
72 views

### How can I get a p-value for a distribution that isn't normally distributed?

I have four independent trials of a random variable. I have run a Shapiro-Wilk test on it and it is not normally distributed. I have two different images of a peak. The background is flat with a ...
• 13
26 views

### Combining cause-specific mortality cox regressions into all-cause mortality

I only have access to the following dataset summary data and not to the dataset itself. In the example, the stats on each line come from a univariate cox regression to model a cause-specific ...
• 3
66 views

### Combined p-value from wilcoxon test result on control experiment - interpreting the null-hypothesis correctly

EDITED I spotted error in my code, and now results are no more contradictory. But still I keep the question for utility of others and myself in understanding meta-analysis and correctly interpret null-...
• 111
43 views

### Combine p-values of several independent samples for combined hypothesis

I have a dataframe with 4 columns, corresponding to different emotions extracted with ML methods from one video. The dataframe describes one person, rows represent moments in time. Now I'm trying to ...
• 101
1 vote
39 views

### How to combine p-values of weighted effects?

Suppose I know the effect of running on strength of muscle 1 and 2 and their $p$-values. These values were calculated from the same sample. Now I want to know the effect of running on the strength of ...
322 views

### What can I do if I want to apply Fisher's method (or some other method of combining p-values), but my p-values have unknown (positive) dependence?

I have p-values from several tests (k~50) that I'd like to combine. I don't have a model of their correlation structure. They aren't independent. In the worst-case scenario, my effective sample size ...
• 261
1 vote
20 views

### Combining association p-values from different H0 families

I have a pipeline that finds pairwise associations between three objects that are in a relationship as follows: B / \ A --- C What I get from this ...
1 vote
265 views

### Ranking genes within GSEA

I know you can rank genes by p-/q-value, or by log2 fold change. But can you combine the two methods somehow? I have read about combining fold change and p-value here but what is one advantage and one ...
• 111
53 views

### Pearson chi squared for correlated sample

I am looking into the STS statistical test for random sequence. There are multiple statistical tests for a random bit sequence and each of them produces a p-value or multiple p-value. After the test ...
• 103
1 vote
34 views

### Significance test for a pattern in union of sets given that the pattern is significant in the individual sets

We have a pattern like $P = \{f_{A},f_{B}\}$ which is statistically significant in multiple sets $S = \{S_{1}, S_{2}, S_{3}, ...\}$. We can think of the features $f_{i}$ to be retail establishments &...
476 views

### Is this ok to average p values?

I have seen other posts about averaging p values. I believe I have a special case where average may apply. Finding the exact probability distribution of my data is an NP hard task, so I am using a non ...
• 183
435 views

### What would you call combining multiple tests of a single hypothesis?

Multiple testing refers to testing more than one hypothesis simultaneously. The situation typically involves having several tests that cannot be combined into a single test, and therefore inference is ...
• 101
173 views

### Complementary p values do not cancel in many combining methods – why is this not a big deal?

What is this about? Suppose I have performed two statistical tests (with a continuous distribution of $p$ values) for the same one-sided research hypothesis on different datasets, yielding the $p$ ...
• 2,291
1 vote
102 views

### How to combine test statistics and p-values from the same (stochastic) test repeated on a single dataset?

I would like to assess a generative model $M$'s goodness-of-fit using a simulation-based approach: I simulate some data from $M$, then compute a test statistic $T$ that measures how similar the ...
• 11
1 vote
29 views

### Hypothesis testing with p-values from similar tests as prior

I hope to work on a hypothesis testing problem to test $\beta$ against the null hypothesis ($H_{null}$) $\beta=0$, and we can consider Wald test with the test statistics  W = \dfrac{\beta^2}{var(\...
1 vote
48 views

### Test for change in all observerd variables

Let us say, I observe 3 variables in a control condition and a treatment condition. I would like to find out whether the treatment has some effect on all 3 variables at the same time. I should mention ...
• 173
1 vote
145 views

### How to summarize multiple independent false discovery rates (FDRs or q-values)?

In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
79 views

### Sequential hypothesis test for uniform distribution

Suppose I have a sequence of data points $x_1,x_2,\dots$. I want to do a sequential hypothesis test, were the null hypothesis is that each $x_i$ is iid uniform on $[0,1]$, and the alternative ...
• 6,668
339 views

### How to interpret the harmonic mean p-value?

I am working on a multiple comparisons problem for several hypothesis tests and have come across the harmonic mean $p$-value (HMP) for combining the $p$-value of these tests. Specifically, I am using ...
• 141
1 vote
259 views

### Combining tests: Maximum test statistic method but one test is a permutation test

I want to combine two tests $A,B$ for the same null and alternative hypotheses. The motivation is that it has been shown that test $A$ has higher power in some situations whereas test $B$ has higher ...
• 1,890
321 views

### Combining p-values from permutation tests from single subjects

I have a question that's very similar to this question: Combine multiple independent permutation tests My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the ...
• 165
176 views

### combined p values from different hypothesis

I am currently working on a project in which I'm simultaneously identifying and quantifying protein in k samples. Experimentally, for each of the k samples, I have n identified proteins, with a p ...
1 vote
25 views

• 664
195 views

### Test for multiple samples of ordinal data

I have 10 independent samples of ordinal data ranked 1-3. I also have predicted ranks for the same data set. I want to test if the predicted ranking matches the observed ranking at a rate better ...
• 1,202
1 vote
44 views

### Marginalizing statistical test results in a two-factor grid

Consider the following toy problem. Suppose that we have several food groups: meat, vegetables, fruit, etc. For any given pair of food types A and B, we ask whether people that eat both A AND B have a ...
1 vote
98 views

### Interim analysis: probability of stopping

Say we do some time-varying study. We check yearly if early stopping is necessary, which is done via interim analysis (O'Brien-Fleming stopping boundaries). The 4 p-values for stopping in 4 ...
• 206
1 vote
62 views

### How to test the significant of multiply chi square tests

I know that if I have a model from which I have a very large sample, so that I can create a null distribution. And I also have an experimental sample, I can use the ChiSquare test (or FisherExact test)...
• 143
71 views

### Combining the conclusions from multiple Diebold-Mariano tests

Suppose I have two forecasting models, A and B. Moreover, suppose I have multiple datasets and I use each of these sets to perform Diebold-Mariano tests. My aim is to find out which of the two models ...
• 475
771 views

### How to combine multiple p-values into one? [closed]

I have the data of 841 genes along with the p-values which represents the association of that gene in a particular tissue (Aorta in this case) with some diseased phenotype. I want to combine the p-...
• 21
4k views

### Does a uniform distribution of many p-values give statistical evidence that H0 is true?

A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
956 views

### Bonferroni-type corrections and Fisher Method Meta-analysis

I'm not an expert on the field, but I need to know more about how to combined multiple significance test p-values after a two sample Kolmogorov Smirnov (KS)test on two distributions. But I have two ...
• 23
3k views

### Problems with Fisher's Method for combining p-values

I am using Fisher's Method to combine p-values, and have noticed some strange behavior for large p-values and large $n.$ In my case I have a large number of not statistically significant results(e.g. ...
• 517
257 views

### Rolling sum of 2 sample KS test results

In order to compare two lists of samples from 'before-treatment' and 'after-treatment', I am doing a two sample KS test using the ks_2samp function from Python's scipy.stats package which gives me the ...
• 23
1k views

### GLMM with highly unbalanced data and statistical power

Although I have used R to explain my question, I think and hope this is of general interest and is not off-topic here. I want to test whether the continuous variable “varA” is different between the ...
1k views

### How to achieve a two-sided combined p-value using Fisher's method?

Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
• 356
1 vote
268 views

### Comparing 2 p values of different sample size

I did 2 linear regression analyses. White woman (WW) N1= 50 Black woman (BW) N2= 49 Correlation btw WW's V1 & V2 P = 0.001 Correlation btw BW's V1 & V2 P = 0.013 How do i know if the 2 p ...
• 211
491 views

### Probability density Function for p-value product

I am running an experiment where I get 22,152,480 p-values in a given order, ranging from 0 to 1, but strongly tilted toward 1 (i.e. not uniformly distributed). I want to say that a given series of ...
• 133
186 views

### Combining p values: what can we say about population of studies

I have N p values (N is around 200) one for each of N independent experiments in my study. These p values range from 0.03 to 0.7 (mean is 0.3), where only 5% of p values are less than 0.05. If I look ...
• 11