Skip to main content

Questions tagged [combining-p-values]

Methods for combining p-values from several hypothesis tests with the same null hypothesis into one aggregate p-value, such as e.g. Fisher's method and Stouffer's method.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

How the combined test under Fisher's method is generally stronger than individual tests, mathematically?

I have gone through the definition of Fisher's method in several materials but it is unclear how Fisher's method is stronger than individual tests. I guess it is not always the case, i.e., there are ...
fermented_bean's user avatar
2 votes
2 answers
82 views

FDR in a gene expression experiment with two similar treatments

Given is the following experiment. Gene expression profiles across thousands of genes are obtained for three conditions: two drugs (= treatments) with a similar phenotypic effect and a control group. ...
alephreish's user avatar
0 votes
0 answers
35 views

How can I combine p-values estimated for individual variables within a single dataset using different statistical methods?

I'm analyzing a multivariate dataset with two classes (case/control). I've calculated p-values for each variable to test their discrimination perfomance using two different methods: 1) univariate t-...
Guillermo Q's user avatar
0 votes
0 answers
44 views

Algorithm for conducting Westfall & Young permutation test under Multiple Hypothesis Testing given a list of raw(unadjusted) p values

I am currently trying to set up a python program to implement Westfall & Young (1993)'s minP permutation procedure using Python. The computationally efficient algorithm I was able to find by ...
Suzorro's user avatar
2 votes
0 answers
41 views

Understaning Fisher’s combined probability test in my analysis

I'm trying to understand Fisher's combined probability test, especially how I used it in my analysis. I have 25 datasets, each one containing genomic data from cancer patients. I conducted a ...
Programming Noob's user avatar
0 votes
1 answer
189 views

How to combine results from Wilcoxon Rank Sum Test for multiple imputed data sets from proc MI in SAS

Endpoint information: We have seizure count collected for every day and therefore there will be some missing for some days. We got average seizure frequency per 28-day, for an interval. That is, (...
Janet Xu's user avatar
0 votes
1 answer
49 views

How can we infer from the combined p-value of Fisher's method whether the null hypotheses are all true?

Using the R package oolong (oolong), I used Fisher's method to combine the p-values of three binomial tests. I have difficulty understanding the combined p-value. I understood it to mean that the ...
Mary-Jane's user avatar
7 votes
2 answers
296 views

Measure of difference or distinctiveness that's comparable across both discrete and continuous variables

I am a product manager for a data analytics startup, and on our platform our clients end up with large tables of data about their customers which has many attributes, which are of mixed datatype: some ...
Steve Estes's user avatar
0 votes
2 answers
138 views

How can the combined p-value be larger than one of the original two p-values?

I have two p-values from independent test which I wish to combine. I do this using fisher's method, but I get some results that I don't quite understand. If I have two p-values, and combine them, this ...
Chris_abc's user avatar
  • 111
-1 votes
1 answer
77 views

difference between two sets of pvalues

I am performing a biological experiment, where I am trying to capture potential mediation effects from some genes. I have 150 significant results, containing 30 tissues with different sample size. For ...
dsbo's user avatar
  • 11
0 votes
1 answer
78 views

How can I get a p-value for a distribution that isn't normally distributed?

I have four independent trials of a random variable. I have run a Shapiro-Wilk test on it and it is not normally distributed. I have two different images of a peak. The background is flat with a ...
zaphod's user avatar
  • 13
0 votes
1 answer
27 views

Combining cause-specific mortality cox regressions into all-cause mortality

I only have access to the following dataset summary data and not to the dataset itself. In the example, the stats on each line come from a univariate cox regression to model a cause-specific ...
DGB's user avatar
  • 3
0 votes
0 answers
73 views

Combined p-value from wilcoxon test result on control experiment - interpreting the null-hypothesis correctly

EDITED I spotted error in my code, and now results are no more contradictory. But still I keep the question for utility of others and myself in understanding meta-analysis and correctly interpret null-...
user305883's user avatar
0 votes
0 answers
46 views

Combine p-values of several independent samples for combined hypothesis

I have a dataframe with 4 columns, corresponding to different emotions extracted with ML methods from one video. The dataframe describes one person, rows represent moments in time. Now I'm trying to ...
SUTerliakov's user avatar
1 vote
1 answer
40 views

How to combine p-values of weighted effects?

Suppose I know the effect of running on strength of muscle 1 and 2 and their $p$-values. These values were calculated from the same sample. Now I want to know the effect of running on the strength of ...
John Smith's user avatar
2 votes
1 answer
404 views

What can I do if I want to apply Fisher's method (or some other method of combining p-values), but my p-values have unknown (positive) dependence?

I have p-values from several tests (k~50) that I'd like to combine. I don't have a model of their correlation structure. They aren't independent. In the worst-case scenario, my effective sample size ...
MRule's user avatar
  • 261
1 vote
0 answers
21 views

Combining association p-values from different H0 families

I have a pipeline that finds pairwise associations between three objects that are in a relationship as follows: B / \ A --- C What I get from this ...
Daniil Khlebnikov's user avatar
1 vote
1 answer
386 views

Ranking genes within GSEA

I know you can rank genes by p-/q-value, or by log2 fold change. But can you combine the two methods somehow? I have read about combining fold change and p-value here but what is one advantage and one ...
katara 's user avatar
  • 111
0 votes
1 answer
55 views

Pearson chi squared for correlated sample

I am looking into the STS statistical test for random sequence. There are multiple statistical tests for a random bit sequence and each of them produces a p-value or multiple p-value. After the test ...
nir's user avatar
  • 103
1 vote
1 answer
34 views

Significance test for a pattern in union of sets given that the pattern is significant in the individual sets

We have a pattern like $P = \{f_{A},f_{B}\}$ which is statistically significant in multiple sets $S = \{S_{1}, S_{2}, S_{3}, ...\}$. We can think of the features $f_{i}$ to be retail establishments &...
gis_grad_student's user avatar
4 votes
1 answer
588 views

Is this ok to average p values?

I have seen other posts about averaging p values. I believe I have a special case where average may apply. Finding the exact probability distribution of my data is an NP hard task, so I am using a non ...
decapicone's user avatar
0 votes
0 answers
479 views

What would you call combining multiple tests of a single hypothesis?

Multiple testing refers to testing more than one hypothesis simultaneously. The situation typically involves having several tests that cannot be combined into a single test, and therefore inference is ...
Galton's user avatar
  • 101
5 votes
1 answer
176 views

Complementary p values do not cancel in many combining methods – why is this not a big deal?

What is this about? Suppose I have performed two statistical tests (with a continuous distribution of $p$ values) for the same one-sided research hypothesis on different datasets, yielding the $p$ ...
Wrzlprmft's user avatar
  • 2,291
1 vote
0 answers
109 views

How to combine test statistics and p-values from the same (stochastic) test repeated on a single dataset?

I would like to assess a generative model $M$'s goodness-of-fit using a simulation-based approach: I simulate some data from $M$, then compute a test statistic $T$ that measures how similar the ...
cjurban's user avatar
  • 11
1 vote
1 answer
30 views

Hypothesis testing with p-values from similar tests as prior

I hope to work on a hypothesis testing problem to test $\beta$ against the null hypothesis ($H_{null}$) $\beta=0$, and we can consider Wald test with the test statistics $$ W = \dfrac{\beta^2}{var(\...
user avatar
1 vote
0 answers
49 views

Test for change in all observerd variables

Let us say, I observe 3 variables in a control condition and a treatment condition. I would like to find out whether the treatment has some effect on all 3 variables at the same time. I should mention ...
Fabian Rost's user avatar
1 vote
1 answer
152 views

How to summarize multiple independent false discovery rates (FDRs or q-values)?

In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
user1799035's user avatar
4 votes
0 answers
81 views

Sequential hypothesis test for uniform distribution

Suppose I have a sequence of data points $x_1,x_2,\dots$. I want to do a sequential hypothesis test, were the null hypothesis is that each $x_i$ is iid uniform on $[0,1]$, and the alternative ...
D.W.'s user avatar
  • 6,688
4 votes
1 answer
362 views

How to interpret the harmonic mean p-value?

I am working on a multiple comparisons problem for several hypothesis tests and have come across the harmonic mean $p$-value (HMP) for combining the $p$-value of these tests. Specifically, I am using ...
hoffee's user avatar
  • 141
1 vote
0 answers
285 views

Combining tests: Maximum test statistic method but one test is a permutation test

I want to combine two tests $A,B$ for the same null and alternative hypotheses. The motivation is that it has been shown that test $A$ has higher power in some situations whereas test $B$ has higher ...
Julian Karch's user avatar
  • 1,980
0 votes
1 answer
331 views

Combining p-values from permutation tests from single subjects

I have a question that's very similar to this question: Combine multiple independent permutation tests My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the ...
user3640617's user avatar
3 votes
1 answer
182 views

combined p values from different hypothesis

I am currently working on a project in which I'm simultaneously identifying and quantifying protein in k samples. Experimentally, for each of the k samples, I have n identified proteins, with a p ...
Gonçalo Justino's user avatar
1 vote
1 answer
26 views

After derivng a test statistc using p-value combination method, what is the associat null hypothesis

First I generate two one-sided p-values from independent samples using null hypothesis $H_0: \Delta\leq0$, so that $p_1$ and $p_2$ are generated. Next, I apply the inverse normal combination function $...
RGG's user avatar
  • 73
2 votes
1 answer
279 views

Fisher's method for combining p-values

I'm using the sumlog function in the metap package of R, and have a few questions about the nuts-and-bolts of this function: 1)Some of my p-values are reported as '<0.001'. How do I include these?...
Keno's user avatar
  • 63
1 vote
1 answer
274 views

Meta-analysis with only p-values

I am doing a systematic review on the risk factors that predict infection after liver and pancreatic surgery. These risk factors include parameters like age, sex, smoking, pre-operative chemo etc. I ...
Keno's user avatar
  • 63
0 votes
0 answers
32 views

Cobine p-values, same data, same test, different labels

I have a dataset (300 samples vs 4000 features), I'm trying to extract meaningful features related to two condition A and B. ...
Giuseppe Minardi's user avatar
2 votes
3 answers
340 views

Combining p-values when the trials are independent but their number is data-dependent and random

Consider the following scenario (it's just a motivating example, not something I am doing for real): I run a trial, drawing an i.i.d. sample $S_1$ from population $P$, to test a hypothesis $\mathcal{H}...
quant_dev's user avatar
  • 664
2 votes
1 answer
200 views

Test for multiple samples of ordinal data

I have 10 independent samples of ordinal data ranked 1-3. I also have predicted ranks for the same data set. I want to test if the predicted ranking matches the observed ranking at a rate better ...
Barker's user avatar
  • 1,212
1 vote
1 answer
44 views

Marginalizing statistical test results in a two-factor grid

Consider the following toy problem. Suppose that we have several food groups: meat, vegetables, fruit, etc. For any given pair of food types A and B, we ask whether people that eat both A AND B have a ...
Artem Sokolov's user avatar
1 vote
1 answer
101 views

Interim analysis: probability of stopping

Say we do some time-varying study. We check yearly if early stopping is necessary, which is done via interim analysis (O'Brien-Fleming stopping boundaries). The 4 p-values for stopping in 4 ...
xzeo's user avatar
  • 206
1 vote
1 answer
62 views

How to test the significant of multiply chi square tests

I know that if I have a model from which I have a very large sample, so that I can create a null distribution. And I also have an experimental sample, I can use the ChiSquare test (or FisherExact test)...
Oren's user avatar
  • 143
2 votes
1 answer
85 views

Combining the conclusions from multiple Diebold-Mariano tests

Suppose I have two forecasting models, A and B. Moreover, suppose I have multiple datasets and I use each of these sets to perform Diebold-Mariano tests. My aim is to find out which of the two models ...
user120911's user avatar
2 votes
0 answers
786 views

How to combine multiple p-values into one? [closed]

I have the data of 841 genes along with the p-values which represents the association of that gene in a particular tissue (Aorta in this case) with some diseased phenotype. I want to combine the p-...
Star's user avatar
  • 21
30 votes
4 answers
4k views

Does a uniform distribution of many p-values give statistical evidence that H0 is true?

A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
Leander Moesinger's user avatar
2 votes
1 answer
982 views

Bonferroni-type corrections and Fisher Method Meta-analysis

I'm not an expert on the field, but I need to know more about how to combined multiple significance test p-values after a two sample Kolmogorov Smirnov (KS)test on two distributions. But I have two ...
FabIO's user avatar
  • 23
4 votes
1 answer
3k views

Problems with Fisher's Method for combining p-values

I am using Fisher's Method to combine p-values, and have noticed some strange behavior for large p-values and large $n.$ In my case I have a large number of not statistically significant results(e.g. ...
Josh's user avatar
  • 527
2 votes
2 answers
259 views

Rolling sum of 2 sample KS test results

In order to compare two lists of samples from 'before-treatment' and 'after-treatment', I am doing a two sample KS test using the ks_2samp function from Python's scipy.stats package which gives me the ...
Aisha's user avatar
  • 23
0 votes
0 answers
1k views

GLMM with highly unbalanced data and statistical power

Although I have used R to explain my question, I think and hope this is of general interest and is not off-topic here. I want to test whether the continuous variable “varA” is different between the ...
user3844454's user avatar
2 votes
1 answer
2k views

How to achieve a two-sided combined p-value using Fisher's method?

Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
Brandon's user avatar
  • 356
1 vote
1 answer
269 views

Comparing 2 p values of different sample size

I did 2 linear regression analyses. White woman (WW) N1= 50 Black woman (BW) N2= 49 Correlation btw WW's V1 & V2 P = 0.001 Correlation btw BW's V1 & V2 P = 0.013 How do i know if the 2 p ...
tsumaranaina's user avatar