Skip to main content

Questions tagged [combining-p-values]

Methods for combining p-values from several hypothesis tests with the same null hypothesis into one aggregate p-value, such as e.g. Fisher's method and Stouffer's method.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Algorithm for conducting Westfall & Young permutation test under Multiple Hypothesis Testing given a list of raw(unadjusted) p values

I am currently trying to set up a python program to implement Westfall & Young (1993)'s minP permutation procedure using Python. The computationally efficient algorithm I was able to find by ...
Suzorro's user avatar
2 votes
0 answers
33 views

Understaning Fisher’s combined probability test in my analysis

I'm trying to understand Fisher's combined probability test, especially how I used it in my analysis. I have 25 datasets, each one containing genomic data from cancer patients. I conducted a ...
Programming Noob's user avatar
0 votes
1 answer
140 views

How to combine results from Wilcoxon Rank Sum Test for multiple imputed data sets from proc MI in SAS

Endpoint information: We have seizure count collected for every day and therefore there will be some missing for some days. We got average seizure frequency per 28-day, for an interval. That is, (...
Janet Xu's user avatar
0 votes
1 answer
44 views

How can we infer from the combined p-value of Fisher's method whether the null hypotheses are all true?

Using the R package oolong (oolong), I used Fisher's method to combine the p-values of three binomial tests. I have difficulty understanding the combined p-value. I understood it to mean that the ...
Mary-Jane's user avatar
7 votes
2 answers
289 views

Measure of difference or distinctiveness that's comparable across both discrete and continuous variables

I am a product manager for a data analytics startup, and on our platform our clients end up with large tables of data about their customers which has many attributes, which are of mixed datatype: some ...
Steve Estes's user avatar
0 votes
2 answers
113 views

How can the combined p-value be larger than one of the original two p-values?

I have two p-values from independent test which I wish to combine. I do this using fisher's method, but I get some results that I don't quite understand. If I have two p-values, and combine them, this ...
Chris_abc's user avatar
  • 111
-1 votes
1 answer
77 views

difference between two sets of pvalues

I am performing a biological experiment, where I am trying to capture potential mediation effects from some genes. I have 150 significant results, containing 30 tissues with different sample size. For ...
dsbo's user avatar
  • 11
0 votes
1 answer
73 views

How can I get a p-value for a distribution that isn't normally distributed?

I have four independent trials of a random variable. I have run a Shapiro-Wilk test on it and it is not normally distributed. I have two different images of a peak. The background is flat with a ...
zaphod's user avatar
  • 13
0 votes
1 answer
27 views

Combining cause-specific mortality cox regressions into all-cause mortality

I only have access to the following dataset summary data and not to the dataset itself. In the example, the stats on each line come from a univariate cox regression to model a cause-specific ...
DGB's user avatar
  • 3
0 votes
0 answers
66 views

Combined p-value from wilcoxon test result on control experiment - interpreting the null-hypothesis correctly

EDITED I spotted error in my code, and now results are no more contradictory. But still I keep the question for utility of others and myself in understanding meta-analysis and correctly interpret null-...
user305883's user avatar
0 votes
0 answers
43 views

Combine p-values of several independent samples for combined hypothesis

I have a dataframe with 4 columns, corresponding to different emotions extracted with ML methods from one video. The dataframe describes one person, rows represent moments in time. Now I'm trying to ...
SUTerliakov's user avatar
1 vote
1 answer
40 views

How to combine p-values of weighted effects?

Suppose I know the effect of running on strength of muscle 1 and 2 and their $p$-values. These values were calculated from the same sample. Now I want to know the effect of running on the strength of ...
John Smith's user avatar
2 votes
1 answer
327 views

What can I do if I want to apply Fisher's method (or some other method of combining p-values), but my p-values have unknown (positive) dependence?

I have p-values from several tests (k~50) that I'd like to combine. I don't have a model of their correlation structure. They aren't independent. In the worst-case scenario, my effective sample size ...
MRule's user avatar
  • 261
1 vote
0 answers
20 views

Combining association p-values from different H0 families

I have a pipeline that finds pairwise associations between three objects that are in a relationship as follows: B / \ A --- C What I get from this ...
Daniil Khlebnikov's user avatar
1 vote
1 answer
278 views

Ranking genes within GSEA

I know you can rank genes by p-/q-value, or by log2 fold change. But can you combine the two methods somehow? I have read about combining fold change and p-value here but what is one advantage and one ...
katara 's user avatar
  • 111
0 votes
1 answer
53 views

Pearson chi squared for correlated sample

I am looking into the STS statistical test for random sequence. There are multiple statistical tests for a random bit sequence and each of them produces a p-value or multiple p-value. After the test ...
nir's user avatar
  • 103
1 vote
1 answer
34 views

Significance test for a pattern in union of sets given that the pattern is significant in the individual sets

We have a pattern like $P = \{f_{A},f_{B}\}$ which is statistically significant in multiple sets $S = \{S_{1}, S_{2}, S_{3}, ...\}$. We can think of the features $f_{i}$ to be retail establishments &...
gis_grad_student's user avatar
4 votes
1 answer
481 views

Is this ok to average p values?

I have seen other posts about averaging p values. I believe I have a special case where average may apply. Finding the exact probability distribution of my data is an NP hard task, so I am using a non ...
decapicone's user avatar
0 votes
0 answers
436 views

What would you call combining multiple tests of a single hypothesis?

Multiple testing refers to testing more than one hypothesis simultaneously. The situation typically involves having several tests that cannot be combined into a single test, and therefore inference is ...
Galton's user avatar
  • 101
5 votes
1 answer
173 views

Complementary p values do not cancel in many combining methods – why is this not a big deal?

What is this about? Suppose I have performed two statistical tests (with a continuous distribution of $p$ values) for the same one-sided research hypothesis on different datasets, yielding the $p$ ...
Wrzlprmft's user avatar
  • 2,291
1 vote
0 answers
102 views

How to combine test statistics and p-values from the same (stochastic) test repeated on a single dataset?

I would like to assess a generative model $M$'s goodness-of-fit using a simulation-based approach: I simulate some data from $M$, then compute a test statistic $T$ that measures how similar the ...
cjurban's user avatar
  • 11
1 vote
1 answer
29 views

Hypothesis testing with p-values from similar tests as prior

I hope to work on a hypothesis testing problem to test $\beta$ against the null hypothesis ($H_{null}$) $\beta=0$, and we can consider Wald test with the test statistics $$ W = \dfrac{\beta^2}{var(\...
user avatar
1 vote
0 answers
48 views

Test for change in all observerd variables

Let us say, I observe 3 variables in a control condition and a treatment condition. I would like to find out whether the treatment has some effect on all 3 variables at the same time. I should mention ...
Fabian Rost's user avatar
1 vote
1 answer
145 views

How to summarize multiple independent false discovery rates (FDRs or q-values)?

In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
user1799035's user avatar
4 votes
0 answers
79 views

Sequential hypothesis test for uniform distribution

Suppose I have a sequence of data points $x_1,x_2,\dots$. I want to do a sequential hypothesis test, were the null hypothesis is that each $x_i$ is iid uniform on $[0,1]$, and the alternative ...
D.W.'s user avatar
  • 6,678
4 votes
1 answer
341 views

How to interpret the harmonic mean p-value?

I am working on a multiple comparisons problem for several hypothesis tests and have come across the harmonic mean $p$-value (HMP) for combining the $p$-value of these tests. Specifically, I am using ...
hoffee's user avatar
  • 141
1 vote
0 answers
259 views

Combining tests: Maximum test statistic method but one test is a permutation test

I want to combine two tests $A,B$ for the same null and alternative hypotheses. The motivation is that it has been shown that test $A$ has higher power in some situations whereas test $B$ has higher ...
Julian Karch's user avatar
  • 1,890
0 votes
1 answer
321 views

Combining p-values from permutation tests from single subjects

I have a question that's very similar to this question: Combine multiple independent permutation tests My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the ...
user3640617's user avatar
3 votes
1 answer
176 views

combined p values from different hypothesis

I am currently working on a project in which I'm simultaneously identifying and quantifying protein in k samples. Experimentally, for each of the k samples, I have n identified proteins, with a p ...
Gonçalo Justino's user avatar
1 vote
1 answer
25 views

After derivng a test statistc using p-value combination method, what is the associat null hypothesis

First I generate two one-sided p-values from independent samples using null hypothesis $H_0: \Delta\leq0$, so that $p_1$ and $p_2$ are generated. Next, I apply the inverse normal combination function $...
RGG's user avatar
  • 73
2 votes
1 answer
272 views

Fisher's method for combining p-values

I'm using the sumlog function in the metap package of R, and have a few questions about the nuts-and-bolts of this function: 1)Some of my p-values are reported as '<0.001'. How do I include these?...
Keno's user avatar
  • 63
1 vote
1 answer
264 views

Meta-analysis with only p-values

I am doing a systematic review on the risk factors that predict infection after liver and pancreatic surgery. These risk factors include parameters like age, sex, smoking, pre-operative chemo etc. I ...
Keno's user avatar
  • 63
0 votes
0 answers
31 views

Cobine p-values, same data, same test, different labels

I have a dataset (300 samples vs 4000 features), I'm trying to extract meaningful features related to two condition A and B. ...
Giuseppe Minardi's user avatar
2 votes
3 answers
310 views

Combining p-values when the trials are independent but their number is data-dependent and random

Consider the following scenario (it's just a motivating example, not something I am doing for real): I run a trial, drawing an i.i.d. sample $S_1$ from population $P$, to test a hypothesis $\mathcal{H}...
quant_dev's user avatar
  • 664
2 votes
1 answer
196 views

Test for multiple samples of ordinal data

I have 10 independent samples of ordinal data ranked 1-3. I also have predicted ranks for the same data set. I want to test if the predicted ranking matches the observed ranking at a rate better ...
Barker's user avatar
  • 1,202
1 vote
1 answer
44 views

Marginalizing statistical test results in a two-factor grid

Consider the following toy problem. Suppose that we have several food groups: meat, vegetables, fruit, etc. For any given pair of food types A and B, we ask whether people that eat both A AND B have a ...
Artem Sokolov's user avatar
1 vote
1 answer
99 views

Interim analysis: probability of stopping

Say we do some time-varying study. We check yearly if early stopping is necessary, which is done via interim analysis (O'Brien-Fleming stopping boundaries). The 4 p-values for stopping in 4 ...
xzeo's user avatar
  • 206
1 vote
1 answer
62 views

How to test the significant of multiply chi square tests

I know that if I have a model from which I have a very large sample, so that I can create a null distribution. And I also have an experimental sample, I can use the ChiSquare test (or FisherExact test)...
Oren's user avatar
  • 143
2 votes
1 answer
71 views

Combining the conclusions from multiple Diebold-Mariano tests

Suppose I have two forecasting models, A and B. Moreover, suppose I have multiple datasets and I use each of these sets to perform Diebold-Mariano tests. My aim is to find out which of the two models ...
user120911's user avatar
2 votes
0 answers
772 views

How to combine multiple p-values into one? [closed]

I have the data of 841 genes along with the p-values which represents the association of that gene in a particular tissue (Aorta in this case) with some diseased phenotype. I want to combine the p-...
Star's user avatar
  • 21
30 votes
4 answers
4k views

Does a uniform distribution of many p-values give statistical evidence that H0 is true?

A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
Leander Moesinger's user avatar
2 votes
1 answer
957 views

Bonferroni-type corrections and Fisher Method Meta-analysis

I'm not an expert on the field, but I need to know more about how to combined multiple significance test p-values after a two sample Kolmogorov Smirnov (KS)test on two distributions. But I have two ...
FabIO's user avatar
  • 23
4 votes
1 answer
3k views

Problems with Fisher's Method for combining p-values

I am using Fisher's Method to combine p-values, and have noticed some strange behavior for large p-values and large $n.$ In my case I have a large number of not statistically significant results(e.g. ...
Josh's user avatar
  • 527
2 votes
2 answers
257 views

Rolling sum of 2 sample KS test results

In order to compare two lists of samples from 'before-treatment' and 'after-treatment', I am doing a two sample KS test using the ks_2samp function from Python's scipy.stats package which gives me the ...
Aisha's user avatar
  • 23
0 votes
0 answers
1k views

GLMM with highly unbalanced data and statistical power

Although I have used R to explain my question, I think and hope this is of general interest and is not off-topic here. I want to test whether the continuous variable “varA” is different between the ...
user3844454's user avatar
2 votes
1 answer
1k views

How to achieve a two-sided combined p-value using Fisher's method?

Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
Brandon's user avatar
  • 356
1 vote
1 answer
268 views

Comparing 2 p values of different sample size

I did 2 linear regression analyses. White woman (WW) N1= 50 Black woman (BW) N2= 49 Correlation btw WW's V1 & V2 P = 0.001 Correlation btw BW's V1 & V2 P = 0.013 How do i know if the 2 p ...
tsumaranaina's user avatar
0 votes
1 answer
491 views

Probability density Function for p-value product

I am running an experiment where I get 22,152,480 p-values in a given order, ranging from 0 to 1, but strongly tilted toward 1 (i.e. not uniformly distributed). I want to say that a given series of ...
con's user avatar
  • 133
0 votes
1 answer
186 views

Combining p values: what can we say about population of studies

I have N p values (N is around 200) one for each of N independent experiments in my study. These p values range from 0.03 to 0.7 (mean is 0.3), where only 5% of p values are less than 0.05. If I look ...
M_S's user avatar
  • 11
31 votes
3 answers
5k views

Can a meta-analysis of studies which are all "not statistically signficant" lead to a "significant" conclusion?

A meta-analysis includes a bunch of studies, all of which reported a P value greater than 0.05. Is it possible for the overall meta-analysis to report a P value less than 0.05? Under what ...
Harvey Motulsky's user avatar