Questions tagged [combining-p-values]

Methods for combining p-values from several hypothesis tests with the same null hypothesis into one aggregate p-value, such as e.g. Fisher's method and Stouffer's method.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Calculate P values for regression models(Linear,Lasso Ridge,Elasticnet) manually [closed]

I want to calculate P value manually from the Multivariate Regression models(Lasso,ridge,Linear and Elasticnet) and i had tried to use the following code as shown below as one person suggested using ...
user avatar
4 votes
1 answer
98 views

Complementary p values do not cancel in many combining methods – why is this not a big deal?

What is this about? Suppose I have performed two statistical tests (with a continuous distribution of $p$ values) for the same one-sided research hypothesis on different datasets, yielding the $p$ ...
user avatar
  • 2,307
0 votes
0 answers
26 views

How to interpret the results of q value

I conducted multiple tests and had a group of P values. I used the R function qvalue() to calculate the corresponding qvalues and plot it. https://www.rdocumentation.org/packages/qvalue/versions/2.4.2/...
user avatar
1 vote
0 answers
31 views

How to combine test statistics and p-values from the same (stochastic) test repeated on a single dataset?

I would like to assess a generative model $M$'s goodness-of-fit using a simulation-based approach: I simulate some data from $M$, then compute a test statistic $T$ that measures how similar the ...
user avatar
  • 11
0 votes
0 answers
24 views

Ordering drug effectiveness from best to worst

Problem The (mean) effectiveness of a drug $d$ on trial $i$ is given by \begin{align} X_{di}:=\widehat{\mu}_{di} \sim \mathcal{N}(\mu_{di}, \sigma_{di}^2), \end{align} with known variances. Hence, for ...
user avatar
1 vote
1 answer
24 views

Hypothesis testing with p-values from similar tests as prior

I hope to work on a hypothesis testing problem to test $\beta$ against the null hypothesis ($H_{null}$) $\beta=0$, and we can consider Wald test with the test statistics $$ W = \dfrac{\beta^2}{var(\...
user avatar
0 votes
0 answers
24 views

Independence condition for Fisher's p-value method

Before describing how this relates to fisher's method, let me provide some background: I have a set of K genes. For each pair of genes, I can calculate a p-value for "probability that they ...
user avatar
  • 1
1 vote
0 answers
28 views

Test for change in all observerd variables

Let us say, I observe 3 variables in a control condition and a treatment condition. I would like to find out whether the treatment has some effect on all 3 variables at the same time. I should mention ...
user avatar
1 vote
1 answer
68 views

How to summarize multiple independent false discovery rates (FDRs or q-values)?

In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
user avatar
3 votes
0 answers
38 views

Sequential hypothesis test for uniform distribution

Suppose I have a sequence of data points $x_1,x_2,\dots$. I want to do a sequential hypothesis test, were the null hypothesis is that each $x_i$ is iid uniform on $[0,1]$, and the alternative ...
user avatar
  • 5,953
2 votes
1 answer
134 views

How to interpret the harmonic mean p-value?

I am working on a multiple comparisons problem for several hypothesis tests and have come across the harmonic mean $p$-value (HMP) for combining the $p$-value of these tests. Specifically, I am using ...
user avatar
  • 121
1 vote
0 answers
64 views

Combining tests: Maximum test statistic method but one test is a permutation test

I want to combine two tests $A,B$ for the same null and alternative hypotheses. The motivation is that it has been shown that test $A$ has higher power in some situations whereas test $B$ has higher ...
user avatar
  • 1,463
0 votes
1 answer
112 views

Combining p-values from permutation tests from single subjects

I have a question that's very similar to this question: Combine multiple independent permutation tests My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the ...
user avatar
3 votes
1 answer
94 views

combined p values from different hypothesis

I am currently working on a project in which I'm simultaneously identifying and quantifying protein in k samples. Experimentally, for each of the k samples, I have n identified proteins, with a p ...
user avatar
1 vote
1 answer
23 views

After derivng a test statistc using p-value combination method, what is the associat null hypothesis

First I generate two one-sided p-values from independent samples using null hypothesis $H_0: \Delta\leq0$, so that $p_1$ and $p_2$ are generated. Next, I apply the inverse normal combination function $...
user avatar
  • 63
2 votes
1 answer
104 views

Fisher's method for combining p-values

I'm using the sumlog function in the metap package of R, and have a few questions about the nuts-and-bolts of this function: 1)Some of my p-values are reported as '<0.001'. How do I include these?...
user avatar
  • 43
1 vote
1 answer
111 views

Meta-analysis with only p-values

I am doing a systematic review on the risk factors that predict infection after liver and pancreatic surgery. These risk factors include parameters like age, sex, smoking, pre-operative chemo etc. I ...
user avatar
  • 43
0 votes
0 answers
31 views

Cobine p-values, same data, same test, different labels

I have a dataset (300 samples vs 4000 features), I'm trying to extract meaningful features related to two condition A and B. ...
user avatar
2 votes
3 answers
108 views

Combining p-values when the trials are independent but their number is data-dependent and random

Consider the following scenario (it's just a motivating example, not something I am doing for real): I run a trial, drawing an i.i.d. sample $S_1$ from population $P$, to test a hypothesis $\mathcal{H}...
user avatar
  • 634
2 votes
1 answer
129 views

Test for multiple samples of ordinal data

I have 10 independent samples of ordinal data ranked 1-3. I also have predicted ranks for the same data set. I want to test if the predicted ranking matches the observed ranking at a rate better ...
user avatar
  • 1,082
1 vote
1 answer
38 views

Marginalizing statistical test results in a two-factor grid

Consider the following toy problem. Suppose that we have several food groups: meat, vegetables, fruit, etc. For any given pair of food types A and B, we ask whether people that eat both A AND B have a ...
user avatar
1 vote
1 answer
77 views

Interim analysis: probability of stopping

Say we do some time-varying study. We check yearly if early stopping is necessary, which is done via interim analysis (O'Brien-Fleming stopping boundaries). The 4 p-values for stopping in 4 ...
user avatar
  • 206
1 vote
1 answer
53 views

How to test the significant of multiply chi square tests

I know that if I have a model from which I have a very large sample, so that I can create a null distribution. And I also have an experimental sample, I can use the ChiSquare test (or FisherExact test)...
user avatar
  • 143
2 votes
1 answer
57 views

Combining the conclusions from multiple Diebold-Mariano tests

Suppose I have two forecasting models, A and B. Moreover, suppose I have multiple datasets and I use each of these sets to perform Diebold-Mariano tests. My aim is to find out which of the two models ...
user avatar
2 votes
0 answers
495 views

How to combine multiple p-values into one? [closed]

I have the data of 841 genes along with the p-values which represents the association of that gene in a particular tissue (Aorta in this case) with some diseased phenotype. I want to combine the p-...
user avatar
  • 21
30 votes
4 answers
3k views

Does a uniform distribution of many p-values give statistical evidence that H0 is true?

A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
user avatar
1 vote
1 answer
661 views

Bonferroni-type corrections and Fisher Method Meta-analysis

I'm not an expert on the field, but I need to know more about how to combined multiple significance test p-values after a two sample Kolmogorov Smirnov (KS)test on two distributions. But I have two ...
user avatar
  • 13
4 votes
1 answer
2k views

Problems with Fisher's Method for combining p-values

I am using Fisher's Method to combine p-values, and have noticed some strange behavior for large p-values and large $n.$ In my case I have a large number of not statistically significant results(e.g. ...
user avatar
  • 469
1 vote
2 answers
168 views

Rolling sum of 2 sample KS test results

In order to compare two lists of samples from 'before-treatment' and 'after-treatment', I am doing a two sample KS test using the ks_2samp function from Python's scipy.stats package which gives me the ...
user avatar
  • 13
0 votes
0 answers
1k views

GLMM with highly unbalanced data and statistical power

Although I have used R to explain my question, I think and hope this is of general interest and is not off-topic here. I want to test whether the continuous variable “varA” is different between the ...
user avatar
2 votes
1 answer
1k views

How to achieve a two-sided combined p-value using Fisher's method?

Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
user avatar
  • 306
0 votes
1 answer
168 views

Comparing 2 p values of different sample size

I did 2 linear regression analyses. White woman (WW) N1= 50 Black woman (BW) N2= 49 Correlation btw WW's V1 & V2 P = 0.001 Correlation btw BW's V1 & V2 P = 0.013 How do i know if the 2 p ...
user avatar
0 votes
1 answer
433 views

Probability density Function for p-value product

I am running an experiment where I get 22,152,480 p-values in a given order, ranging from 0 to 1, but strongly tilted toward 1 (i.e. not uniformly distributed). I want to say that a given series of ...
user avatar
  • 123
0 votes
1 answer
172 views

Combining p values: what can we say about population of studies

I have N p values (N is around 200) one for each of N independent experiments in my study. These p values range from 0.03 to 0.7 (mean is 0.3), where only 5% of p values are less than 0.05. If I look ...
user avatar
  • 11
30 votes
3 answers
4k views

Can a meta-analysis of studies which are all "not statistically signficant" lead to a "significant" conclusion?

A meta-analysis includes a bunch of studies, all of which reported a P value greater than 0.05. Is it possible for the overall meta-analysis to report a P value less than 0.05? Under what ...
user avatar
1 vote
1 answer
324 views

Performing a meta-analysis of wilcoxon rank sum analyses

Suppose an experiment is done and the following is presented in a paper: median for experimental, median for control, and a WRS p-value, and (n) for EXP and CTRL arms. Now suppose experiment 2 is ...
user avatar
  • 11
3 votes
1 answer
359 views

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say?

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say? What can I say about the underlying distribution's being a Student's-$t$...
user avatar
  • 11.8k
2 votes
1 answer
74 views

How would you compare subgroup Stouffer Zs?

p-curve analyses (http://www.p-curve.com/) use significant p-values to estimate evidential value and effect sizes. To estimate evidential value, p-values are computed for p-values (pp-values), and ...
user avatar
6 votes
0 answers
48 views

Can one use Stouffer's method to integrate t-values? [duplicate]

I would like to integrate T-statistics values from multiple, independent tests. How can this be done? Since my degrees of freedom are many ($n>1000$) I was thinking to use the Stouffer's method,...
user avatar
6 votes
1 answer
142 views

Using p-values reported as inequalities such as $p<0.05$ in meta-analysis; can I convert them to $0.05$?

I am currently undertaking a meta-analysis of the expression profile of specific biomarkers in a specific disease. During the extraction process, I extracted the name of the biomarkers and their ...
user avatar
1 vote
1 answer
523 views

Testing multiple comparisons of paired proportions

I want to know if a proportion increased significantly after an event, and then find out if that holds true over multiple events. I'm using a paired t-test to measure this with continuous data (e.g. ...
user avatar
2 votes
1 answer
456 views

Difference between Fisher's method and the sum of chi-square test statistics

I'm learning about combining $p$-values and I have a couple of (somewhat elementary, perhaps) questions. Suppose I have $n$ independent tests, each of these to test its own null hypothesis $H_0 (i)$, $...
user avatar
  • 167
0 votes
2 answers
1k views

How to calculate probability of success based on prior p-value

The FDA often requires a sponsor to conduct multiple clinical trials prior to approval. Given the following observations in a ph2 and ph3 trial, how would you go about predicting the probability of ...
user avatar
2 votes
1 answer
216 views

Combine multiple independent permutation tests

I want to find out whether a neural measure measured at 12 time-points is correlated with a behavioral measure measured on the same time-points. I have data from 20 subjects. I'm hoping to perform a ...
user avatar
3 votes
1 answer
509 views

Fisher's combined test vs testing uniform distribution of p-values

In testing a global null hypothesis, with independent tests, the p-values are distributed as $U(0,1)$. There are many goodness-of-fit tests to check that the distribution is uniform. For example, this ...
user avatar
  • 1,254
4 votes
0 answers
336 views

Reference for tail bound of chi square distribution [closed]

Fisher's method allows to combine a set of p values $p_1,\ldots,p_n$. In order to compute the p value associated with Fisher's method one needs to compute the tail of a chi square distribution with ...
user avatar
1 vote
1 answer
119 views

Combining different normality transformations for Fisher's method

I have 2 data sets that are applicable to Fisher's method. One has a p-value of 0.07 and the other 0.15. Neither sets were normal or homoscedastic. The first satisfied the ANOVA assumptions only with ...
user avatar
  • 11
2 votes
2 answers
1k views

meta-analysis of p values

I am a bit confused about doing the meta analysis with my data. Given z-scores I calculate p values in a way: p.values = 2*pnorm(-abs(z)). Then I would like to ...
user avatar
  • 449
3 votes
0 answers
102 views

Fisher's method for P-value more conservative than OLS?

I am currently observing an interesting phenomenon in my analysis. I have a simple logistic regression model for independent Inds. The model is as follows: $$\operatorname{logit}(Y) = \beta_0+\...
user avatar
  • 31
7 votes
2 answers
441 views

Test for significant excess of significant p-values across multiple comparisons

I have what feels like a simple question, but was unable to find answers easily. The situation Let's say I have a gene microarray dataset with tens of thousands of genes and small (<100) number ...
user avatar