Questions tagged [combining-p-values]
Methods for combining p-values from several hypothesis tests with the same null hypothesis into one aggregate p-value, such as e.g. Fisher's method and Stouffer's method.
78
questions
0
votes
0
answers
13
views
Calculate P values for regression models(Linear,Lasso Ridge,Elasticnet) manually [closed]
I want to calculate P value manually from the Multivariate Regression models(Lasso,ridge,Linear and Elasticnet) and i had tried to use the following code as shown below as one person suggested using ...
4
votes
1
answer
98
views
Complementary p values do not cancel in many combining methods – why is this not a big deal?
What is this about?
Suppose I have performed two statistical tests (with a continuous distribution of $p$ values) for the same one-sided research hypothesis on different datasets, yielding the $p$ ...
- 2,307
0
votes
0
answers
26
views
How to interpret the results of q value
I conducted multiple tests and had a group of P values.
I used the R function qvalue() to calculate the corresponding qvalues and plot it.
https://www.rdocumentation.org/packages/qvalue/versions/2.4.2/...
1
vote
0
answers
31
views
How to combine test statistics and p-values from the same (stochastic) test repeated on a single dataset?
I would like to assess a generative model $M$'s goodness-of-fit using a simulation-based approach: I simulate some data from $M$, then compute a test statistic $T$ that measures how similar the ...
- 11
0
votes
0
answers
24
views
Ordering drug effectiveness from best to worst
Problem
The (mean) effectiveness of a drug $d$ on trial $i$ is given by
\begin{align}
X_{di}:=\widehat{\mu}_{di} \sim \mathcal{N}(\mu_{di}, \sigma_{di}^2),
\end{align}
with known variances.
Hence, for ...
1
vote
1
answer
24
views
Hypothesis testing with p-values from similar tests as prior
I hope to work on a hypothesis testing problem to test $\beta$ against the null hypothesis ($H_{null}$) $\beta=0$, and we can consider Wald test with the test statistics
$$
W = \dfrac{\beta^2}{var(\...
user42004
0
votes
0
answers
24
views
Independence condition for Fisher's p-value method
Before describing how this relates to fisher's method, let me provide some background: I have a set of K genes. For each pair of genes, I can calculate a p-value for "probability that they ...
- 1
1
vote
0
answers
28
views
Test for change in all observerd variables
Let us say, I observe 3 variables in a control condition and a treatment condition. I would like to find out whether the treatment has some effect on all 3 variables at the same time. I should mention ...
- 173
1
vote
1
answer
68
views
How to summarize multiple independent false discovery rates (FDRs or q-values)?
In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
- 31
3
votes
0
answers
38
views
Sequential hypothesis test for uniform distribution
Suppose I have a sequence of data points $x_1,x_2,\dots$. I want to do a sequential hypothesis test, were the null hypothesis is that each $x_i$ is iid uniform on $[0,1]$, and the alternative ...
- 5,953
2
votes
1
answer
134
views
How to interpret the harmonic mean p-value?
I am working on a multiple comparisons problem for several hypothesis tests and have come across the harmonic mean $p$-value (HMP) for combining the $p$-value of these tests. Specifically, I am using ...
- 121
1
vote
0
answers
64
views
Combining tests: Maximum test statistic method but one test is a permutation test
I want to combine two tests $A,B$ for the same null and alternative hypotheses. The motivation is that it has been shown that test $A$ has higher power in some situations whereas test $B$ has higher ...
- 1,463
0
votes
1
answer
112
views
Combining p-values from permutation tests from single subjects
I have a question that's very similar to this question: Combine multiple independent permutation tests
My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the ...
- 165
3
votes
1
answer
94
views
combined p values from different hypothesis
I am currently working on a project in which I'm simultaneously identifying and quantifying protein in k samples. Experimentally, for each of the k samples, I have n identified proteins, with a p ...
1
vote
1
answer
23
views
After derivng a test statistc using p-value combination method, what is the associat null hypothesis
First I generate two one-sided p-values from independent samples using null hypothesis $H_0: \Delta\leq0$, so that $p_1$ and $p_2$ are generated. Next, I apply the inverse normal combination function $...
- 63
2
votes
1
answer
104
views
Fisher's method for combining p-values
I'm using the sumlog function in the metap package of R, and have a few questions about the nuts-and-bolts of this function:
1)Some of my p-values are reported as '<0.001'. How do I include these?...
- 43
1
vote
1
answer
111
views
Meta-analysis with only p-values
I am doing a systematic review on the risk factors that predict infection after liver and pancreatic surgery. These risk factors include parameters like age, sex, smoking, pre-operative chemo etc.
I ...
- 43
0
votes
0
answers
31
views
Cobine p-values, same data, same test, different labels
I have a dataset (300 samples vs 4000 features), I'm trying to extract meaningful features related to two condition A and B. ...
- 113
2
votes
3
answers
108
views
Combining p-values when the trials are independent but their number is data-dependent and random
Consider the following scenario (it's just a motivating example, not something I am doing for real): I run a trial, drawing an i.i.d. sample $S_1$ from population $P$, to test a hypothesis $\mathcal{H}...
- 634
2
votes
1
answer
129
views
Test for multiple samples of ordinal data
I have 10 independent samples of ordinal data ranked 1-3. I also have predicted ranks for the same data set. I want to test if the predicted ranking matches the observed ranking at a rate better ...
- 1,082
1
vote
1
answer
38
views
Marginalizing statistical test results in a two-factor grid
Consider the following toy problem. Suppose that we have several food groups: meat, vegetables, fruit, etc. For any given pair of food types A and B, we ask whether people that eat both A AND B have a ...
- 133
1
vote
1
answer
77
views
Interim analysis: probability of stopping
Say we do some time-varying study. We check yearly if early stopping is necessary, which is done via interim analysis (O'Brien-Fleming stopping boundaries). The 4 p-values for stopping in 4 ...
- 206
1
vote
1
answer
53
views
How to test the significant of multiply chi square tests
I know that if I have a model from which I have a very large sample, so that I can create a null distribution. And I also have an experimental sample, I can use the ChiSquare test (or FisherExact test)...
- 143
2
votes
1
answer
57
views
Combining the conclusions from multiple Diebold-Mariano tests
Suppose I have two forecasting models, A and B. Moreover, suppose I have multiple datasets and I use each of these sets to perform Diebold-Mariano tests. My aim is to find out which of the two models ...
- 445
2
votes
0
answers
495
views
How to combine multiple p-values into one? [closed]
I have the data of 841 genes along with the p-values which represents the association of that gene in a particular tissue (Aorta in this case) with some diseased phenotype. I want to combine the p-...
- 21
30
votes
4
answers
3k
views
Does a uniform distribution of many p-values give statistical evidence that H0 is true?
A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure ...
1
vote
1
answer
661
views
Bonferroni-type corrections and Fisher Method Meta-analysis
I'm not an expert on the field, but I need to know more about how to combined multiple significance test p-values after a two sample Kolmogorov Smirnov (KS)test on two distributions.
But I have two ...
- 13
4
votes
1
answer
2k
views
Problems with Fisher's Method for combining p-values
I am using Fisher's Method to combine p-values, and have noticed some strange behavior for large p-values and large $n.$
In my case I have a large number of not statistically significant results(e.g. ...
- 469
1
vote
2
answers
168
views
Rolling sum of 2 sample KS test results
In order to compare two lists of samples from 'before-treatment' and 'after-treatment', I am doing a two sample KS test using the ks_2samp function from Python's scipy.stats package which gives me the ...
- 13
0
votes
0
answers
1k
views
GLMM with highly unbalanced data and statistical power
Although I have used R to explain my question, I think and hope this is of general interest and is not off-topic here.
I want to test whether the continuous variable “varA” is different between the ...
- 71
2
votes
1
answer
1k
views
How to achieve a two-sided combined p-value using Fisher's method?
Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
- 306
0
votes
1
answer
168
views
Comparing 2 p values of different sample size
I did 2 linear regression analyses.
White woman (WW) N1= 50
Black woman (BW) N2= 49
Correlation btw WW's V1 & V2 P = 0.001
Correlation btw BW's V1 & V2 P = 0.013
How do i know if the 2 p ...
- 191
0
votes
1
answer
433
views
Probability density Function for p-value product
I am running an experiment where I get 22,152,480 p-values in a given order, ranging from 0 to 1, but strongly tilted toward 1 (i.e. not uniformly distributed). I want to say that a given series of ...
- 123
0
votes
1
answer
172
views
Combining p values: what can we say about population of studies
I have N p values (N is around 200) one for each of N independent experiments in my study.
These p values range from 0.03 to 0.7 (mean is 0.3), where only 5% of p values are less than 0.05. If I look ...
- 11
30
votes
3
answers
4k
views
Can a meta-analysis of studies which are all "not statistically signficant" lead to a "significant" conclusion?
A meta-analysis includes a bunch of studies, all of which reported a P value greater than 0.05. Is it possible for the overall meta-analysis to report a P value less than 0.05? Under what ...
- 15.1k
1
vote
1
answer
324
views
Performing a meta-analysis of wilcoxon rank sum analyses
Suppose an experiment is done and the following is presented in a paper: median for experimental, median for control, and a WRS p-value, and (n) for EXP and CTRL arms. Now suppose experiment 2 is ...
- 11
3
votes
1
answer
359
views
Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say?
Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say?
What can I say about the underlying distribution's being a Student's-$t$...
- 11.8k
2
votes
1
answer
74
views
How would you compare subgroup Stouffer Zs?
p-curve analyses (http://www.p-curve.com/) use significant p-values to estimate evidential value and effect sizes. To estimate evidential value, p-values are computed for p-values (pp-values), and ...
6
votes
0
answers
48
views
Can one use Stouffer's method to integrate t-values? [duplicate]
I would like to integrate T-statistics values from multiple, independent tests.
How can this be done?
Since my degrees of freedom are many ($n>1000$) I was thinking to use the Stouffer's method,...
- 189
6
votes
1
answer
142
views
Using p-values reported as inequalities such as $p<0.05$ in meta-analysis; can I convert them to $0.05$?
I am currently undertaking a meta-analysis of the expression profile of specific biomarkers in a specific disease.
During the extraction process, I extracted the name of the biomarkers and their ...
- 71
1
vote
1
answer
523
views
Testing multiple comparisons of paired proportions
I want to know if a proportion increased significantly after an event, and then find out if that holds true over multiple events. I'm using a paired t-test to measure this with continuous data (e.g. ...
- 11
2
votes
1
answer
456
views
Difference between Fisher's method and the sum of chi-square test statistics
I'm learning about combining $p$-values and I have a couple of (somewhat elementary, perhaps) questions. Suppose I have $n$ independent tests, each of these to test its own null hypothesis $H_0 (i)$, $...
- 167
0
votes
2
answers
1k
views
How to calculate probability of success based on prior p-value
The FDA often requires a sponsor to conduct multiple clinical trials prior to approval. Given the following observations in a ph2 and ph3 trial, how would you go about predicting the probability of ...
2
votes
1
answer
216
views
Combine multiple independent permutation tests
I want to find out whether a neural measure measured at 12 time-points is correlated with a behavioral measure measured on the same time-points. I have data from 20 subjects.
I'm hoping to perform a ...
3
votes
1
answer
509
views
Fisher's combined test vs testing uniform distribution of p-values
In testing a global null hypothesis, with independent tests, the p-values are distributed as $U(0,1)$. There are many goodness-of-fit tests to check that the distribution is uniform. For example, this ...
- 1,254
4
votes
0
answers
336
views
Reference for tail bound of chi square distribution [closed]
Fisher's method allows to combine a set of p values $p_1,\ldots,p_n$. In order to compute the p value associated with Fisher's method one needs to compute the tail of a chi square distribution with ...
- 101
1
vote
1
answer
119
views
Combining different normality transformations for Fisher's method
I have 2 data sets that are applicable to Fisher's method. One has a p-value of 0.07 and the other 0.15. Neither sets were normal or homoscedastic. The first satisfied the ANOVA assumptions only with ...
- 11
2
votes
2
answers
1k
views
meta-analysis of p values
I am a bit confused about doing the meta analysis with my data. Given z-scores I calculate p values in a way: p.values = 2*pnorm(-abs(z)). Then I would like to ...
- 449
3
votes
0
answers
102
views
Fisher's method for P-value more conservative than OLS?
I am currently observing an interesting phenomenon in my analysis. I have a simple logistic regression model for independent Inds. The model is as follows:
$$\operatorname{logit}(Y) = \beta_0+\...
- 31
7
votes
2
answers
441
views
Test for significant excess of significant p-values across multiple comparisons
I have what feels like a simple question, but was unable to find answers easily.
The situation
Let's say I have a gene microarray dataset with tens of thousands of genes and small (<100) number ...
- 4,387