Questions tagged [combining-p-values]
Methods for combining p-values from several hypothesis tests with the same null hypothesis into one aggregate p-value, such as e.g. Fisher's method and Stouffer's method.
93
questions
1
vote
1
answer
265
views
Ranking genes within GSEA
I know you can rank genes by p-/q-value, or by log2 fold change. But can you combine the two methods somehow?
I have read about combining fold change and p-value here but what is one advantage and one ...
CommunityBot
- 1
0
votes
0
answers
25
views
Algorithm for conducting Westfall & Young permutation test under Multiple Hypothesis Testing given a list of raw(unadjusted) p values
I am currently trying to set up a python program to implement Westfall & Young (1993)'s minP permutation procedure using Python. The computationally efficient algorithm I was able to find by ...
- 1
2
votes
0
answers
33
views
Understaning Fisher’s combined probability test in my analysis
I'm trying to understand Fisher's combined probability test, especially how I used it in my analysis. I have 25 datasets, each one containing genomic data from cancer patients. I conducted a ...
- 78.1k
4
votes
1
answer
339
views
How to interpret the harmonic mean p-value?
I am working on a multiple comparisons problem for several hypothesis tests and have come across the harmonic mean $p$-value (HMP) for combining the $p$-value of these tests. Specifically, I am using ...
CommunityBot
- 1
0
votes
1
answer
138
views
How to combine results from Wilcoxon Rank Sum Test for multiple imputed data sets from proc MI in SAS
Endpoint information:
We have seizure count collected for every day and therefore there will be some missing for some days.
We got average seizure frequency per 28-day, for an interval. That is, (...
- 4,513
7
votes
2
answers
288
views
Measure of difference or distinctiveness that's comparable across both discrete and continuous variables
I am a product manager for a data analytics startup, and on our platform our clients end up with large tables of data about their customers which has many attributes, which are of mixed datatype: some ...
- 1,246
0
votes
1
answer
44
views
How can we infer from the combined p-value of Fisher's method whether the null hypotheses are all true?
Using the R package oolong (oolong), I used Fisher's method to combine the p-values of three binomial tests. I have difficulty understanding the combined p-value. I understood it to mean that the ...
- 15.2k
0
votes
2
answers
112
views
How can the combined p-value be larger than one of the original two p-values?
I have two p-values from independent test which I wish to combine. I do this using fisher's method, but I get some results that I don't quite understand. If I have two p-values, and combine them, this ...
- 38.7k
-1
votes
1
answer
77
views
difference between two sets of pvalues
I am performing a biological experiment, where I am trying to capture potential mediation effects from some genes.
I have 150 significant results, containing 30 tissues with different sample size.
For ...
- 1,058
0
votes
1
answer
72
views
How can I get a p-value for a distribution that isn't normally distributed?
I have four independent trials of a random variable. I have run a Shapiro-Wilk test on it and it is not normally distributed.
I have two different images of a peak. The background is flat with a ...
- 13
0
votes
1
answer
26
views
Combining cause-specific mortality cox regressions into all-cause mortality
I only have access to the following dataset summary data and not to the dataset itself.
In the example, the stats on each line come from a univariate cox regression to model a cause-specific ...
- 78.5k
0
votes
0
answers
66
views
Combined p-value from wilcoxon test result on control experiment - interpreting the null-hypothesis correctly
EDITED
I spotted error in my code, and now results are no more contradictory. But still I keep the question for utility of others and myself in understanding meta-analysis and correctly interpret null-...
- 111
2
votes
1
answer
71
views
Combining the conclusions from multiple Diebold-Mariano tests
Suppose I have two forecasting models, A and B. Moreover, suppose I have multiple datasets and I use each of these sets to perform Diebold-Mariano tests. My aim is to find out which of the two models ...
- 67.6k
0
votes
0
answers
43
views
Combine p-values of several independent samples for combined hypothesis
I have a dataframe with 4 columns, corresponding to different emotions extracted with ML methods from one video. The dataframe describes one person, rows represent moments in time. Now I'm trying to ...
- 101
1
vote
1
answer
39
views
How to combine p-values of weighted effects?
Suppose I know the effect of running on strength of muscle 1 and 2 and their $p$-values. These values were calculated from the same sample. Now I want to know the effect of running on the strength of ...
- 2,291
2
votes
1
answer
322
views
What can I do if I want to apply Fisher's method (or some other method of combining p-values), but my p-values have unknown (positive) dependence?
I have p-values from several tests (k~50) that I'd like to combine. I don't have a model of their correlation structure. They aren't independent. In the worst-case scenario, my effective sample size ...
- 635
1
vote
0
answers
20
views
Combining association p-values from different H0 families
I have a pipeline that finds pairwise associations between three objects that are in a relationship as follows:
B
/ \
A --- C
What I get from this ...
0
votes
1
answer
53
views
Pearson chi squared for correlated sample
I am looking into the STS statistical test for random sequence.
There are multiple statistical tests for a random bit sequence and each of them produces a p-value or multiple p-value.
After the test ...
- 17.8k
1
vote
1
answer
34
views
Significance test for a pattern in union of sets given that the pattern is significant in the individual sets
We have a pattern like $P = \{f_{A},f_{B}\}$ which is statistically significant in multiple sets $S = \{S_{1}, S_{2}, S_{3}, ...\}$.
We can think of the features $f_{i}$ to be retail establishments &...
- 17.8k
4
votes
1
answer
475
views
Is this ok to average p values?
I have seen other posts about averaging p values. I believe I have a special case where average may apply.
Finding the exact probability distribution of my data is an NP hard task, so I am using a non ...
- 124k
0
votes
0
answers
432
views
What would you call combining multiple tests of a single hypothesis?
Multiple testing refers to testing more than one hypothesis simultaneously. The situation typically involves having several tests that cannot be combined into a single test, and therefore inference is ...
- 78.1k
5
votes
1
answer
173
views
Complementary p values do not cancel in many combining methods – why is this not a big deal?
What is this about?
Suppose I have performed two statistical tests (with a continuous distribution of $p$ values) for the same one-sided research hypothesis on different datasets, yielding the $p$ ...
- 17.8k
2
votes
2
answers
257
views
Rolling sum of 2 sample KS test results
In order to compare two lists of samples from 'before-treatment' and 'after-treatment', I am doing a two sample KS test using the ks_2samp function from Python's scipy.stats package which gives me the ...
- 2,291
1
vote
0
answers
102
views
How to combine test statistics and p-values from the same (stochastic) test repeated on a single dataset?
I would like to assess a generative model $M$'s goodness-of-fit using a simulation-based approach: I simulate some data from $M$, then compute a test statistic $T$ that measures how similar the ...
- 11
8
votes
2
answers
8k
views
Understanding Fisher's combined test
I am using Fisher's combined test to fuse several different independent tests. I have a problem understanding the results in some cases.
Example:
Let's say I run two different tests, both with the ...
1
vote
1
answer
29
views
Hypothesis testing with p-values from similar tests as prior
I hope to work on a hypothesis testing problem to test $\beta$ against the null hypothesis ($H_{null}$) $\beta=0$, and we can consider Wald test with the test statistics
$$
W = \dfrac{\beta^2}{var(\...
- 2,820
1
vote
0
answers
48
views
Test for change in all observerd variables
Let us say, I observe 3 variables in a control condition and a treatment condition. I would like to find out whether the treatment has some effect on all 3 variables at the same time. I should mention ...
- 173
1
vote
1
answer
145
views
How to summarize multiple independent false discovery rates (FDRs or q-values)?
In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
- 31
4
votes
0
answers
79
views
Sequential hypothesis test for uniform distribution
Suppose I have a sequence of data points $x_1,x_2,\dots$. I want to do a sequential hypothesis test, were the null hypothesis is that each $x_i$ is iid uniform on $[0,1]$, and the alternative ...
- 6,668
24
votes
4
answers
11k
views
Stouffer's Z-score method: what if we sum $z^2$ instead of $z$?
I am performing $N$ independent statistical tests with the same null hypothesis, and would like to combine the results into one $p$-value. It seems that there are two "accepted" methods: Fisher's ...
- 78.1k
7
votes
1
answer
2k
views
What does inverse-chi-square in Fisher method (classifying) exactly do?
In the book “Programming Collective Intelligence” Segaran explains the Fisher method for categorizing text as an alternative to Naive Bayes classifier. The Fisher method uses inverse-chi-square-...
- 78.1k
4
votes
0
answers
376
views
Reference for tail bound of chi square distribution [closed]
Fisher's method allows to combine a set of p values $p_1,\ldots,p_n$. In order to compute the p value associated with Fisher's method one needs to compute the tail of a chi square distribution with ...
- 78.1k
1
vote
0
answers
259
views
Combining tests: Maximum test statistic method but one test is a permutation test
I want to combine two tests $A,B$ for the same null and alternative hypotheses. The motivation is that it has been shown that test $A$ has higher power in some situations whereas test $B$ has higher ...
- 78.1k
0
votes
1
answer
321
views
Combining p-values from permutation tests from single subjects
I have a question that's very similar to this question: Combine multiple independent permutation tests
My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the ...
- 17.8k
2
votes
1
answer
294
views
Combine multiple independent permutation tests
I want to find out whether a neural measure measured at 12 time-points is correlated with a behavioral measure measured on the same time-points. I have data from 20 subjects.
I'm hoping to perform a ...
- 17.8k
1
vote
1
answer
616
views
Testing multiple comparisons of paired proportions
I want to know if a proportion increased significantly after an event, and then find out if that holds true over multiple events. I'm using a paired t-test to measure this with continuous data (e.g. ...
- 17.8k
6
votes
1
answer
147
views
Using p-values reported as inequalities such as $p<0.05$ in meta-analysis; can I convert them to $0.05$?
I am currently undertaking a meta-analysis of the expression profile of specific biomarkers in a specific disease.
During the extraction process, I extracted the name of the biomarkers and their ...
- 17.8k
3
votes
1
answer
175
views
combined p values from different hypothesis
I am currently working on a project in which I'm simultaneously identifying and quantifying protein in k samples. Experimentally, for each of the k samples, I have n identified proteins, with a p ...
- 17.8k
13
votes
3
answers
15k
views
R package for combining p-values using Fisher's or Stouffer's method
Is there an R package (or even a base R function) that implements Fisher's or Stouffer's method for combining p-values? Coding this should be almost trivial, but I'd rather use (and cite) a package.
...
1
vote
1
answer
25
views
After derivng a test statistc using p-value combination method, what is the associat null hypothesis
First I generate two one-sided p-values from independent samples using null hypothesis $H_0: \Delta\leq0$, so that $p_1$ and $p_2$ are generated. Next, I apply the inverse normal combination function $...
- 17.8k
2
votes
1
answer
271
views
Fisher's method for combining p-values
I'm using the sumlog function in the metap package of R, and have a few questions about the nuts-and-bolts of this function:
1)Some of my p-values are reported as '<0.001'. How do I include these?...
- 17.8k
1
vote
1
answer
263
views
Meta-analysis with only p-values
I am doing a systematic review on the risk factors that predict infection after liver and pancreatic surgery. These risk factors include parameters like age, sex, smoking, pre-operative chemo etc.
I ...
- 17.8k
10
votes
1
answer
3k
views
Why is my combined p-value, obtained using the Fisher's method, so low?
I have multiple individuals, for whom I collected two time series of some parameter. For each individual, I calculated, whether these time series are correlated. So, if I have 20 individuals, as a ...
0
votes
0
answers
31
views
Cobine p-values, same data, same test, different labels
I have a dataset (300 samples vs 4000 features), I'm trying to extract meaningful features related to two condition A and B. ...
- 113
59
votes
3
answers
28k
views
When combining p-values, why not just averaging?
I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
- 1
2
votes
3
answers
307
views
Combining p-values when the trials are independent but their number is data-dependent and random
Consider the following scenario (it's just a motivating example, not something I am doing for real): I run a trial, drawing an i.i.d. sample $S_1$ from population $P$, to test a hypothesis $\mathcal{H}...
- 1,057
2
votes
1
answer
194
views
Test for multiple samples of ordinal data
I have 10 independent samples of ordinal data ranked 1-3. I also have predicted ranks for the same data set. I want to test if the predicted ranking matches the observed ranking at a rate better ...
- 56.3k
1
vote
1
answer
44
views
Marginalizing statistical test results in a two-factor grid
Consider the following toy problem. Suppose that we have several food groups: meat, vegetables, fruit, etc. For any given pair of food types A and B, we ask whether people that eat both A AND B have a ...
- 133
1
vote
1
answer
98
views
Interim analysis: probability of stopping
Say we do some time-varying study. We check yearly if early stopping is necessary, which is done via interim analysis (O'Brien-Fleming stopping boundaries). The 4 p-values for stopping in 4 ...
- 206
1
vote
1
answer
62
views
How to test the significant of multiply chi square tests
I know that if I have a model from which I have a very large sample, so that I can create a null distribution. And I also have an experimental sample, I can use the ChiSquare test (or FisherExact test)...
- 143